[Xapian-discuss] Filesystems

[Arjen van der Meijden]

My colleague is testing several filesystems on our new search-machine. 
He has been looking at a few of the filesystems available in the 2.6.30 
linux kernel, ext2/3/4, xfs, btrfs, nilfs2 and reiser4.

"Unfortunately" the new machine has 24GB ram and 4x ssd in raid5. To get 
somewhat IO-bound results we had to cripple the machine (by making sure 
it couldn't use 20gb of those 24gb for file-cache) *and* cherry-pick our 
queries (only the heaviest with phrase-queries and such).

In the normal scenario of having the full ram (or even 4gb) available 
and the ssd's backing up any cache-miss, it is simply cpu-bound. And 
that is with the fastest x86 2-socket cpu's available right now, a pair 
of intel X5570's. The good news is that it actually appears to scale 
very well when using more cpu-cores (this one has 8 cores with 8 
hyper-threading cores) and that we can get about 90 searches per second 
out of it, which is more than we do now per minute (and we haven't 
benchmarked the compacted database yet).

I.e. our results indicate that for our reads it hardly matters which 
filesystem to pick, most of the database will be in RAM any way.

With the crippled, extra-io, read-scenario, we do see differences in 
performance between de filesystems tested.

When finished, we'll have numbers for linear writes (copying the 25GB 
database from another disk array), non-linear writes (updating the 
database) and semi-linear writes (compacting the database) with 
semi-linear (memory-backed) reads.
And of course the numbers for the crippled read-scenario.

So far ext4 and xfs seem to be the best choices, both in read and write 
scenario's. But obviously, our numbers are done with ssd, not normal disks.

We haven't yet tested the various mount/mkfs-options (apart from 
enabling noatime), but we'll probably settle for ext4 and then try a few 
options to better suit the filesystem to the underlying blockdevice. 
After that we'll also try what the various compaction-options do to the 
read-performance.

Best regards,

Arjen

On 1-7-2009 3:18 James Aylett wrote:
> On Tue, Jun 30, 2009 at 06:12:55PM -0700, Kevin Duraj wrote:
> 
>> Based on my observation Flint runs best on ext3 filesystems
> 
> I don't suppose you're able to share any of your numbers from this? I
> know it's not always possible, but having something on the wiki would
> be useful to people, if only to point them in useful directions of how
> to construct their own testing. (I assume you were comparing against
> JFS and XFS, maybe Reiser?)
> 
> J
>