[Xapian-devel] goodbye QuartzBufferedTable

[Olly Betts]

On Fri, Aug 13, 2004 at 07:56:42PM +0100, Sam Liddicott wrote:
> Whoa! Stop there. This is exciting. You are getting 160 documents per 
> second even with half a million in the index?

Yes.  160/sec was sustained until around 600K docs, then it started to
tail off a little.  Currently it's doing 110/sec with 1800K docs.

I'll try flushing after 10K docs next I guess.

> Wow. Thats TOO much. [stops to jump up and down crazily] Wow.

I think there's still lots of scope for improvement.  I've only tried
a performance tuning a few things so far.

> Does "dangerous" mean it could corrupt the btree if interrupted?

Yes - if the power fails or the program is shut down uncleanly, the
database might be corrupt (normally a Xapian database should survive
either event).  But that doesn't give a huge boost - compare the two
graphs on the old box:

http://www.survex.com/~olly/gmaneindexrate.html

"No positional information and O_STREAMING, reuse btree object, no
fdatasync, "dangerous" patch applied:"

vs:

"No positional information and O_STREAMING, reuse btree object, no
fdatasync:"

The patch changes quartz to rewrite blocks in place rather than copying
them so that the switchover can be atomic.  The main benefit here is
that it's a bit more diskcache friendly.  And if you're rebuilding a
large database from scratch on a box in a datacentre connected to a
UPS, you're probably happy to trade a small risk of corruption for
a reduced build time.

Cheers,
    Olly