[Xapian-devel] goodbye QuartzBufferedTable

Olly Betts olly at survex.com
Fri Aug 13 22:04:25 BST 2004

On Fri, Aug 13, 2004 at 07:56:42PM +0100, Sam Liddicott wrote:
> Whoa! Stop there. This is exciting. You are getting 160 documents per 
> second even with half a million in the index?

Yes.  160/sec was sustained until around 600K docs, then it started to
tail off a little.  Currently it's doing 110/sec with 1800K docs.

I'll try flushing after 10K docs next I guess.

> Wow. Thats TOO much. [stops to jump up and down crazily] Wow.

I think there's still lots of scope for improvement.  I've only tried
a performance tuning a few things so far.

> Does "dangerous" mean it could corrupt the btree if interrupted?

Yes - if the power fails or the program is shut down uncleanly, the
database might be corrupt (normally a Xapian database should survive
either event).  But that doesn't give a huge boost - compare the two
graphs on the old box:


"No positional information and O_STREAMING, reuse btree object, no
fdatasync, "dangerous" patch applied:"


"No positional information and O_STREAMING, reuse btree object, no

The patch changes quartz to rewrite blocks in place rather than copying
them so that the switchover can be atomic.  The main benefit here is
that it's a bit more diskcache friendly.  And if you're rebuilding a
large database from scratch on a box in a datacentre connected to a
UPS, you're probably happy to trade a small risk of corruption for
a reduced build time.


More information about the Xapian-devel mailing list