[Xapian-discuss] Xapian and quartz scalability - feedback of current users

[Olly Betts]

On Tue, Mar 22, 2005 at 01:13:23PM +0100, Arjen van der Meijden wrote:
> Does that handle updated documents? I.e. a document in the old database, 
> that needs to be replaced with one in the new database.

Not directly - you'd have to run an update on the compacted database.

> Would one need to specify a private document ID to achieve that, or
> will a scriptindex-style unique boolean suffice?  And document
> deletes?

Yes, you could use a unique boolean.  Minor updates wouldn't touch most
of the Btrees (less true for the postlist table) so the database would
still be fairly compact afterwards too.

> Would compacting an addition to an already compacted database be faster 
> than compacting to a normal database?

Should be - compacting is I/O bound in my experience.

Incidentally, I'll be adding a "fuller" compaction option soon which
will allow item chunks to be larger after compaction.  Currently they're
limited to allow at least 4 in a block, which is generally good for a
database you want to update, but does add some overhead.  It should help
record (if document data is ever more than about 2000 bytes), termlist,
and position tables.  It's hard to estimate how much until I try it
though...

Cheers,
    Olly