[Xapian-discuss] minor problem

[Olly Betts]

On Tue, Dec 25, 2007 at 03:18:53PM -0800, Kevin Duraj wrote:
> I am tracking the performance of indexing now for second year. The
> indexing performance has been dramatically deteriorating. Only users
> who haven't actually re indexed their data for long time as in  this
> example Michael, can only notice the slowness of Xapian indexing. As
> you can track my postings from about year ago I used to index 20
> million documents within an hour. Now I am indexing 50 million
> documents in about 29 hours.

Kevin - *you* are the only person to have reported this dramatically
deteriorating indexing performance.  We're simply not able to reproduce
it.  And, despite several requests, you've not shown us any information
which would help us to work out what is different about your setup, and
so what we can do to improve things.

If you actually want to get your problems resolved, then you really
are going to have to supply us with the information we've asked for.

On the other hand, if you're just interested in pointless trolling and
FUD-spreading, please celebrate the new year by finding yourself some
new people to bother.

> The biggest downshift in Xapian indexing performance was introducing
> Flint database with compression. The second down shift in performance
> was introducing locking Flint databases. However Xapian indexing is
> still the fastest compare to other technologies otherwise we wouldn't
> be here ...

The issue here is entirely different to what you've reported.

Flint has an increased overhead for the operation of opening a database
for writing compared to quartz.  This isn't a performance sensitive
operation for most people since if you're indexing a lot of data, you'll
open the database just once.  In exchange for this increased overhead,
we no longer have to put up with stale lock files getting left behind if
an indexing process is killed.

Cheers,
    Olly