[Xapian-discuss] Feature request: Ligthen pressure on backup

[Arjen van der Meijden]

On 28-3-2008 7:21, Kevin Duraj wrote:
> I hate to distract community with and old issue about dropping Quartz
> database in Xapian 1.0 and adopting Flint database. Several of us
> posted threads that when we upgraded to Xapian 1.0 converting to Flint
> database we noticed severe degradation in indexing performance.

I haven't really looked at the indexing performance for a while, in our 
case its "fast enough", nor can I realistically compare it to Quartz 
results, since we're running with flint for quite a while and our 
dataset has grown quite a bit since than.

> We were told that Flint database is more compact than Quartz database
> and uses less disk space and we must sacrificed some performance
> because of it. It seems that this might not be entirely true, because
> after some inserts and deletes from Flint database, Flint database
> grows exponentially to the size of Quartz database and often must be
> compact. Looks like we do not have smaller database with Flint
> database than we had with Quartz database, but for sure we lost
> significant performance during indexing.

One of the changes in the later 1.0.x's was to reinstate the 
functionality to index all terms with both a stemmed and a non-stemmed 
version, rather than only with a stemmed version and for words with 
capitals both. Obviously that yields a increased index size. Are you 
sure that isn't what you're seeing?

For my database with the 0.9.7-version the non-compacted, non-zlib flint 
database was 19G and the 1.0.5 flint database is now 24G, almost all of 
this size increase is in the postlist-table (5G -> 9.7G). Obviously the 
new database hasn't seen as much updates as the 0.9.x yet.

What is odd, is that the compacted version of the new database is 
relatively a bit smaller than the old one (18G new vs 15G old) and that 
the new position-table hasn't been decreased in size compared to the old 
one (11G in both cases vs 11G new compacted and 9.7G old compacted).

> Who is behind the Flint database? Can we vote on this issue as community?

Olly is, and we for one are happy with the increased search performance 
it delivers. Its probably more usefull to figure out why its slower with 
indexing, than suggesting to remove it from the codebase.

Best regards,

Arjen