[Xapian-discuss] improve indexing performance

[Kevin Duraj]

For such a small amount of documents as 10,000 you can use any search engine as MySQL, Lucene or even MSSQL. 

If you want to index 300 millions of documents as I do at 
http://myhealthcare.com nothing else works, only Xapian. 

PS: Hopefully, one day you become a professional search engine developer and you will know it, when you start counting your documents in millions, not thousands. 

Kevin Duraj
MyHealthcare.com

On Sep 6, 2012, at 3:51 AM, Graham Jones <your-name-here at grahamjones.org> wrote:

> 1) index in ram - i.e. put your files in a ramdisk
> 2) Index in parallel and merge with xapian-compact afterwards
> 3) Just use the Xapian api as documented - you don't need to do anything special.
> Its good for over 10,000 documents a second with a modest number of parallel processes (say 10-20) on a typical enterprise server.
> 
> 
> On 06/09/2012, at 7:30 PM, Előd Biszak <biszakelod at gmail.com> wrote:
> 
>> Hi!
>> 
>> I'm indexing a huge amount of documents. I'm adding the documents one by
>> one to tha database. Is there a way of improving indexing performance? I'm
>> interested in suggestions also programmatical and hardware wise.
>> 
>> Thanks in advance,
>> Biszak Előd
>> _______________________________________________
>> Xapian-discuss mailing list
>> Xapian-discuss at lists.xapian.org
>> http://lists.xapian.org/mailman/listinfo/xapian-discuss
> 
> 
> _______________________________________________
> Xapian-discuss mailing list
> Xapian-discuss at lists.xapian.org
> http://lists.xapian.org/mailman/listinfo/xapian-discuss