[Xapian-discuss] Xapian support for huge data sets?

Bill Hendrickson wjhendrickson at gmail.com
Thu May 12 19:18:29 BST 2011


Hello,

I’m currently using another open source search engine/indexer and am
having performance issues, which brought me to learn about Xapian.  We
have approximately 350 million docs/10TB data that doubles every 3
years.  The data mostly consists of Oracle DB records, webpage-ish
files (HTML/XML, etc.) and office-type docs (doc, pdf, etc.).  There
are anywhere from 2 to 4 dozen users on the system at any one time.
The indexing server has upwards of 28GB memory, but even then, it gets
extremely taxed, and will only get worse.

In the opinion of this list, would Xapian be able to handle this kind
of load, or should I evaluate more “enterprise”-like solutions (GSA,
etc.)?

Thanks.



More information about the Xapian-discuss mailing list