[Xapian-discuss] Re: indexing performance
Hongyan Ma
hym at ucla.edu
Wed Oct 13 15:56:13 BST 2004
At 08:14 PM 10/12/2004, Olly wrote:
> > XAPIAN_FLUSH_THRESHOLD defaults to 10000 anyway. I'd suggest trying
> > a larger value for that - 50000 works well for the gmane box which
> > has 3G of RAM. Your documents are tiny, so you could consider using
> > a larger value.
> I tried 100,000 as defaults. Sometimes it shows error " Exception:
> Error reading block ****: Result too large".
Olly: >Hmm, that means it's trying to read a bogus block number, which
>shouldn't ever happen. I tried a quick test with 2637157 documents and
>XAPIAN_FLUSH_THRESHOLD=100000 but I can't reproduce this.
The error doesn't come up regularly. I tried it again. This time, it shows the
following error with memory handling:
*** malloc: vm_allocate(size=176975872) failed (error code=3)
*** malloc[1276]: error: Can't allocate region
I put every 2,000,000 docs in a quartz db. It works ok with tiny docs -
average 80 bytes. But when it comes to larger ones - about 500 bytes each. The
errors show up.
Thanks,
Hongyan
More information about the Xapian-discuss
mailing list