[Xapian-discuss] Re: indexing performance

Hongyan Ma hym at ucla.edu
Wed Oct 13 15:56:13 BST 2004


At 08:14 PM 10/12/2004, Olly wrote:
>  > XAPIAN_FLUSH_THRESHOLD defaults to 10000 anyway.  I'd suggest trying
>  > a larger value for that - 50000 works well for the gmane box which
>  > has 3G of RAM.  Your documents are tiny, so you could consider using
>  > a larger value.

> I tried 100,000 as defaults. Sometimes it shows error " Exception:
> Error reading block ****: Result too large".

Olly: >Hmm, that means it's trying to read a bogus block number, which
>shouldn't ever happen.  I tried a quick test with 2637157 documents and
>XAPIAN_FLUSH_THRESHOLD=100000 but I can't reproduce this.  
The error doesn't come up regularly. I tried it again. This time, it shows the 
following error with memory handling:
*** malloc: vm_allocate(size=176975872) failed (error code=3)
*** malloc[1276]: error: Can't allocate region
I put every 2,000,000 docs in a quartz db. It works ok with tiny docs - 
average 80 bytes. But when it comes to larger ones - about 500 bytes each. The 
errors show up.

Thanks,

Hongyan




More information about the Xapian-discuss mailing list