[Xapian-discuss] FLAG_WILDCARD, add_database and performance

[Olly Betts]

On Mon, Aug 04, 2008 at 02:50:33PM +0200, Oliver Flimm wrote:
> 1915840  80.4207  libc-2.3.6.so            libc-2.3.6.so
> _int_malloc
>   1915840  99.9993  libc-2.3.6.so            libc-2.3.6.so
> _int_malloc [self]

Thanks.  That's not totally helpful (due to lack of debugging symbols in
your libxapian), but something seems to be calling malloc() or similar a
lot, which gives me a few clues, and I found a class which would benefit
from a rewrite.

Can you try this patch?  It's against SVN trunk, but will probably apply
to 1.0.7.  If not and you can't get it to apply, let me know and I'll
fix it up, but I need to go to bed right now.

http://oligarchy.co.uk/xapian/patches/multiallterms.patch

This passes the testsuite, but I wouldn't recommend it for production
use without further testing.

I suspect this will help quite a bit, but that more may be possible, so
when you rebuild Xapian, make sure that debugging information is being
generated ("-g" in CXXFLAGS, which is the default) and that you don't
strip the library.  Then if you need to rerun oprofile we can see where
in libxapian is making all these calls to libc.  Or whatever the new
bottleneck is.

Cheers,
    Olly