[Xapian-discuss] FLAG_WILDCARD, add_database and performance

[Olly Betts]

On Fri, Aug 08, 2008 at 10:40:35AM +0200, Oliver Flimm wrote:
> The request for 'java program*' now takes 235 seconds and here is the
> output of oprofile:

Hmm, so that's actually a bit slower.

>   1        100.000  libc-2.3.6.so            libc-2.3.6.so
> __rpc_thread_destroy
> 487970   65.9735  libc-2.3.6.so            libc-2.3.6.so
> _int_malloc
>   487970   99.9990  libc-2.3.6.so            libc-2.3.6.so
> _int_malloc [self]
>   4        8.2e-04  libxapian.so.15.5.1      libc-2.3.6.so
> Xapian::Query::Internal::get_description() const
>   1        2.0e-04  libxapian.so.15.5.1      libc-2.3.6.so
> std::basic_string<char, std::char_traits<char>, std:
> :allocator<char> > std::operator+<char, std::char_traits<char>,
> std::allocator<char> >(char const*, std::basic_string<char
> , std::char_traits<char>, std::allocator<char> > const&)

Hmm, this doesn't seem to make much sense - it's saying that
_int_malloc() in libc is calling
Xapian::Query::Internal::get_description()!

But a bit of digging around the web turned up this, which is a bug which
manifests in a scenario much like the one here:

http://sourceware.org/bugzilla/show_bug.cgi?id=4349

The age is probably about right for Debian etch too.

So this might be significantly improved by using an alternative malloc
implementation or a newer glibc.

> 80236    10.8479  libxapian.so.15.5.1      libxapian.so.15.5.1
> OrPostList::get_termfreq_est() const
>   80236    100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
> OrPostList::get_termfreq_est() const [self]

It looks like we're spending a lot of time in here though - 10.8479% of
235 seconds is 25.5 seconds.  It's essentially just a fairly simple
floating point computation.  I suspect it would save quite a bit of time
to handle this in a single object rather than a tree of pairwise
objects as that would avoid a lot of converting to and from integers
and numerous virtual method calls.

Is your testcase something you could sanely give me a copy of to prod?
If not, I'll try to build something similar when I get a chance.

Cheers,
    Olly