[Xapian-discuss] FLAG_WILDCARD, add_database and performance

Fri Aug 8 09:40:35 BST 2008

Hi,

On Tue, Aug 05, 2008 at 03:42:23PM +0100, Olly Betts wrote:
> On Mon, Aug 04, 2008 at 02:50:33PM +0200, Oliver Flimm wrote:
> > 1915840  80.4207  libc-2.3.6.so            libc-2.3.6.so
> > _int_malloc
> >   1915840  99.9993  libc-2.3.6.so            libc-2.3.6.so
> > _int_malloc [self]
> 
> Can you try this patch?  It's against SVN trunk, but will probably apply
> to 1.0.7.  If not and you can't get it to apply, let me know and I'll
> fix it up, but I need to go to bed right now.
> 
> http://oligarchy.co.uk/xapian/patches/multiallterms.patch
> 
> This passes the testsuite, but I wouldn't recommend it for production
> use without further testing.
> 
> I suspect this will help quite a bit, but that more may be possible, so
> when you rebuild Xapian, make sure that debugging information is being
> generated ("-g" in CXXFLAGS, which is the default) and that you don't
> strip the library.  Then if you need to rerun oprofile we can see where
> in libxapian is making all these calls to libc.  Or whatever the new
> bottleneck is.

Sorry for the delay. I patched 1.0.7 with your patch and built it with
the debugging information without stripping the library.

The request for 'java program*' now takes 235 seconds and here is the
output of oprofile:

CPU: Core 2, speed 2666.68 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 10000
0
samples  %        image name               app name
symbol name
-------------------------------------------------------------------------------
  1        100.000  libc-2.3.6.so            libc-2.3.6.so
__rpc_thread_destroy
487970   65.9735  libc-2.3.6.so            libc-2.3.6.so
_int_malloc
  487970   99.9990  libc-2.3.6.so            libc-2.3.6.so
_int_malloc [self]
  4        8.2e-04  libxapian.so.15.5.1      libc-2.3.6.so
Xapian::Query::Internal::get_description() const
  1        2.0e-04  libxapian.so.15.5.1      libc-2.3.6.so
std::basic_string<char, std::char_traits<char>, std:
:allocator<char> > std::operator+<char, std::char_traits<char>,
std::allocator<char> >(char const*, std::basic_string<char
, std::char_traits<char>, std::allocator<char> > const&)
-------------------------------------------------------------------------------
91871    12.4210  no-vmlinux               no-vmlinux
(no symbols)
  91871    100.000  no-vmlinux               no-vmlinux
(no symbols) [self]
-------------------------------------------------------------------------------
80236    10.8479  libxapian.so.15.5.1      libxapian.so.15.5.1
OrPostList::get_termfreq_est() const
  80236    100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
OrPostList::get_termfreq_est() const [self]
-------------------------------------------------------------------------------
11564     1.5635  libperl.so.5.8.8         libperl.so.5.8.8
(no symbols)
  11564    100.000  libperl.so.5.8.8         libperl.so.5.8.8
(no symbols) [self]
-------------------------------------------------------------------------------
10784     1.4580  libxapian.so.15.5.1      libxapian.so.15.5.1
LeafPostList::get_termfreq_est() const
  10784    100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
LeafPostList::get_termfreq_est() const [self]
-------------------------------------------------------------------------------
9925      1.3419  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string const
, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned
int> >, std::less<std::string>, std::allocator<std:
:pair<std::string const, unsigned int> > >::lower_bound(std::string
const&)
  9925     100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string con
st, unsigned int>, std::_Select1st<std::pair<std::string const,
unsigned int> >, std::less<std::string>, std::allocator<st
d::pair<std::string const, unsigned int> > >::lower_bound(std::string
const&) [self]
-------------------------------------------------------------------------------
6540      0.8842  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string const
, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned
int> >, std::less<std::string>, std::allocator<std:
:pair<std::string const, unsigned int> > >::find(std::string const&)
const
 6540     100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string con
st, unsigned int>, std::_Select1st<std::pair<std::string const,
unsigned int> >, std::less<std::string>, std::allocator<st
d::pair<std::string const, unsigned int> > >::find(std::string const&)
const [self]
-------------------------------------------------------------------------------
5551      0.7505  libstdc++.so.6.0.8       libstdc++.so.6.0.8
(no symbols)
  5551     100.000  libstdc++.so.6.0.8       libstdc++.so.6.0.8
(no symbols) [self]
-------------------------------------------------------------------------------
3922      0.5303  libxapian.so.15.5.1      libxapian.so.15.5.1
FlintPostList::get_termfreq() const
  3922     100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
FlintPostList::get_termfreq() const [self]
-------------------------------------------------------------------------------
2634      0.3561  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string const
, Xapian::MSet::Internal::TermFreqAndWeight>,
std::_Select1st<std::pair<std::string const,
Xapian::MSet::Internal::TermFre
qAndWeight> >, std::less<std::string>,
std::allocator<std::pair<std::string const,
Xapian::MSet::Internal::TermFreqAndWeig
ht> > >::insert_unique(std::pair<std::string const,
Xapian::MSet::Internal::TermFreqAndWeight> const&)
  2634     100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string con
st, Xapian::MSet::Internal::TermFreqAndWeight>,
std::_Select1st<std::pair<std::string const,
Xapian::MSet::Internal::TermF
reqAndWeight> >, std::less<std::string>,
std::allocator<std::pair<std::string const,
Xapian::MSet::Internal::TermFreqAndWe
ight> > >::insert_unique(std::pair<std::string const,
Xapian::MSet::Internal::TermFreqAndWeight> const&) [self]
-------------------------------------------------------------------------------
2533      0.3425  libxapian.so.15.5.1      libxapian.so.15.5.1
Key_::operator<(Key_) const
  2533     100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
Key_::operator<(Key_) const [self]
-------------------------------------------------------------------------------
2525      0.3414  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string const
, Xapian::MSet::Internal::TermFreqAndWeight>,
std::_Select1st<std::pair<std::string const,
Xapian::MSet::Internal::TermFre
qAndWeight> >, std::less<std::string>,
std::allocator<std::pair<std::string const,
Xapian::MSet::Internal::TermFreqAndWeig
ht> > >::find(std::string const&)
  2525     100.000  libxapian.so.15.5.1      libxapian.so.15.5.1
std::_Rb_tree<std::string, std::pair<std::string con
st, Xapian::MSet::Internal::TermFreqAndWeight>,
std::_Select1st<std::pair<std::string const,
Xapian::MSet::Internal::TermF
reqAndWeight> >, std::less<std::string>,
std::allocator<std::pair<std::string const,
Xapian::MSet::Internal::TermFreqAndWe
ight> > >::find(std::string const&) [self]

Regards,

Oliver

-- 
Universitaet zu Koeln :: Universitaets- und Stadtbibliothek
IT-Dienste :: Abteilung Universitaetsgesamtkatalog
Universitaetsstr. 33 :: D-50931 Koeln
Tel.: +49 221 470-3330 :: Fax: +49 221 470-5166
flimm at ub.uni-koeln.de :: www.ub.uni-koeln.de