[Xapian-tickets] [Xapian] #326: Some queries slower with chert than flint due to doclen access (was: Searches with chert are slow, due to slow doclen access)

[Xapian]

#326: Some queries slower with chert than flint due to doclen access
---------------------------+------------------------------------------------
 Reporter:  richard        |       Owner:  olly     
     Type:  defect         |      Status:  assigned 
 Priority:  highest        |   Milestone:  1.1.4    
Component:  Backend-Chert  |     Version:  SVN trunk
 Severity:  normal         |    Keywords:           
Blockedby:                 |    Platform:  All      
 Blocking:                 |  
---------------------------+------------------------------------------------

Comment(by olly):

 Retitling to better describe the issue (before any of the fixes, chert
 managed
 795 queries per second which isn't "slow", it's just slower than flint is
 on the
 same testcase), and this is particular to some queries - with more terms,
 or when the database isn't entirely in the OS cache, chert is likely to be
 faster (because it
 doesn't have to parse the doclength in every postlist, and because it uses
 less
 disk space).

 The recent changes to the key format are likely to improve chert's speed
 relative
 to flint, although probably more for the "can't cache it all case".

 While it would be nice to have chert faster than flint for every case
 before we make
 it the default backend in the current stable release, I don't think it is
 useful to
 hold up 1.2.0 indefinitely when chert is faster than flint for larger
 databases,
 just not in this one case you've singled out (and if the fair release
 blocking test
 would be trunk with chert against 1.0.18 with flint anyway).

 So I'm currently thinking that we shouldn't hold up 1.2.0 for this.  We
 can slot in
 further tweaks which don't change the format (e.g. to the pack and unpack
 routines)
 in 1.2.x if we find them.

 Rewriting the doclength storage at this point is potentially
 destabilising, so I
 think it's better to put that in the new development backend (brass)
 instead since
 we haven't even started to code that yet.

 Delaying 1.2.0 won't make brass won't be any faster or slower relative to
 flint,
 but it will inevitably push 1.4.0 further back too, which is a long-term
 loss.

-- 
Ticket URL: <http://trac.xapian.org/ticket/326#comment:17>
Xapian <http://xapian.org/>
Xapian