[Xapian-tickets] [Xapian] #671: Performance issues when querying over large number of local databases (shards)

[Xapian]

#671: Performance issues when querying over large number of local databases
(shards)
----------------------------------+--------------------------
 Reporter:  wgreenberg            |             Owner:  olly
     Type:  defect                |            Status:  new
 Priority:  normal                |         Milestone:
Component:  Matcher               |           Version:
 Severity:  normal                |        Resolution:
 Keywords:  sharding performance  |        Blocked By:
 Blocking:                        |  Operating System:  Linux
----------------------------------+--------------------------
Changes (by olly):

 * component:  Other => Matcher


Comment:

 Discussion on IRC reveals that wgreenbeeg's postlists only have one level
 (plus root) for wgreenberg's situation, so the patch does do what's
 intended there (as the root block should be in `C[level]`.  For the
 general case, it either needs to not descend the tree, to actually read
 the branch blocks touched, or to make multiple passes to read a level and
 pre-read the one below.

 Gmane's postlist.DB is 3 levels (plus the root I think) for 120 million
 documents, so the trees don't get very deep.  A pre-read of just the
 wanted blocks in the level below the root is likely to help even for those
 with deeper trees.

--
Ticket URL: <http://trac.xapian.org/ticket/671#comment:4>
Xapian <http://xapian.org/>
Xapian