[Xapian-tickets] [Xapian] #671: Performance issues when querying over large number of local databases (shards)

Xapian nobody at xapian.org
Thu Mar 19 21:52:10 GMT 2015


#671: Performance issues when querying over large number of local databases
(shards)
-------------------------------+----------------------------------
        Reporter:  wgreenberg  |      Owner:  olly
            Type:  defect      |     Status:  new
        Priority:  normal      |  Milestone:
       Component:  Other       |    Version:
        Severity:  normal      |   Keywords:  sharding performance
      Blocked By:              |   Blocking:
Operating System:  Linux       |
-------------------------------+----------------------------------
 Xapian seems to struggle when performing queries over a large(ish) number
 of local databases. For my tests, I used a database of ~800,000 documents
 with ~920,000 terms total sharded among 100 separate xapian databases.
 These were then "joined" in a single stub database file, and queries were
 performed by opening that stub database.

 The following metrics were gathered on linux 3.13, xapian 1.3.2, on a
 7200rpm spinning disk HDD.

 I found that calling Enquire::get_mset for single-term queries on this
 sharded database would take 1300-1600ms to complete, sometimes more. A
 query on a non-sharded database containing the same data (same # of docs
 and terms) usually takes <300ms to get the same mset. Running strace on
 the process doing the queries showed that nearly all of this time was
 spent waiting for pread64(2) to read a single block of each of the 100
 shard's postlist files[1]. This read occurs when MultiMatch is
 accumulating stats on all of its submatches, and requests termfreqs on
 each postlist.

 Subsequent queries of the same single term would result in times closer to
 100ms, implying that these pread64 delays are mitigated when the VFS cache
 has entries for the postlist files. Working based on this assumption, I
 tried out a pretty gross patch[2] which calls readahead(2) on the desired
 postlist blocks well before they're read. The results were significant:
 Enquire::get_mset for queries on the sharded database with the patched
 xapian took ~600ms. Presumably, other bad-but-predictable IO paths can be
 mitigated in the same way (e.g. record.DB reads from Document::get_data).

 There are probably other ways of addressing this issue without using
 readahead(2) or its more portable cousin, posix_fadvise(2). I personally
 think that the memory hit from priming the cache is acceptable, but I'd be
 interested to know what others think.

 [1] Here's an typical example strace of a sharded query that I ran through
 xapian's strace-analyse script:
 https://gist.github.com/wgreenberg/1f8264ff79815fefe5a5

 [2] Super hacky patch to test the readahead theory:
 https://github.com/wgreenberg/xapian/commit/d8af1fde685626283826ecaf52d4ac49cebdea98

--
Ticket URL: <http://trac.xapian.org/ticket/671>
Xapian <http://xapian.org/>
Xapian



More information about the Xapian-tickets mailing list