[Xapian-tickets] [Xapian] #671: Performance issues when querying over large number of local databases (shards)
Xapian
nobody at xapian.org
Thu Mar 19 21:52:10 GMT 2015
#671: Performance issues when querying over large number of local databases
(shards)
-------------------------------+----------------------------------
Reporter: wgreenberg | Owner: olly
Type: defect | Status: new
Priority: normal | Milestone:
Component: Other | Version:
Severity: normal | Keywords: sharding performance
Blocked By: | Blocking:
Operating System: Linux |
-------------------------------+----------------------------------
Xapian seems to struggle when performing queries over a large(ish) number
of local databases. For my tests, I used a database of ~800,000 documents
with ~920,000 terms total sharded among 100 separate xapian databases.
These were then "joined" in a single stub database file, and queries were
performed by opening that stub database.
The following metrics were gathered on linux 3.13, xapian 1.3.2, on a
7200rpm spinning disk HDD.
I found that calling Enquire::get_mset for single-term queries on this
sharded database would take 1300-1600ms to complete, sometimes more. A
query on a non-sharded database containing the same data (same # of docs
and terms) usually takes <300ms to get the same mset. Running strace on
the process doing the queries showed that nearly all of this time was
spent waiting for pread64(2) to read a single block of each of the 100
shard's postlist files[1]. This read occurs when MultiMatch is
accumulating stats on all of its submatches, and requests termfreqs on
each postlist.
Subsequent queries of the same single term would result in times closer to
100ms, implying that these pread64 delays are mitigated when the VFS cache
has entries for the postlist files. Working based on this assumption, I
tried out a pretty gross patch[2] which calls readahead(2) on the desired
postlist blocks well before they're read. The results were significant:
Enquire::get_mset for queries on the sharded database with the patched
xapian took ~600ms. Presumably, other bad-but-predictable IO paths can be
mitigated in the same way (e.g. record.DB reads from Document::get_data).
There are probably other ways of addressing this issue without using
readahead(2) or its more portable cousin, posix_fadvise(2). I personally
think that the memory hit from priming the cache is acceptable, but I'd be
interested to know what others think.
[1] Here's an typical example strace of a sharded query that I ran through
xapian's strace-analyse script:
https://gist.github.com/wgreenberg/1f8264ff79815fefe5a5
[2] Super hacky patch to test the readahead theory:
https://github.com/wgreenberg/xapian/commit/d8af1fde685626283826ecaf52d4ac49cebdea98
--
Ticket URL: <http://trac.xapian.org/ticket/671>
Xapian <http://xapian.org/>
Xapian
More information about the Xapian-tickets
mailing list