[Xapian-discuss] Bug: get_docid() and multiple databases

[Jeff Rand]

I'm using the python bindings for xapian 1.2.12 and I'm getting some
unexpected behavior which I believe is a bug. While searching multiple
databases I am getting inconsistent values from doc.get_docid() when using
an overloaded KeyMaker class for custom sorting. The id value in the
document's data is the same as the id set for each document.

The behavior is expected when searching only one database: doc.get_docid()
== int(json.loads(doc.get_data())['id']) .

When searching more than one database the doc.get_data() will return a
value that is not the same as int(json.loads(doc.get_data())['id']).

According to the docs:
docid Xapian::Document::get_docid ( )  const

Get the document id which is associated with this document (if any).

NB If multiple databases are being searched together, then this will be the
document id in the individual database, not the merged database!

Here's my sample code and some output:

import xapian as x
import simplejson as json

db = x.Database()
db.add_database(x.Database('/var/xapian/db1.db')) #has XTYPA

q = x.Query('XTYPA')
q = x.Query(x.Query.OP_OR, q, x.Query('XTYPB'))

class WhatsTheId(x.KeyMaker):
    def __init__(self):

        return super(WhatsTheId, self).__init__()

    def __call__(self, doc):

        my_doc_id = json.loads(doc.get_data())['id']

        if my_doc_id <= 10:

            print doc.get_docid(), my_doc_id,
json.loads(doc.get_data())['type']
        return x.sortable_serialise(1)


e = x.Enquire(db)
e.set_query(q)
e.set_sort_by_key(WhatsTheId())
e.get_mset(0, 1000000000, 0, None)

# Expected results

2 2 A
3 3 A
4 4 A
5 5 A
6 6 A
7 7 A
8 8 A
9 9 A
10 10 A

db.add_database(x.Database('/var/xapian/db2.db')) #has XTYPB

e = x.Enquire(db)
e.set_query(q)
e.set_sort_by_key(WhatsTheId())
r = e.get_mset(0, 1000000000, 0, None)

# Add another, unexpected results

3 2 A
5 3 A
7 4 A
9 5 A
11 6 A
13 7 A
15 8 A
17 9 A
19 10 A
2 1 B
4 2 B

# It will consistently modify the internal get_docid value when adding more
databases:

q = x.Query(x.Query.OP_OR, q, x.Query('XTYPC'))
db.add_database(x.Database('/var/xapian/db3.db')) #has XTYPC
e = x.Enquire(db)
e.set_query(q)
e.set_sort_by_key(WhatsTheId())
r = e.get_mset(0, 1000000000, 0, None)

4 2 A
7 3 A
10 4 A
13 5 A
16 6 A
19 7 A
22 8 A
25 9 A
28 10 A
2 1 B
5 2 B
3 1 C
6 2 C
9 3 C
12 4 C
15 5 C
18 6 C
21 7 C
24 8 C
27 9 C
30 10 C

As I said, I believe that this is a bug.