[Xapian-tickets] [Xapian] #678: Remote backend doesn't work with large databases (was: Replication of >4GB files fails)

Xapian nobody at xapian.org
Sun Jun 21 14:03:53 BST 2015


#678: Remote backend doesn't work with large databases
----------------------------+-----------------------------
 Reporter:  matf            |             Owner:  olly
     Type:  defect          |            Status:  assigned
 Priority:  normal          |         Milestone:  1.3.4
Component:  Backend-Remote  |           Version:  1.2.20
 Severity:  normal          |        Resolution:
 Keywords:                  |        Blocked By:
 Blocking:                  |  Operating System:  All
----------------------------+-----------------------------
Changes (by olly):

 * component:  Replication => Backend-Remote


Comment:

 Ah, OK - thanks for the correction.

 In fact replication was already fixed for this back in 1.2.13:

 > + Allow files > 32G to be be copied by replication.

 (That NEWS entry is wrongly worded - it should say "> 4G" or "with a size
 which needs a > 32-bit integer").

 I just reviewed where `decode_length()` is called, and I think the problem
 isn't actually the size of the database in bytes (the remote backend never
 needs to send that information), but rather the total length of all
 documents in the database - we send that and then divide it by the number
 of documents to get the average document length.

 Can you check your database with `delve` (which might be installed as
 `xapian-delve`) to get the number of documents and the average length,
 like so:

 {{{
 $ delve db
 UUID = 00cb3616-dfc5-4113-a041-0f3d81961b0b
 number of documents = 566
 average document length = 109.346
 document length lower bound = 2
 document length upper bound = 532
 highest document id ever used = 566
 has positional information = true
 }}}

 So for this example, the total length is approximately 566*109.346 =
 61889.836.  If I've correctly diagnosed the cause of this, you should get
 > 4294967295.

--
Ticket URL: <http://trac.xapian.org/ticket/678#comment:7>
Xapian <http://xapian.org/>
Xapian



More information about the Xapian-tickets mailing list