[Xapian-tickets] [Xapian] #678: Remote backend doesn't work with large databases (was: Replication of >4GB files fails)
Xapian
nobody at xapian.org
Sun Jun 21 14:03:53 BST 2015
#678: Remote backend doesn't work with large databases
----------------------------+-----------------------------
Reporter: matf | Owner: olly
Type: defect | Status: assigned
Priority: normal | Milestone: 1.3.4
Component: Backend-Remote | Version: 1.2.20
Severity: normal | Resolution:
Keywords: | Blocked By:
Blocking: | Operating System: All
----------------------------+-----------------------------
Changes (by olly):
* component: Replication => Backend-Remote
Comment:
Ah, OK - thanks for the correction.
In fact replication was already fixed for this back in 1.2.13:
> + Allow files > 32G to be be copied by replication.
(That NEWS entry is wrongly worded - it should say "> 4G" or "with a size
which needs a > 32-bit integer").
I just reviewed where `decode_length()` is called, and I think the problem
isn't actually the size of the database in bytes (the remote backend never
needs to send that information), but rather the total length of all
documents in the database - we send that and then divide it by the number
of documents to get the average document length.
Can you check your database with `delve` (which might be installed as
`xapian-delve`) to get the number of documents and the average length,
like so:
{{{
$ delve db
UUID = 00cb3616-dfc5-4113-a041-0f3d81961b0b
number of documents = 566
average document length = 109.346
document length lower bound = 2
document length upper bound = 532
highest document id ever used = 566
has positional information = true
}}}
So for this example, the total length is approximately 566*109.346 =
61889.836. If I've correctly diagnosed the cause of this, you should get
> 4294967295.
--
Ticket URL: <http://trac.xapian.org/ticket/678#comment:7>
Xapian <http://xapian.org/>
Xapian
More information about the Xapian-tickets
mailing list