[Xapian-discuss] How to update DB concurrently?
James Aylett
james-xapian at tartarus.org
Thu May 18 12:16:14 BST 2006
On Wed, May 17, 2006 at 10:20:39PM +0100, Olly Betts wrote:
> Or even build them in the spool directory itself with ".tmp" on appended
> and skip those in the indexer (but I tend to favour separate directories
> as directories with lots of files are slower, significantly so on so
> filing systems.)
On a good OS, the directory size per se won't be a problem; the issue
will be whether you have to scan that directory. If you do, doom all
the way home.
> The only slight wrinkle is that on NFS rename() might return -1 even
> though it succeeded, as the Linux man page explains:
>
> If the server does the rename operation and then crashes, the
> retransmitted RPC which will be processed when the server is up
> again causes a failure.
>
> But so long as you're aware of this issue, it's easy enough to work
> around without incurring extra cost in the normal case. In perl,
> something like:
>
> while (!rename($tmp, $final)) {
> # Check if the rename really failed, or NFS is just being lame.
> last unless -f $tmp;
> # Handle the error condition by just waiting 5 minutes and retrying.
> print STDERR "*** rename($tmp,$final) failed ($?)\n";
> print STDERR "*** will retry in 300 seconds\n";
> sleep(300);
> }
Bear in mind that the underlying stat() behind ``-f $tmp'' can return
EINTR, so under heavy server/network load you might get -1 on the
rename() and then EINTR on the -f, despite the rename having succeeded
on the server.
I've no idea what happens when you get EINTR on -f, so I'd probably
rewrite that to while (!stat...) instead. I can't see a reason why
/that/ would break, though...
J
--
/--------------------------------------------------------------------------\
James Aylett xapian.org
james at tartarus.org uncertaintydivision.org
More information about the Xapian-discuss
mailing list