[Xapian-discuss] How to update DB concurrently?

James Aylett james-xapian at tartarus.org
Thu May 18 12:16:14 BST 2006


On Wed, May 17, 2006 at 10:20:39PM +0100, Olly Betts wrote:

> Or even build them in the spool directory itself with ".tmp" on appended
> and skip those in the indexer (but I tend to favour separate directories
> as directories with lots of files are slower, significantly so on so
> filing systems.)

On a good OS, the directory size per se won't be a problem; the issue
will be whether you have to scan that directory. If you do, doom all
the way home.

> The only slight wrinkle is that on NFS rename() might return -1 even
> though it succeeded, as the Linux man page explains:
> 
>     If the server does the rename operation and then crashes, the
>     retransmitted RPC which will be processed when the server is up
>     again causes a failure.
>
> But so long as you're aware of this issue, it's easy enough to work
> around without incurring extra cost in the normal case.  In perl,
> something like:
> 
>   while (!rename($tmp, $final)) {
>     # Check if the rename really failed, or NFS is just being lame.
>     last unless -f $tmp;
>     # Handle the error condition by just waiting 5 minutes and retrying.
>     print STDERR "*** rename($tmp,$final) failed ($?)\n";
>     print STDERR "*** will retry in 300 seconds\n";
>     sleep(300);
>   }

Bear in mind that the underlying stat() behind ``-f $tmp'' can return
EINTR, so under heavy server/network load you might get -1 on the
rename() and then EINTR on the -f, despite the rename having succeeded
on the server.

I've no idea what happens when you get EINTR on -f, so I'd probably
rewrite that to while (!stat...) instead. I can't see a reason why
/that/ would break, though...

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org



More information about the Xapian-discuss mailing list