[Xapian-discuss] How to update DB concurrently?

[Peter Karman]

Olly Betts scribbled on 5/17/06 2:17 PM:

> chdir "spool" or die $!;
> while (1) {
>     my @files = glob "*.dump";
>     if (@files) {
> 	system "scriptindex", $database, $indexscript, @files or die $!;
> 	unlink @files;
>     } else {
> 	sleep 60;
>     }
> }
> 
> This "spool directory" style of design is both simple and suitably
> robust.


a good idea. probably need to do some file locking though, lest you feed 
a .dump file to scriptindex that your spooler is in the middle of 
writing to. Perl's flock() is decent, but if you deal with NFS, useless. 
I set up a similar system at one point, something like:

  open(LOCK, "> $dump.lock") or die "can't lock $dump: $!";
  print LOCK time();
  open(DUMP, ">> $dump") or die "can't write $dump: $!";
  # write your stuff to DUMP
  close(DUMP);
  close(LOCK);

and then in your indexing code:

  while (1) {
     my @files = glob "*.dump";
     if (@files) {
       for (@files)
       {
         next if -s "$_.lock";
	system "scriptindex", $database, $indexscript, $_ or die $!;
	unlink $_;
       }
     } else {
	sleep 60;
     }
  }

of course, it is Perl, so more than one way to do it. :)
-- 
Peter Karman  .  http://peknet.com/  .  peter at peknet.com