[Xapian-discuss] Atomic DB rebuilds

[Sam Liddicott]

I've been doing work to keep a xapian DB in-step with a mysql DB.

I spent quite some time freaking about with mysqlbinlog replication, 
custom perl tail-ers, using perl mysql parsers and all kinds of magic to 
detect which rows in which tables have changed and therefore which 
dbi2omega'd records have been updated/removed.

In the end various issues including buggy sql parsers made me give up, 
the dataset isn't very big anyway, so I an afford to rebuild every hour 
or so, but what would not be nice is for some poor user to get no data 
at the point its rebuilt.

I think I can't rely on scriptindex not flushing as it will have to if 
the dataset gets too large, and in anycase I want to start by deleting 
everything.

But if I use symlinks (and I do) in /var/lib/omega/data to point to the 
db - well as we know, its impossible to atomically update a symlink, so 
I can't update another DB and then symlink to it once done.

Any tips for me?

Otherise, as I'm parsing the XML results in php so I can just wait a 
second and try again if I get no data, and presume it is the 
not-so-atomic update.

Sam