[Xapian-discuss] Backing up a constantly updated Xapian index

[Adam Sjøgren]

I have an application that uses and updates a Xapian index of around
200 GB constantly.

I would like to take consistent backups of the index - and I can't stop
the updating for the time it would take to copy (because... users).

So it looks like I can use the replication functionality to maintain an
up to date copy of the index, and when I want to make a backup, I can
stop xapian-replicate, backup the replicated index, and then start
xapian-replicate again.

If I have XAPIAN_MAX_CHANGESETS set high enough, so that when I start
xapian-replicate after the backup, it won't copy the entire index again
from scratch... I learned :-)

So that is what I am testing. I got the first replication finished
(going full speed over the network), stopped xapian-replicate, and
checked the replicated index - it was fine.

Now I am starting xapian-replicate again - it doesn't copy the entire
index again from scratch I think (because disk-usage of the replicate is
constant, ~200GB).

I am wondering if it is possible to get xapian-replicate to tell some
more about what is happening, because after starting it again, it is
doing something, but I don't know what/how much.

I am running it like this:

  $ xapian-replicate --verbose --host=xx.xx.xx.xx --port=7010 -m index replica/index.backup
  Connecting to xx.xx.xx.xx:7010
  Getting update for replica/index.backup from index

And no more output - so far for about 1 hour.

I can see a trickle of data coming over port 7010, but not very much. Is
this to be expected after breaking the replication link for a while?

Can I get xapian-replicate to say a little more? - Should I dive into the
code and add extra verbosity (pointers welcome)?

Sorry for the longwindedness of this email.


  Thanks,

    Adam

-- 
 "A distributed system is one in which the failure of         Adam Sjøgren
  a computer you didn't even know existed can render     asjo at koldfront.dk
  your own computer unusable."