[Xapian-discuss] flush problem

[James Aylett]

On Sun, Jan 20, 2008 at 01:32:19PM -0500, Michael A. Lewis wrote:

> I am having a problem with flushing a database. I am adding N
> records to the DB (which amounts to 1 - 2000). At then end of the
> run, I issue a flush() call. The problem is that the flush call
> never seems to do anything. Every 10000 additions to the database
> and the library performs a flush (which can take up to 3 hours on a
> 560,000 document database) as if my flush call was never performed.
>  
> 1) This seems entirely too long, is it?

Sounds high to me, but it depends on so many factors: number of terms,
size of document data, available memory, how much memory is used by
Xapian to hold the 10k documents before flushing, logical to physical
volume layout, file systems involved...

What are you seeing as the main activity during flush? If you're on a
Unix machine it'll probably be one of system, user or iowait.

> 2) Why would my flush be ignored (no tranactions being used, just
> straight add using the term generator).
>  
> This is my flush code:
> 
> try {
>       Xapian::WritableDatabase* database = getDatabase();
>       database->flush();
> } catch (const Xapian::Error & err) {
>       s="ERROR:"+err.get_msg();
>       log_it(s.c_str());
>       write(c_id,"ERROR:-3",8);
> }
> return;

Assuming that getDatabase() implements the Singleton pattern
correctly, that you aren't clearing its instance, and that you aren't
using threading (or if you are you know what you're doing with
Singleton), this is odd.

I've had a quick look over the flint code, and I can't see how it
could not be working for you. If you compile with --enable-log and
then run with XAPIAN_DEBUG_LOG set to a file, and XAPIAN_DEBUG_FLAGS
set to -1, you'll get (lots!) of messages. You particularly should get
an apply call from flint after your flush; if you don't, it's not
working for some reason.

Everything will be slower with debugging on, of course.

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org