[Xapian-devel] InMemory backend

[Olly Betts]

On Thu, Sep 09, 2004 at 10:36:11AM +0100, James Aylett wrote:
> On Thu, Sep 09, 2004 at 02:28:58AM +0100, Olly Betts wrote:
> 
> > It is useful to be able to open small temporary databases in memory,
> > but I think from a performance and from a code maintenance point of
> > view we'd be better off providing an alternative lower level for
> > quartz which uses memory instead of disk to store the Btree.  Or
> > perhaps replaces the Btree with a suitable map<>, but I think
> > a lower level is more appropriate.
> 
> At what layer does Quartz start caring about files? Immediately above
> the btree manager, or is there still a higher mechanism that abstracts
> that?

If the lower level stuff is "down", files only really matter at the
bottom part of the btree manager, except for creating directories
in QuartzWritableDatabase, and also the log file and meta file.

The simplest approach would probably be to allocate large blocks of
memory with mmap and MAP_ANON/MAP_ANONYMOUS.  Then use that memory
instead of a file - read and write just become memcpy.  Or even tweak
the cursors to just point into the mmaped area maybe.

IIRC, MAP_ANON in some form is supported by every modern UNIX, and
there's always malloc as a fallback (malloc doesn't allow the memory
to be returned to the OS after use).  Windows has a suprisingly usable
API for allocating large blocks of memory outside the malloc heap
that can be used here.

QuartzWritableDatabase can simply not create directories, the log
file isn't relevant, and the meta file is probably not needed except
to allow the database to persist on disk.  Similarly the baseA and
baseB files probably aren't needed for inmemory use.

Probably makes sense to leave this until after quartz has been
reworked a bit though, but bear it in mind during that.

Cheers,
    Olly