YottaDB as a data store for Xapian

[Olly Betts]

On Thu, Feb 13, 2020 at 12:25:13PM -0500, K.S. Bhaskar wrote:
> YottaDB is 100% FOSS, licensed under AGPL v3 with a clarification
> (https://gitlab.com/YottaDB/DB/YDB/-/blob/master/LICENSE) that software
> which uses YottaDB is not considered a covered work. So if you make changes
> to YottaDB itself, your changes must follow AGPL v3, but if you write
> software that uses YottaDB, you can use any license  you want, FOSS or
> proprietary.

This licensing doesn't seem ideal for integrating with Xapian.

Currently Xapian is GPLv2+, but we're working towards relicensing
(probably as MPL 2.0) - see: https://trac.xapian.org/wiki/Licensing
One point raised in previous discussions was that users didn't want to
have additional restrictions over those of GPLv2.

The YottaDB licensing "clarification" doesn't seem to entirely help
here.

Consider a web application using Xapian packaged for a Linux distro like
Debian.  Debian requires the ability to apply patches to packages to
address security issues and severe bugs in stable releases (packaging a
whole new upstream version to fix one bug is not generally acceptable).
This means that a Debian package of YottaDB must really be assumed to
be a modified version of the upstream sources (because even if the
current package isn't modified, a patch could get applied at any point
in a stable release).  So that means one really needs to assume that
the AGPLv3 applies to the YottaDB package itself, so to comply with
that the application needs to provide a feature to allow downloading
the source code of the YottaDB package.  That would be an additional
requirement over the current GPLv2 requirements from using Xapian,
and seems non-ideal for us given the user feedback I mentioned above.

> As one of our users has suggested that YottaDB would make a good data store
> for Xapian, I am writing to explore this idea. On a system where YottaDB is
> installed, using it is just a matter of compiling a program including the
> YottaDB header file. So, if Xapian has an abstraction layer for the data
> store, perhaps we can add a software shim to Xapian that allows it to use
> YottaDB. I poked around the Xapian source code repository, but such an
> abstraction layer did not jump out at me.

You'd need to implement a new "database backend" in Xapian terms.  It's
more work that just "plug in a key-value store here", and to be honest
the backend interface is not currently entirely clearly defined in one
single place, but at least one external person has managed to implement
a working new backend.

This interface has seen some clean-up in git master since 1.4.x, so
look at git master because (a) it's probably a bit easier and (b)
then you don't have to rework it before it can actually be merged.

The main component is a subclass of Database::Internal
(backends/databaseinternal.h) and then there are subclasses of a few
other classes needed too (mostly iterators such as LeafPostList).
Some features (spelling, synonyms, user metadata, probably others too)
could be skipped in an initial version - it's probably a good idea to
get something that can at least index and search to check that there
isn't a terrible requirements vs features mismatch which could make the
whole thing unworkable.  For example, you're probably going to struggle
if the key-value store doesn't efficiently support searching for a key
and returning a cursor to the closest before or after if not found.

If you (or anyone) is actually going to try to implement a new backend
I'm happy to provide guidance, and to try to tidy up the location and
documentation for these classes.

Cheers,
    Olly