YottaDB as a data store for Xapian
olly at survex.com
Thu Feb 13 19:13:11 GMT 2020
On Thu, Feb 13, 2020 at 12:25:13PM -0500, K.S. Bhaskar wrote:
> YottaDB is 100% FOSS, licensed under AGPL v3 with a clarification
> (https://gitlab.com/YottaDB/DB/YDB/-/blob/master/LICENSE) that software
> which uses YottaDB is not considered a covered work. So if you make changes
> to YottaDB itself, your changes must follow AGPL v3, but if you write
> software that uses YottaDB, you can use any license you want, FOSS or
This licensing doesn't seem ideal for integrating with Xapian.
Currently Xapian is GPLv2+, but we're working towards relicensing
(probably as MPL 2.0) - see: https://trac.xapian.org/wiki/Licensing
One point raised in previous discussions was that users didn't want to
have additional restrictions over those of GPLv2.
The YottaDB licensing "clarification" doesn't seem to entirely help
Consider a web application using Xapian packaged for a Linux distro like
Debian. Debian requires the ability to apply patches to packages to
address security issues and severe bugs in stable releases (packaging a
whole new upstream version to fix one bug is not generally acceptable).
This means that a Debian package of YottaDB must really be assumed to
be a modified version of the upstream sources (because even if the
current package isn't modified, a patch could get applied at any point
in a stable release). So that means one really needs to assume that
the AGPLv3 applies to the YottaDB package itself, so to comply with
that the application needs to provide a feature to allow downloading
the source code of the YottaDB package. That would be an additional
requirement over the current GPLv2 requirements from using Xapian,
and seems non-ideal for us given the user feedback I mentioned above.
> As one of our users has suggested that YottaDB would make a good data store
> for Xapian, I am writing to explore this idea. On a system where YottaDB is
> installed, using it is just a matter of compiling a program including the
> YottaDB header file. So, if Xapian has an abstraction layer for the data
> store, perhaps we can add a software shim to Xapian that allows it to use
> YottaDB. I poked around the Xapian source code repository, but such an
> abstraction layer did not jump out at me.
You'd need to implement a new "database backend" in Xapian terms. It's
more work that just "plug in a key-value store here", and to be honest
the backend interface is not currently entirely clearly defined in one
single place, but at least one external person has managed to implement
a working new backend.
This interface has seen some clean-up in git master since 1.4.x, so
look at git master because (a) it's probably a bit easier and (b)
then you don't have to rework it before it can actually be merged.
The main component is a subclass of Database::Internal
(backends/databaseinternal.h) and then there are subclasses of a few
other classes needed too (mostly iterators such as LeafPostList).
Some features (spelling, synonyms, user metadata, probably others too)
could be skipped in an initial version - it's probably a good idea to
get something that can at least index and search to check that there
isn't a terrible requirements vs features mismatch which could make the
whole thing unworkable. For example, you're probably going to struggle
if the key-value store doesn't efficiently support searching for a key
and returning a cursor to the closest before or after if not found.
If you (or anyone) is actually going to try to implement a new backend
I'm happy to provide guidance, and to try to tidy up the location and
documentation for these classes.
More information about the Xapian-discuss