Xapian on Windows for users with a non-ASCII login

[Jean-Francois Dockes]

Olly Betts writes:
 > On Wed, Aug 19, 2026 at 05:19:30PM +0200, Xapian Discussion wrote:
 > > As already mentionned a few years ago, I maintain a xapian-core patch for having
 > > an index under a non-ASCII path (e.g. Korean) on MS-Windows.
 > 
 > Looks like this thread:
 > 
 > https://lists.xapian.org/pipermail/xapian-devel/2020-June/thread.html
 > 
 > BTW, in answer to:
 > 
 > | > In terms of workarounds, simply changing directory to where the
 > | > database lives and then using a relative non-wide path should work.
 > |
 > | It quite probably would, assuming that Xapian never computes an
 > | absolute path (which you know, but I don't without scanning the code), and
 > | also that there are no getcwd/chdir Windows pitfalls waiting for me...
 > 
 > There are no calls to getcwd() at all.
 > 
 > In general, I'd expect a library to avoid chdir() (unless that's part
 > of its purpose perhaps) as it changes the current directory for the
 > application too.  Xapian has a single call to chdir() in C/C++ code in
 > Unix-specific locking code where we chdir("/") after fork() when we
 > don't have OFD locks available (so the current directory of the
 > lock-holding subprocess can't block unmounting of a filesystem).
 > 
 > I'm pretty sure we don't compute absolute paths anywhere.  We do resolve
 > relative paths against a base directory, but that base directory can
 > itself be a relative path in which case the result is still relative.

The Recoll indexer changes its current directory to one specified by the
runtime configuration (TMPDIR by default), because some executed external
format handlers tend/used to create temporary files in the current
directory and are/were not too good with cleanup.

 > > I just updated the patch for Xapian 2.1.0.
 > > 
 > > This is useful for Recoll users with a non-ASCII user name, as the
 > > Recoll index is under their home directory by default. Recoll uses
 > > UTF-8 for the Xapian DB API calls, and the patch manages the
 > > conversion to wide chars (Unicode) Windows API calls.
 > > 
 > > The patch was judged too hacky for inclusion in the standard Xapian.
 > > It's not that much actually, the number of needed changes is quite
 > > small, in stable areas, and they have proved easy to maintain.
 > 
 > I think the best way forwards is probably something you suggested in the
 > previous thread - move to using std::filesystem.  We now require C++17,
 > so it should be available.  I'm not sure how big a job that is though
 > (I've not used std::filesystem anywhere yet).

I had another look at std::filesystem, and I don't think that it really
solves the problem, it just dispatches depending on the input type (char vs
wchar_t), and/or an explicit locale spec instead of having separate
calls. To make use of it, it seems to me that you would have to increase
the complexity of the Database interface by providing overloads for the
methods which take paths. Maybe I'm wrong.

I still think that the approach in the patch is best on Windows: use the
wide Win32 API when UTF-8 is detected, else use the narrow API and hope for
the best. This keeps the Xapian API unchanged and maintains compatibility
for hypothetical code which might be relying on code pages. The
implementation is a bit wasteful if the input is actually ASCII, but the
performance impact is probably negligible.

I could not find a way to have things working without special-casing Windows.

 > > Just in case this may be useful to anyone else:
 > > https://framagit.org/medoc92/xapian-core-unicode-filenames/-/tree/xapian-v2?ref_type=heads
 > 
 > I think we can at least merge the changes to use "A" suffixed versions
 > of API functions and associated structures, which would reduce the size
 > and scope of your patch.  They should be directly equivalent without
 > UNICODE defined, and clearly required for cases where we're passing a
 > non-wide string to the function (or expecting one returned, if anything
 > directly returns a string).
 > 
 > Would you mind creating a PR with those changes?

I created a PR on Github.

Have a good day !

jf