Xapian on Windows for users with a non-ASCII login

[Jean-Francois Dockes]

Olly Betts writes:
 > Thanks for the -A patch - I've merged it.
Thanks !
 > On Thu, Aug 20, 2026 at 11:14:47AM +0200, Jean-Francois Dockes wrote:
 > > Olly Betts writes:
 > >  > On Wed, Aug 19, 2026 at 05:19:30PM +0200, Jean-Francois Dockes wrote:
 > >  > | > In terms of workarounds, simply changing directory to where the
 > >  > | > database lives and then using a relative non-wide path should work.
 > >  > |
 > >  > | It quite probably would, assuming that Xapian never computes an
 > >  > | absolute path (which you know, but I don't without scanning the code), and
 > >  > | also that there are no getcwd/chdir Windows pitfalls waiting for me...
 > >  > 
 > >  > There are no calls to getcwd() at all.
 > >  > 
 > >  > In general, I'd expect a library to avoid chdir() (unless that's part
 > >  > of its purpose perhaps) as it changes the current directory for the
 > >  > application too.  Xapian has a single call to chdir() in C/C++ code in
 > >  > Unix-specific locking code where we chdir("/") after fork() when we
 > >  > don't have OFD locks available (so the current directory of the
 > >  > lock-holding subprocess can't block unmounting of a filesystem).
 > >  > 
 > >  > I'm pretty sure we don't compute absolute paths anywhere.  We do resolve
 > >  > relative paths against a base directory, but that base directory can
 > >  > itself be a relative path in which case the result is still relative.
 > > 
 > > The Recoll indexer changes its current directory to one specified by the
 > > runtime configuration (TMPDIR by default), because some executed external
 > > format handlers tend/used to create temporary files in the current
 > > directory and are/were not too good with cleanup.
 > 
 > You could probably fork() then chdir() then exec() so the parent indexer
 > process doesn't need to chdir() to the scratch directory.

Yes, this is a good idea, I probably did not think long enough at the
time. Plugging a chdir in the programs' init function was simplest. Windows
CreateProcess() also has a current directory parameter, so this would have
been portable.

 > I've also definitely seen some filters leave junk in the current
 > directory, but omindex doesn't currently chdir() at all.  I guess we
 > must have worked around it with chdir() in a local script to run omindex
 > rather than addressing it in the omega code.  For an upstream-able
 > omindex fix we'd need to ensure the filenames of files being indexed
 > were absolute (which probably can be achieved by resolving a relative
 > start directory to an absolute path once at the start of an indexing
 > run).  I've made a note to look into this.

Being a user app, recoll can be started from anywhere, so it converts all
paths to absolute.

 > >  > I think the best way forwards is probably something you suggested in the
 > >  > previous thread - move to using std::filesystem.  We now require C++17,
 > >  > so it should be available.  I'm not sure how big a job that is though
 > >  > (I've not used std::filesystem anywhere yet).
 > > 
 > > I had another look at std::filesystem, and I don't think that it really
 > > solves the problem, it just dispatches depending on the input type (char vs
 > > wchar_t), and/or an explicit locale spec instead of having separate
 > > calls. To make use of it, it seems to me that you would have to increase
 > > the complexity of the Database interface by providing overloads for the
 > > methods which take paths. Maybe I'm wrong.
 > 
 > We might want to anyway, since that's useful for user code which uses
 > std::filesystem.

I looked a bit closer, and I think that using std::filesystem::path solves
both the case of unspecified char input (it uses the Windows code page,
including of course UTF-8 if this is in use) and the case of explicit UTF-8
input: use u8path (c++17) or char8_t (c++20). I agree that this is the way
forward, but of course this would have been a vastly more intrusive change
than the current patch.

 > > I still think that the approach in the patch is best on Windows: use the
 > > wide Win32 API when UTF-8 is detected, else use the narrow API and hope for
 > > the best. This keeps the Xapian API unchanged and maintains compatibility
 > > for hypothetical code which might be relying on code pages. The
 > > implementation is a bit wasteful if the input is actually ASCII, but the
 > > performance impact is probably negligible.
 > 
 > Didn't Microsoft finally add support for UTF-8 locales though?  Does
 > this work correctly in that case?

I think that, yes, if the current code page is UTF-8, the unpatched code
should work. I did not try though.

According to Gemini: UTF-8 is not the default code page for any Windows
version at the moment, but, for recent versions (Windows 10 version 1903),
you can force UTF-8 for a specific application through an "application
manifest".

The latter would be another possible approach, which I would be tempted to
use if the current patch did not exist and work unconditionally.

Cheers,

jf