Xapian on Windows for users with a non-ASCII login

[Olly Betts]

Thanks for the -A patch - I've merged it.

On Thu, Aug 20, 2026 at 11:14:47AM +0200, Jean-Francois Dockes wrote:
> Olly Betts writes:
>  > On Wed, Aug 19, 2026 at 05:19:30PM +0200, Jean-Francois Dockes wrote:
>  > | > In terms of workarounds, simply changing directory to where the
>  > | > database lives and then using a relative non-wide path should work.
>  > |
>  > | It quite probably would, assuming that Xapian never computes an
>  > | absolute path (which you know, but I don't without scanning the code), and
>  > | also that there are no getcwd/chdir Windows pitfalls waiting for me...
>  > 
>  > There are no calls to getcwd() at all.
>  > 
>  > In general, I'd expect a library to avoid chdir() (unless that's part
>  > of its purpose perhaps) as it changes the current directory for the
>  > application too.  Xapian has a single call to chdir() in C/C++ code in
>  > Unix-specific locking code where we chdir("/") after fork() when we
>  > don't have OFD locks available (so the current directory of the
>  > lock-holding subprocess can't block unmounting of a filesystem).
>  > 
>  > I'm pretty sure we don't compute absolute paths anywhere.  We do resolve
>  > relative paths against a base directory, but that base directory can
>  > itself be a relative path in which case the result is still relative.
> 
> The Recoll indexer changes its current directory to one specified by the
> runtime configuration (TMPDIR by default), because some executed external
> format handlers tend/used to create temporary files in the current
> directory and are/were not too good with cleanup.

You could probably fork() then chdir() then exec() so the parent indexer
process doesn't need to chdir() to the scratch directory.

I've also definitely seen some filters leave junk in the current
directory, but omindex doesn't currently chdir() at all.  I guess we
must have worked around it with chdir() in a local script to run omindex
rather than addressing it in the omega code.  For an upstream-able
omindex fix we'd need to ensure the filenames of files being indexed
were absolute (which probably can be achieved by resolving a relative
start directory to an absolute path once at the start of an indexing
run).  I've made a note to look into this.

>  > I think the best way forwards is probably something you suggested in the
>  > previous thread - move to using std::filesystem.  We now require C++17,
>  > so it should be available.  I'm not sure how big a job that is though
>  > (I've not used std::filesystem anywhere yet).
> 
> I had another look at std::filesystem, and I don't think that it really
> solves the problem, it just dispatches depending on the input type (char vs
> wchar_t), and/or an explicit locale spec instead of having separate
> calls. To make use of it, it seems to me that you would have to increase
> the complexity of the Database interface by providing overloads for the
> methods which take paths. Maybe I'm wrong.

We might want to anyway, since that's useful for user code which uses
std::filesystem.

> I still think that the approach in the patch is best on Windows: use the
> wide Win32 API when UTF-8 is detected, else use the narrow API and hope for
> the best. This keeps the Xapian API unchanged and maintains compatibility
> for hypothetical code which might be relying on code pages. The
> implementation is a bit wasteful if the input is actually ASCII, but the
> performance impact is probably negligible.

Didn't Microsoft finally add support for UTF-8 locales though?  Does
this work correctly in that case?

Cheers,
    Olly