[Xapian-devel] Re: [Xapian-commits]7990: trunk/xapian-core/trunk/xapian-core/bin/trunk/xapian-core/tests/harness/

[Olly Betts]

On Sun, Apr 01, 2007 at 03:50:53PM +0100, Olly Betts wrote:
> On Mon, Apr 02, 2007 at 12:05:44AM +1000, Mark Hammond wrote:
> > It's all working fine for me - thanks!  I've a few outstanding patches from
> > xapian-maintainer-tools/win32msvc, but I'll add them to bugzilla over the
> > next few days.
> 
> Interesting - does "make check" currently pass for you under MSVC?

Still interested to know this!

> I found that the WinsockInitializer mechanism just didn't work as
> it currently is in SVN for TcpClient because the TcpClient constructor
> creates the socket and passes it to initialise the parent class which
> happens before the winsock_initializer member is initialised.
> 
> I've fixed this in my working copy by making the WinsockInitializer a
> base class of TcpClient (and listing it before RemoteDatabase) and then
> winsock gets initialised as intended.

This is now checked in.

> The problem is that tests seem to randomly fail to connect to the remote
> backend, which I think must be down to timing issues with starting
> xapian-tcpsrv versus trying to connect to it.  Under Unix we carefully
> wait until xapian-tcpsrv outputs "Listening" to avoid this problem.
> Perhaps adding a short "sleep" after starting xapian-tcpsrv will provide
> an acceptable workaround.

It doesn't appear to be timing issues.  It seems to be particular tests
which fail to connect.  The error isn't much help:

C:\xapian-core\tests>apitest -v -b remotetcp emptyquery1
Running anydb tests with remotetcp backend...
Running test: emptyquery1...The process cannot access the file because it is bei
ng used by another process.
 EXCEPTION: Program failed with exit code 1: start /...Program failed with exit
code 1: start /B ..\bin\xapian-tcpsrv --one-shot --interface 127.0.0.1 --port 12
39 -t300000 .flint/db=apitest_simpledata > %TEMP%\xapian-tcpsrv.out 2>&1' (outpu
t should be in %TEMP%\xapian-tcpsrv.out)
apitest completed test run: 0 tests passed, 1 failed.

Also, I've noticed that I can run multiple copies of xapian-tcpsrv on the *SAME
PORT* - a bit of inspired googling reveals this is Windows implementing bizarre
semantics for SO_REUSEADDR:

http://itamarst.org/writings/win32sockets.html

    SO_REUSEADDR is weird
    On Windows it lets you bind to the same TCP port multiple times without
    errors. Verified experimentally by failing test, I never bothered to check
    out MSDN docs for this one, I just don't use it on Windows.

I checked the MSDN docs the incorrect semantics seem to be deliberate, yet
result in non-deterministic behaviour - either listening process could get
incoming connections:

http://msdn2.microsoft.com/en-us/library/ms740621.aspx

They've also realised this is a security nightmare as a malcious process can
hijack connections to a port another process is already listening on by opening
it using SO_REUSEADDR, so as of NT 4.0 SP4 they've added SO_EXCLUSIVEADDRUSE.
Essentially this just trumps SO_REUSEADDR and so prevents the attack from
working - there doesn't seem to be a way to get the useful UNIX SO_REUSEADDR
semantics.  Worse, the docs describe problems with using SO_EXCLUSIVEADDRUSE
which sound similar to the reasons why you use SO_REUSEADDR on UNIX, but worse.

So in conclusion, I think we need to stop using SO_REUSEADDR on Windows.  And
perhaps we should also use SO_EXCLUSIVEADDRUSE where available but there seem
to be drawbacks so I'd like input from someone who actually knows more about
the murky world of winsock...

It appears cygwin also suffers from this, or at least did in 2004.

Cheers,
    Olly