[Xapian-tickets] [Xapian] #337: Extend test of behaviour of databases after close

[Xapian]

#337: Extend test of behaviour of databases after close
------------------------+------------------------------
 Reporter:  richard     |             Owner:  gp1308
     Type:  defect      |            Status:  new
 Priority:  low         |         Milestone:  1.4.6
Component:  Test Suite  |           Version:  SVN trunk
 Severity:  normal      |        Resolution:
 Keywords:              |        Blocked By:
 Blocking:              |  Operating System:  All
------------------------+------------------------------

Comment (by olly):

 I'm not sure the local database case is as interesting as the remote one,
 as just closing the file descriptors isn't a realistic simulation of an
 actual failure mode, but FWIW it'd really need to happen in the test
 harness code, not the backend code.  The `Database::Internal` object (e.g.
 `GlassDatabase`) knows what fds are open on the database, but there's no
 way to ask it to report that information, and it doesn't seem an
 appropriate public API.  And better to avoid adding special private APIs
 just for the test harness if we can, not least because it's har

 The test harness supports running tests on different "backends" (which are
 slightly different to the library's concept of a backend, since a test
 harness "backend" specifies the whole setup - e.g. the test harness backed
 "remotetcp_glass" is the remote library-backend talking TCP to a remove
 server using the glass library-backend for its database).

 The harness knows the backend it's running tests for, and what features
 that supports (e.g. spelling, transactions) and testcases are flagged with
 what feature combinations they need.  The testcase just tells the harness
 things like "make me a `WritableDatabase` filled with data by generator
 function X", and the harness creates one of the right type with that data
 in.

 So the harness could (at least on platforms with `/proc/self/fd`) note
 which fds are in use, create the database, and check which additional fds
 are now in use - those are presumably open on the database we just opened.
 One potential flaw is that some library call made during DB
 opening/creation might open some file used internally lazily - e.g.
 `/dev/urandom` might get opened to generate a UUID for the database.  If
 that happens, we'd assume that file was one for the database.  It's also
 hard to make work if the test harness becomes multi-threaded, which is
 something that will probably happen in the future to speed up running the
 testsuite.

 Or perhaps compare the targets of the symlinks in `/proc/self/fd` with the
 directory path of the database, though that potentially has path
 normalisation issues, and I'm not sure if all platforms with
 `/proc/self/fd` have symlinks there (Linux does at least).

 There's existing code in `xapian-core/tests/harness/fdtracker.cc` which
 tracks fds so it can report when a testcase fails to close all the file
 descriptors it opened.  This works on Linux and a few other platforms
 which have `/proc/self/fd` - on other platforms we'll fail to open that
 special directory (which contains an entry for each fd open by the current
 process) and the FD checking is disabled automatically when that happens.

 I notice it has a subtle bug actually - the `DIR` handle that `opendir()`
 returns wraps a file descriptor, so that'll be reported by `readdir()` -
 you can see this effect if you `ls /proc/self/fd`:

 {{{
 $ ls -l /proc/self/fd
 total 0
 lrwx------ 1 olly olly 64 Apr 18 09:16 0 -> /dev/pts/8
 lrwx------ 1 olly olly 64 Apr 18 09:16 1 -> /dev/pts/8
 lrwx------ 1 olly olly 64 Apr 18 09:16 2 -> /dev/pts/8
 lr-x------ 1 olly olly 64 Apr 18 09:16 3 -> /proc/20289/fd
 }}}

 The first three entries are stdin, stdout, stderr all pointing to the
 terminal, the last is open on the directory `/proc/self/fd`  symlinks to
 and is from `ls` calling `opendir()` (or perhaps actually some lower level
 system call which works in a similar way).

 So really the FD tracking should get that fd using `dirfd()` on the `DIR`
 handle and discount it.

--
Ticket URL: <https://trac.xapian.org/ticket/337#comment:22>
Xapian <https://xapian.org/>
Xapian