[Xapian-tickets] [Xapian] #651: Protect writable fds by setting filepos very high
Xapian
nobody at xapian.org
Fri Dec 16 06:00:27 GMT 2016
#651: Protect writable fds by setting filepos very high
---------------------------+-----------------------------
Reporter: olly | Owner: olly
Type: enhancement | Status: assigned
Priority: normal | Milestone: 1.5.0
Component: Backend-Glass | Version:
Severity: normal | Resolution:
Keywords: | Blocked By:
Blocking: | Operating System: All
---------------------------+-----------------------------
Changes (by olly):
* milestone: 1.4.2 => 1.5.0
Comment:
Looking at different Linux FSes on x86-64 with kernel 4.7.0, these are the
max lseek-able offsets for some FSes:
* btrfs - 0x7fffffffffffffff (max positive `off_t` value)
* ext4 (4KB blocksize) - 0xffffffff000 (one block below 16TB, which is the
documented max filesize)
* ext2/3 (1KB blocksize) - 0x404043000
* ext2/3 (4KB blocksize) - 0x1feff7fc000
Even with an enhanced version of the probing code, some of these take a
lot of lseek() calls to probe, and that is multiplied by the number of
tables in use (up to 6) so naively probing on every open to write doesn't
seem a good idea.
The extra protection seems nice, but given we now avoid fds < 3 this is
just an interesting idea which isn't actually worth the trouble. It also
doesn't protect if the code writing to a stale fd calls `lseek()` first.
But to think it through a bit more, the lseek offset to use could be
cached in the DB (e.g. saved in the version file with the other per-table
info like root block number, freelist data, etc). We need to detect if
the file is moved to a different partition, or the FS rebuilt, or other
things which might invalidate our cached value (e.g. also cache device ID
and inode of the DB file). Or we just check with `lseek()` first to one
more than the cached value (which should fail), then to the cached value
(which should succeed). Most of the time that's all we need, but if we
don't get the failure or success, we would need to reprobe.
The only alternative to caching I can see if to build a lot of FS-specific
knowledge into the library and check the type of FS and things like the
block size if necessary, which sounds like a bad idea.
So caching seems to be needed but adding info to the version file would
need to wait for glass+1 - that means that unless someone can think of a
better plan, this isn't 1.4.x material (or at least not unless we have a
development backend in 1.4.x).
--
Ticket URL: <https://trac.xapian.org/ticket/651#comment:15>
Xapian <https://xapian.org/>
Xapian
More information about the Xapian-tickets
mailing list