[Xapian-tickets] [Xapian] #651: Protect writable fds by setting filepos very high

Xapian nobody at xapian.org
Fri Dec 16 06:00:27 GMT 2016


#651: Protect writable fds by setting filepos very high
---------------------------+-----------------------------
 Reporter:  olly           |             Owner:  olly
     Type:  enhancement    |            Status:  assigned
 Priority:  normal         |         Milestone:  1.5.0
Component:  Backend-Glass  |           Version:
 Severity:  normal         |        Resolution:
 Keywords:                 |        Blocked By:
 Blocking:                 |  Operating System:  All
---------------------------+-----------------------------
Changes (by olly):

 * milestone:  1.4.2 => 1.5.0


Comment:

 Looking at different Linux FSes on x86-64 with kernel 4.7.0, these are the
 max lseek-able offsets for some FSes:

 * btrfs - 0x7fffffffffffffff (max positive `off_t` value)
 * ext4 (4KB blocksize) - 0xffffffff000 (one block below 16TB, which is the
 documented max filesize)
 * ext2/3 (1KB blocksize) - 0x404043000
 * ext2/3 (4KB blocksize) - 0x1feff7fc000

 Even with an enhanced version of the probing code, some of these take a
 lot of lseek() calls to probe, and that is multiplied by the number of
 tables in use (up to 6) so naively probing on every open to write doesn't
 seem a good idea.

 The extra protection seems nice, but given we now avoid fds < 3 this is
 just an interesting idea which isn't actually worth the trouble.  It also
 doesn't protect if the code writing to a stale fd calls `lseek()` first.

 But to think it through a bit more, the lseek offset to use could be
 cached in the DB (e.g. saved in the version file with the other per-table
 info like root block number, freelist data, etc).  We need to detect if
 the file is moved to a different partition, or the FS rebuilt, or other
 things which might invalidate our cached value (e.g. also cache device ID
 and inode of the DB file).  Or we just check with `lseek()` first to one
 more than the cached value (which should fail), then to the cached value
 (which should succeed).  Most of the time that's all we need, but if we
 don't get the failure or success, we would need to reprobe.

 The only alternative to caching I can see if to build a lot of FS-specific
 knowledge into the library and check the type of FS and things like the
 block size if necessary, which sounds like a bad idea.

 So caching seems to be needed but adding info to the version file would
 need to wait for glass+1 - that means that unless someone can think of a
 better plan, this isn't 1.4.x material (or at least not unless we have a
 development backend in 1.4.x).

--
Ticket URL: <https://trac.xapian.org/ticket/651#comment:15>
Xapian <https://xapian.org/>
Xapian



More information about the Xapian-tickets mailing list