[Xapian-tickets] [Xapian] #222: omindex should make use of O_NOATIME where available

[Xapian]

#222: omindex should make use of O_NOATIME where available
-------------------------+--------------------------------------------------
 Reporter:  olly         |        Owner:  olly     
     Type:  enhancement  |       Status:  assigned 
 Priority:  normal       |    Milestone:  1.2.x    
Component:  Omega        |      Version:  SVN trunk
 Severity:  normal       |   Resolution:           
 Keywords:               |    Blockedby:           
 Platform:  All          |     Blocking:           
-------------------------+--------------------------------------------------
Changes (by olly):

  * severity:  minor => normal


Old description:

> On Linux >= 2.6.8, open() accepts a O_NOATIME flag which is intended for
> use by
> "indexing or backup programs".  That means us!
>
> I have a patch for this, which I'll attach shortly.
>
> There's a wrinkle though - in some cases O_NOATIME will cause open to
> fail with
> EPERM and you need to retry the open call without O_NOATIME:
>
>        EPERM  The  O_NOATIME  flag was specified, but the effective user
> ID of
>               the caller did not match the owner of the file  and  the
> caller
>               was not privileged (CAP_FOWNER).
>
> So for example, if we're indexing /usr/share/doc as a non-root user, we
> incur an
> extra syscall for each file - in this case it would be more efficient not
> to use
> O_NOATIME at all.
>
> We need to quantify this overhead, and (if it's an issue) look at how to
> reduce
> it.  One thought I had was, on a per-directory basis, to give up on using
> O_NOATIME if we failed to open a file using it.  Then we only incur one
> syscall
> per directory for a read-only tree.  Various tweaks to this are possible
> - e.g.
> give up for this directory and all subdirectories.

New description:

 On Linux >= 2.6.8, open() accepts a O_NOATIME flag which is intended for
 use by
 "indexing or backup programs".  That means us!

 I have a patch for this, which I'll attach shortly.

 There's a wrinkle though - in some cases O_NOATIME will cause open to fail
 with
 EPERM and you need to retry the open call without O_NOATIME:

 {{{
        EPERM  The  O_NOATIME  flag was specified, but the effective user
 ID of
               the caller did not match the owner of the file  and  the
 caller
               was not privileged (CAP_FOWNER).
 }}}

 So for example, if we're indexing /usr/share/doc as a non-root user, we
 incur an
 extra syscall for each file - in this case it would be more efficient not
 to use
 O_NOATIME at all.

 We need to quantify this overhead, and (if it's an issue) look at how to
 reduce
 it.  One thought I had was, on a per-directory basis, to give up on using
 O_NOATIME if we failed to open a file using it.  Then we only incur one
 syscall
 per directory for a read-only tree.  Various tweaks to this are possible -
 e.g.
 give up for this directory and all subdirectories.

--

-- 
Ticket URL: <http://trac.xapian.org/ticket/222#comment:9>
Xapian <http://xapian.org/>
Xapian