[Xapian-tickets] [Xapian] #222: omindex should make use of O_NOATIME where available

Xapian nobody at xapian.org
Tue Sep 30 08:34:18 BST 2008


#222: omindex should make use of O_NOATIME where available
-------------------------+--------------------------------------------------
 Reporter:  olly         |        Owner:  olly    
     Type:  enhancement  |       Status:  assigned
 Priority:  normal       |    Milestone:          
Component:  Omega        |      Version:  SVN HEAD
 Severity:  minor        |   Resolution:          
 Keywords:               |    Blockedby:          
 Platform:  All          |     Blocking:          
-------------------------+--------------------------------------------------

Old description:

> On Linux >= 2.6.8, open() accepts a O_NOATIME flag which is intended for
> use by
> "indexing or backup programs".  That means us!
>
> I have a patch for this, which I'll attach shortly.
>
> There's a wrinkle though - in some cases O_NOATIME will cause open to
> fail with
> EPERM and you need to retry the open call without O_NOATIME:
>
>        EPERM  The  O_NOATIME  flag was specified, but the effective user
> ID of
>               the caller did not match the owner of the file  and  the
> caller
>               was not privileged (CAP_FOWNER).
>
> So for example, if we're indexing /usr/share/doc as a non-root user, we
> incur an
> extra syscall for each file - in this case it would be more efficient not
> to use
> O_NOATIME at all.
>
> We need to quantify this overhead, and (if it's an issue) look at how to
> reduce
> it.  One thought I had was, on a per-directory basis, to give up on using
> O_NOATIME if we failed to open a file using it.  Then we only incur one
> syscall
> per directory for a read-only tree.  Various tweaks to this are possible
> - e.g.
> give up for this directory and all subdirectories.

New description:

 On Linux >= 2.6.8, open() accepts a O_NOATIME flag which is intended for
 use by
 "indexing or backup programs".  That means us!

 I have a patch for this, which I'll attach shortly.

 There's a wrinkle though - in some cases O_NOATIME will cause open to fail
 with
 EPERM and you need to retry the open call without O_NOATIME:

        EPERM  The  O_NOATIME  flag was specified, but the effective user
 ID of
               the caller did not match the owner of the file  and  the
 caller
               was not privileged (CAP_FOWNER).

 So for example, if we're indexing /usr/share/doc as a non-root user, we
 incur an
 extra syscall for each file - in this case it would be more efficient not
 to use
 O_NOATIME at all.

 We need to quantify this overhead, and (if it's an issue) look at how to
 reduce
 it.  One thought I had was, on a per-directory basis, to give up on using
 O_NOATIME if we failed to open a file using it.  Then we only incur one
 syscall
 per directory for a read-only tree.  Various tweaks to this are possible -
 e.g.
 give up for this directory and all subdirectories.

--

Comment(by olly):

 For Unix, we can look at {{{st_dev}}} in {{{struct stat}}} to determine if
 this is the same FS - O_NOATIME is likely to apply at the FS level.

-- 
Ticket URL: <http://trac.xapian.org/ticket/222#comment:4>
Xapian <http://xapian.org/>
Xapian



More information about the Xapian-tickets mailing list