[Xapian-tickets] [Xapian] #390: Include a way to log IO for profiling

[Xapian]

#390: Include a way to log IO for profiling
-------------------------+-------------------------
 Reporter:  richard      |             Owner:  olly
     Type:  enhancement  |            Status:  new
 Priority:  normal       |         Milestone:
Component:  Other        |           Version:
 Severity:  normal       |        Resolution:
 Keywords:               |        Blocked By:
 Blocking:               |  Operating System:  All
-------------------------+-------------------------
\
\
\
\
\
\

Comment (by olly):

 The attached script analyses output from strace to produce a list of
 blocks read from database tables.

 It tracks the filename associated with each fd (or at least with those
 which matter here) so it can report which database and table the block was
 read from.

 I think overall this is a better approach than adding logging calls to our
 code:

  * explicit logging calls may not get added everywhere - strace can't miss
 a call

  * strace should be lower overhead, at least if you tell it which syscalls
 are interesting with something like: {{{strace -e
 trace=open,pread,close,dup,dup2,dup3}}}

  * no need to recompile xapian-core to get the IO log (or carry the
 overhead of the logging when you don't need it)

 The drawback is strace isn't as portable - strace 4.7 actually dropped
 support for non-Linux platforms.  My thought is that an LD_PRELOAD wrapper
 is a good approach for other platforms - LD_PRELOAD or equivalent is
 available fairly widely.

 I've not tried to make this a drop-in replacement compatible with the log
 format from the patch here, but that shouldn't be hard to do if the
 visualisation tool is worth keeping.  You can get strace to log the time
 of each call to the nearest microsecond using {{{strace -ttt}}}).
\
\
\

--
Ticket URL: <http://trac.xapian.org/ticket/390#comment:1>
Xapian <http://xapian.org/>
Xapian