[Xapian-tickets] [Xapian] #278: When changesets are being generated, old changesets aren't cleaned up

[Xapian]

#278: When changesets are being generated, old changesets aren't cleaned up
---------------------------+------------------------------------------------
 Reporter:  richard        |        Owner:  richard 
     Type:  defect         |       Status:  assigned
 Priority:  normal         |    Milestone:  1.1.0   
Component:  Backend-Chert  |      Version:  SVN HEAD
 Severity:  normal         |   Resolution:          
 Keywords:                 |    Blockedby:          
 Platform:  All            |     Blocking:  227     
---------------------------+------------------------------------------------

Old description:

> Currently, changesets are generated when the "XAPIAN_MAX_CHANGESETS"
> environment variable is set to a non-empty value.  However, they are
> never removed.  Whenever a changeset is generated, the number of
> changesets around should be checked, and old changesets should be removed
> if too many old changesets exist.
>
> Alternatively (or as well), a different criteria might be useful for the
> changesets: it might be useful to be able to set an absolute limit on the
> total size of the changesets, or perhaps, a limit on the total size of
> the changesets as a proportion of the total database size.
>
> Finally, it is never useful to have a changeset for the first revision of
> a database, so creation of that changeset should be automatically
> disabled.

New description:

 Currently, changesets are generated when the "XAPIAN_MAX_CHANGESETS"
 environment variable is set to a non-empty value.  However, they are never
 removed.  Whenever a changeset is generated, the number of changesets
 around should be checked, and old changesets should be removed if too many
 old changesets exist.

 Alternatively (or as well), a different criteria might be useful for the
 changesets: it might be useful to be able to set an absolute limit on the
 total size of the changesets, or perhaps, a limit on the total size of the
 changesets as a proportion of the total database size.

--

Comment(by olly):

 Fixed to not generate a changeset for the first revision in trunk r11069
 (description updated).

 In order to efficiently implement keeping a maximum number of changesets
 and/or maximum size of changesets, I think we want to store the lowest
 changeset number somewhere.  Then we can calculate current_revision -
 lowest_changeset_number and compare with maximum_number to implement that.
 And if we also store the total size (as number of blocks would make most
 sense probably), we can then delete the lowest numbered changeset files
 and decrease this total appropriately until it is within the specified
 limit.  We can also age by date by checking the timestamps of the lowest
 numbered changeset files.

 (We could establish the lowest changeset number in
 O(log(current_revision)) by binary chop, but that seems unnecessary
 work...)

-- 
Ticket URL: <http://trac.xapian.org/ticket/278#comment:3>
Xapian <http://xapian.org/>
Xapian