[Xapian-tickets] [Xapian] #198: Add support for multiple values in each value slot in a Document.

[Xapian]

#198: Add support for multiple values in each value slot in a Document.
---------------------------+------------------------------------------------
 Reporter:  richard        |        Owner:  richard  
     Type:  defect         |       Status:  assigned 
 Priority:  high           |    Milestone:  1.1.4    
Component:  Backend-Chert  |      Version:  SVN trunk
 Severity:  normal         |   Resolution:           
 Keywords:                 |    Blockedby:           
 Platform:  All            |     Blocking:  199      
---------------------------+------------------------------------------------

Comment(by olly):

 Um, I think you mean !StringListSerialiser...

 While you could use a Sorter to get out the first value for sorting (and
 probably for collapsing too eventually), that's a lot of extra virtual
 method calls, and worse, a lot of extra data to fetch that you didn't
 want.

 I'm not completely sold on the space overhead argument - it's misleading
 to claim that encode_length() doesn't store the length - that's what the
 top bit of each byte is used for, so it really incurs a 1 bit in 7
 overhead to do so (on average, roughly, since it rounds up the value's
 length to a multiple of 7 or 8 bits in each case).  And we could store
 multivalues by just concatenating the non-primary values and storing the
 split points using interpolative coding, which would probably be similar
 to 1 bit in 7.  It can certainly do better in some cases - e.g. if you're
 storing evenly distributed values from 0 to 255, then encode_length()
 would need 1.5 * N bytes, but the interpolative approach would need N (no
 overhead, since interpolative coding a full range takes no space!)

 And if you have to store the primary value in another slot to avoid the
 I/O overhead when you want to sort on it, that will cancel out any saving
 you made here and then some.

 Also you don't have to use multi-values if you think you can encode your
 stuff into a single value more efficiently than the multi-value
 implementation.

 So I think I still like this idea more than supplying serialisation
 helpers.

 I think the conclusion when we discussed this before is that the non-
 primary values wouldn't go in the stream, so you'd need to store the
 secondary values.  That certainly seems to make most sense to me.  Sure
 that's extra code, but then !StringListSerialiser is extra code too.  It
 might be a bit more complex to implement this internally, but multi-values
 seem a cleaner external API.

-- 
Ticket URL: <http://trac.xapian.org/ticket/198#comment:17>
Xapian <http://xapian.org/>
Xapian