Weighting the author of a doc when that term can also appear as a frequent term in other docs

[Olly Betts]

On Thu, Sep 28, 2017 at 01:27:18PM -0400, Alex Aminoff wrote:
> We have a corpus of academic papers. Sometimes it happens that there is an
> academic controversy and one paper is a response or rebuttal to another
> paper. The name of the author of the first paper may appear many times in
> the second paper. So in light of this, how should we set our weight on the
> author field?
> 
> Here is an example:
> 
> http://www.nber.org/papers/w11215
> 
>  in which the term "Hoxby" appears 315 times, referring to several previous
> papers by Hoxby
> 
> http://www.nber.org/papers/w11216
> 
>  in which the term "Rothstein" is used 47 times
> 
> So if a user searches for "Hoxby", I would prefer that the comment on Hoxby
> not utterly dominate search results for which Hoxby is the author. But I
> don't want to set the weight on the author field to like 300, that would
> cause a search for "Moore's Law" to be dominated by results written by
> authors named Moore.
> 
> One suggestion someone had was what if the 300th mention of Hoxby was not as
> important as the first.

That's a valid intuition, and if you look at the various weighting formulae,
it's usually true that each additional occurrence adds less importance than
the previous one.  For some formulae extra occurrences may eventually even
reduce the importance.

> I tried to read
> 
>  https://xapian.org/docs/bm25.html
> 
> and I think I conclude that as long as f is small relative to L or K, the
> value of the expression will increase linearly with f.

If b is 0, then that term varies with f as:

  (k1 + 1) f / (k1 + f)

In the limit this tends to (k1 + 1) from below (though f can't grow without
limit as f <= L), and it is approximately linear for small f, but f >= 1
and an integer so e.g. with default k1=1 then f = 1, 2, 3, 4 gives 1, ~1.33,
1.5, 1.6 and the limit is 2 so it becomes sublinear quite quickly.

If b is 1, K is k1.L and that term varies with f as:

  (k1 + 1) f / (k1.L + f)
= (k1 + 1) Q / (k1 + Q)    where Q = f/L

1 <= f <= L so 0 < Q <= 1 (though 1 means f = L which is a document consisting
of just one word repeated over and over - in real data 0 < Q << 1).

At the default k1=1, if you graph this it is approximately linear close to
0 and drops below linear more as Q increases.  Again, not every value is
seen.

Values of b between 0 and 1 effectively blend the two, so smaller b will
probably work better in this regard (the default is b=0.5), though smaller
b will tend to prefer longer documents which can be undesirable.

> To make it less than linear, we might invoke
>
> >BM25 originally introduced another constant, as a power to which f and K
> >are raised. However, Stephen remarks that powers other than 1 were /'not
> >helpful'/, and other tests confirm this, so Xapian's implementation of
> >BM25 ignores this.
>
> If I could raise f to a power less than 1, that would do what I want. But I
> am not at all sure this is the right approach.

I've never seen a BM25 implementation which includes this, and in fact most
descriptions of BM25 don't even mention it - I think it was really an idea
during BM25's development which was abandonned.

It could be implemented, but it's probably of limited use, and will be
slower (partly because of a lot of calls to pow(), but also because it'll
make it harder to give a good upper bound).

I'd first look at tuning the available parameters to BM25Weight, and if
that doesn't solve the problem, I'd try some of the other weighting schemes
that Xapian already supports.  For example DPHWeight is apparently quite
effective at penalising term spamming, which is a similar problem to the
one you raise.

> Perhaps in real use this will turn out to be a minor issue.

It's certainly possible to overthink such issues.

Cheers,
    Olly