Weighting the author of a doc when that term can also appear as a frequent term in other docs

Alex Aminoff aminoff at nber.org
Thu Sep 28 18:27:18 BST 2017


We have a corpus of academic papers. Sometimes it happens that there is 
an academic controversy and one paper is a response or rebuttal to 
another paper. The name of the author of the first paper may appear many 
times in the second paper. So in light of this, how should we set our 
weight on the author field?

Here is an example:

http://www.nber.org/papers/w11215

  in which the term "Hoxby" appears 315 times, referring to several 
previous papers by Hoxby

http://www.nber.org/papers/w11216

  in which the term "Rothstein" is used 47 times

So if a user searches for "Hoxby", I would prefer that the comment on 
Hoxby not utterly dominate search results for which Hoxby is the author. 
But I don't want to set the weight on the author field to like 300, that 
would cause a search for "Moore's Law" to be dominated by results 
written by authors named Moore.

One suggestion someone had was what if the 300th mention of Hoxby was 
not as important as the first. I tried to read

  https://xapian.org/docs/bm25.html

and I think I conclude that as long as f is small relative to L or K, 
the value of the expression will increase linearly with f. To make it 
less than linear, we might invoke

> BM25 originally introduced another constant, as a power to which f and 
> K are raised. However, Stephen remarks that powers other than 1 were 
> /'not helpful'/, and other tests confirm this, so Xapian's 
> implementation of BM25 ignores this.
>
If I could raise f to a power less than 1, that would do what I want. 
But I am not at all sure this is the right approach.

Perhaps in real use this will turn out to be a minor issue.

  - Alex








More information about the Xapian-discuss mailing list