Weighting Schemes: Evaluation results

[Vivek Pal]

Hi James,

> We probably don't want them committed in git where they're evaluation
> runs (because we can recreate them); a gist might be more appropriate.

Sorry, I have moved results files over to gist for each individual
weighting scheme.
Link: https://gist.github.com/ivmarkp/secret

> I can't tell, but are some of those files from FIRE? If so, they
> shouldn't be committed either; access to FIRE is via our usage
> agreement, and shouldn't be just public on the internet
> anywhere.

No, those files are generated each time a run is completed, and just
contain evaluation results that are displayed on terminal.

> Is there time in your schedule to get evaluation into the main xapian
> repo? That would avoid the first part of this. I don't think we're
> looking at lots more work to get this done, are we?

No, getting evaluation module merged in xapian is not a part of project
schedule but it is one of the additional tasks kept for later attention.
And now that I've run some evaluations, I think module is in good shape
also with support for more weighting schemes due to be added through these
PR's https://goo.gl/D2fviW.

> Can you remind me what sort of corpus you're using from FIRE for this?

The corpus we are using contains sorted news articles/stories based on
section and time period
from two different news providers; BDNews 24 and The Telegraph.

> Do you have any idea what 'very long' means in this case, in terms of
> number of terms (or maybe multiple of mean terms)

Very long documents in terms of no. of terms as specified in the paper; in
general, where |D| is much larger than avdl.

It is mentioned in the paper that "the MAP improvements of BM25+ over BM25
are much larger on Web collections than on the news collection. In
particular, the MAP improvements on all Web collections are statistically
significant." Therefore, they seem to have used four TREC collections: WT2G,
WT10G, Terabyte, and Robust04, which represent different sizes and genre of
text collections.

> Is this pure Dirichlet, or two-stage smoothing using Dir+
> versus Dir? What smoothing parameters were you using?

That is pure Dirichlet vs Dir+ and sorry, I should have also uploaded the
config which has the parameter details.
For Dir+ I used following parameters:

lmparam_log  0.0
lmparam_select_smoothing DIRICHLET_SMOOTHING
lmparam_smoothing1 0.9
lmparam_smoothing2 2000.0
lmparam_delta 0.05
lmparam_enable_dirplus 1

I've added config files in gists as well.

> Sorry you've been sick; make sure you're fully recovered before diving
> back in full throttle!

Thanks, I've gotten better. Should be no more hindrance in the days to come
:)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20160725/1fb8c79a/attachment.html>