GSoC-2017 Introduction and Project Discussion

[Shivang Bansal]

Hello,


I'm Shivang Bansal, a 3rd year Computer Science Engineering undergraduate
at Institute of Engineering & Technology in Lucknow, India. This mail is an
expression of my interest for Google Summer of Code program of this year. I
want to apologize for getting in so late. Actually I would have contacted
earlier, but sudden demise of my Grandfather disabled me in doing so.

I am interested in woking with your organisation for two reasons. *First*,
I believe that Xapian provides an incredibly useful tool for any developer
to avail the advantages of a Search Engine. Had I known of its existence
earlier I would have used it extensively in a project regarding web
scraping in which I had scraped the products' information from various
e-commerce websites in a database enabling the users to search for their
required product and compare its price across different sites (just like
any well known price comparator website). *Second*, I am fascinated by
Information Retrieval and I am eager to explore its computational side.
Specifically, I am interested in updating and complementing the current
library to further improve its functionalities.


The project I would like to work on in the summers is *Weighting Schemes*.
Till now, I have successfully build Xapian on my PC along with reading the
complete guide given on the following link:
http://getting-started-with-xapian.readthedocs.io/en/latest/index.html and
worked on *A practical example *given in the guide, in which, although I
faced some problems regarding the *shared library* *xapian-delve-1.5* but
solved them eventually after some googling and going through some of the
related messages in IRC Archives.

Moreover, I have gone through the code base particularly *xapian-core* and
studied *Xapian::Weight* class thoroughly and looked at the implementations
of some of the already defined weighting schemes.

Currently, I've started to look at the ticket https://trac.xapian.
org/ticket/744 and trying to devise a way so that *get_sumpart() *method in
every weight subclass does not need to be updated after the merging. Also,
I am going through *xapian-api *to get more essence of the code base.


The project ideas which I would like to propose and get some feedback on
are:

*1)* Currently, Xapian supports the weighting schemes which rely on
bag-of-word representation of documents assuming that each term in the
document is independent of each other which however is a debatable topic as
in some places word order and word dependence do matter for eg- *Mary is
quicker than John *and* John is quicker than Mary* are different.
I want to implement *Graph-of-word* representation in Xapian which is a
solution to such cases as it considers the relationship order between the
terms in a document using an unweighted directed graph of terms. This
representation can be further used to define a new weighting scheme,
*TW-IDF* (TW = Term Weight , IDF = Inverse Document Frequency) which
*significantly
outperforms* *TF-IDF *&* BM25* and in some cases its extension *BM25+* on
various standard TREC datasets. This effectiveness is not achieved at the
cost of its efficiency. It is confirmed by various experiments shown
in [2].

The papers which I have referred for the above are :-
[1] https://www.researchgate.net/publication/220479875_Graph
-based_term_weighting_for_information_retrieval
[2] https://pdfs.semanticscholar.org/8eac/d0f01ab0f53706561d
da0ce8d1f96544a348.pdf


*2)* There is another Weighting Scheme I would like to implement in Xapian,
*TF-ATO* (Term Frequency - Average Term Occurrences) mentioned in
http://eprints.nottingham.ac.uk/31329/1/dls_ukci2014.pdf with a
discriminative approach which uses* document centroid* as a *threshold* for
normalising documents in document collection. This document centroid is
used to remove less significant weights in the documents and helps to
achieve higher retrieval effectiveness. The average improving rate in
*precision* (non-interpolated average precision) between TF-IDF and TF-ATO
is *40-50 %.* This scheme with some extra work, also performs effectively
for dynamic data collection.


It would be great to have your opinions on these project ideas as it will
help me to come up with a proposal on how to implement them. I realise that
it's quite late but I will start working on my proposal asap so that I can
improve by getting your feedback on the same.

I'm sorry if this mail gets too long. Thank you so much for your time.


Shivang Bansal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20170316/f6ddd708/attachment.html>