Hey James, Hi ,Hope your doing fine :) Thanks for your detailed feedback . I'm really sorry for the shabby code,I had just begun to get acquainted with Xapian when I wrote this.In order to improve the example according to the points that you have mentioned,here's what I'd like to do :-<br>
<br>1.) I had actually thought of just ignoring the first word of every sentence but then realized that that would eliminate some genuine proper nouns as well.There are two ways I can use to find the proper nouns :-<br> a.) Use the Named Entity Recognition provided by the nltk NLP library of python.It will just directly pinpoint the proper nouns in the sentence to me.But then any one who wants to run the example,will have to install nltk as well as the associated corpora for it.However,it's performance is extremely good.<br>
<br> b.) I can just check to see if the first word of a sentence is a standard dictionary word or not (by using something like PyEnchant to speed up the process ) ,if it is,I wont consider is it as a proper noun.PyEnchant is relatively simple to install but this method will make mistakes depending on the words present in the standard dictionary because some dictionaries include a lot of proper nouns in them. <br>
<br>2.) I agree,I just read the code of the xapian.TermGenerator.index_text( ) and realized that it already does a lot of processing like tokenization,stemming etc. So what I'll now do is (because I only want to index a single word) first produce the unstemmed/stemmed form of the words (by directly using the Xapian::Stem object) depending on the stemming strategy which I'll now take from the command line (and then use TermGenerator.set_stemming_strategy as this will help my example be a good example for the various stemming strategies we provide),combine them with the prefix (which will again be selected by him ) and then instead of using index_text( ),Ill directly use the add_term(term,wdf_inc) function of my Document object.I don't need to use the Document.add_posting( ) function as our example does not need phrase searching.<br>
<br>3.)Ill work on the code to include support for sentences broken across lines.I just somehow didn't do this when I wrote the example.<br><br>The modified code will now show various aspects of Xapian such as stemming strategies,specifying the prefix for the terms,some methods of the Document object etc.<br>
<br>Please let me know what you think and thank you so much for your time :)<br><br>-Regards<br>-Aarsh<br><blockquote class="gmail_quote"></blockquote><br> <br><br><div class="gmail_quote">On Fri, Feb 8, 2013 at 3:11 AM, James Aylett <span dir="ltr"><<a href="mailto:james-xapian@tartarus.org" target="_blank">james-xapian@tartarus.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 27 Jan 2013, at 20:09, aarsh shah <<a href="mailto:aarshkshah1992@gmail.com">aarshkshah1992@gmail.com</a>> wrote:<br>
<br>
> Hey guys,I have added a python indexer example to the SampleCode page of our wiki.Please do have a look.The code can also be found here :-<br>
><br>
> <a href="https://github.com/aarshkshah1992/xapian/blob/efcf443527b74326119bbc0935fc41a002ce60db/xapian-bindings/python/docs/examples/simpleindexgrep.py/" target="_blank">https://github.com/aarshkshah1992/xapian/blob/efcf443527b74326119bbc0935fc41a002ce60db/xapian-bindings/python/docs/examples/simpleindexgrep.py/</a><br>
<br>
</div>Aarsh — what are you actually trying to do here? Because what your comments say you're doing isn't what the code does. Three problems:<br>
<br>
1) English uses capitals at the start of sentences, so you're actually just indexing more or less everything<br>
<br>
2) you're running xapian.TermGenerator.index_text() on single words, which isn't really what it's designed to do (it has its own word-splitting algorithm)<br>
<br>
3) you don't support sentences broken across lines, which doesn't match the majority of use cases — although you may have a particular one in mind<br>
<br>
Does what you're trying to do show how to use an aspect of Xapian that we don't already show in the existing examples? Or at least show it more clearly?<br>
<span class="HOEnZb"><font color="#888888"><br>
J<br>
<br>
--<br>
James Aylett, occasional trouble-maker<br>
<a href="http://xapian.org" target="_blank">xapian.org</a><br>
<br>
</font></span></blockquote></div><br>