Hi all,<div><br></div><div>I have jotted down a plan for how to handle or add Bi-gram in Query Object though QueryParser.</div><div><br></div><div>PFA as a sequence diagram which depicts what i got to know about how parser works and query is build from tokens provided by the lexer.I have highlighted some area in blue where i think there is possibility of having bi-grams.While Integrating bi-gram in the Parser ,Query our aim is to generate and add bi-grams for all the consecutive terms to the query.</div>
<div><div><br></div><div>Following are categories sent to Parser from Lexer to Form Query Object:</div><div><br></div><div><b>Near - </b>2 or more terms with near in between.It is a type of query these two term are in window of 10 words.Since we are seeking these two words in vicinity of 10 Words window.It wont hurt to have them as bi-grams as we are seeking them in 10 words window so having them next is better.<b>(Bigram can be added)</b></div>
<div><br></div><div><b>Example:</b></div><div><b><br></b></div><div>Failed NEAR Assertion</div><div><br></div><div><b>Currently parser output.</b></div><div><br></div><div>Query((failed@1 NEAR 11 assertion@2))</div><div><br>
</div><div><b>Output With Bigram:</b></div><div><b><br></b></div><div>Query((failed@1 NEAR 11 assertion@2) OR failed assertion@3)</div><div><br></div><div><b>Implementation:</b></div><div><br></div><div>Since the all terms detected as near is added to class <b>Terms</b> so when we ask for Queries from the Class <b>Terms </b>using as_near_query , as_adj_query,as_opwindow_query then while parsing terms we can just add the bigrams while iterating list of term.</div>
<div><br></div><div><br></div><div><b>Adj: </b>exactly similar to <b>NEAR(Bigram can be added)</b></div><div><br></div><div><b>phrase : </b>Terms giving in a Quotes.Since they are terms user want to have together.Bigram can be added<b>(Bigram can be added)</b></div>
<div>Implementation is similar to Near,adj.</div><div><b><br></b></div><div><b>Phrased: </b>Single term which is actually two or more term linked with punctuation.These terms can be treated as bi-grams as they are terms which must exist together.<b>(Bigram can be added)</b></div>
<div>Implementation is similar to Near,adj.</div><div><br></div><div><b>Group: </b>A group of term separated only by white-spaces.<b>(Bigram can be added)</b></div><div><div><b><br></b></div><div><b>Implementation:</b></div>
<div><br></div><div>Since the all terms detected as group are added to class <b>TermGroup</b> so when we ask for Query from the Class <b>TermGroup </b>using as_group_query then while parsing terms we can just add the bigrams while iterating list of terms.</div>
</div><div><br></div><div><b>Wild:</b></div><div><b>Partial:</b></div><div><b>Synonym:</b></div><div><b></b>This is expanding which follow the pattern,synonym of term.It will pull out lot of similar terms and form a query with all those words.So considering this for bi-gram doesn't seem important.Please suggest if you feel it should be included.</div>
<div><b><br></b></div><div><b>BRA-KET: </b>These are bracketed expression.Currently the grammar have rule <b>BRA expr(E) KET</b> .so if there will be any scope of bi-grams in query inside BRA-KET it would have been consider while working on internal expression.</div>
<div><b><br></b></div><div><b>ValueRange: </b>No relation with Bi-grams.</div><div><b><br></b></div><div><b>Love: </b></div><div><b>Hate: </b></div><div>Since we are trying to avoid query or have a single term.We can restrain adding them as bi-grams.</div>
<div><b><br></b></div><div><b>bool_operator:</b><a href="https://github.com/sehaj-sk/xapian/blob/mybranch/xapian-core/docs/queryparser_new.rst#boolean-query">https://github.com/sehaj-sk/xapian/blob/mybranch/xapian-core/docs/queryparser_new.rst#boolean-query</a></div>
<div><b><br></b></div><div>Boolean operator are done by following type of grammer rules. bool_arg(E) bool_operator bool_arg(P).</div><div><br></div><div>since <b>bool_arg</b> are <b>expr </b>i.e they are Query Object hence getting Bigrams would be difficult.Please suggest something.</div>
<div><br></div><div>Example.</div><div><br></div><div>assertion OR failed</div><div><br></div><div><b>Current parsed Query:</b></div><div><b><br></b></div><div>Query((Zfail@1 OR Zassert@2))</div><div><b><br></b></div><div>
<b>Since Terms are converted to Query object how to make bigrams for these simple OR operator of terms.</b></div><div><b><br></b></div><div>Major work of handling the bi-grams will be taken care of by adding bi-grams to the terms while iterating terms in <b>TermGroup </b>and <b>Terms </b>class.</div>
<div><br></div><div>Please guide me and provide feedback about how to adding bi-grams in Query Object.</div><div><br></div><div>Thanks,</div><div><br></div><div>Gaurav Arora</div>
</div>