<div class="gmail_quote"><div><b style="font-family:'Times New Roman';font-size:medium"><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Hi olly,</span></b></div>
<div><b style="font-family:'Times New Roman';font-size:medium"><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"><br>
</span></b></div><div><b style="font-family:'Times New Roman';font-size:medium"><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">The following is my general idea for the project. For a complete query parser I still need to consider more details. Please give me feedback because the description of this project is lack of detailed information, and I can submit my proposal without giant deviation.</span></b></div>
<div><b style="font-family:'Times New Roman';font-size:medium"><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"><br></span></b></div><span><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">design principle of query parsing: </span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1) better understanding user input. All search engine do is understanding what human is trying to find and return the best matching results. </span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">2) balance between query time and precision. Users’ time is more valuable, they can’t wait 5 more minutes for a better result. So how to achieve the best result within certain amount of time is important.</span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"></span><br><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Here are my initial ideas of free text query parsing.</span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1) deal with mis-spelling. User maybe input mis-spelling words, thus our search engine should support tolerant retrieval. Search engine infers the correctness of a word based on the whether the word appears in dictionary and the number of returned documents. If the number of returned documents less than a preset value, it is more likely to be a mis-spelling word. Thus, we search for the potential correctly spelling words. We can first employ </span><span style="font-weight:bold;font-size:15px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">k-gram indexes</span><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"> to further limit the set of vocabulary terms, then compute the </span><span style="font-weight:bold;font-size:15px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">edit distance</span><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"> of strings. </span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">2) deal with synonyms. This can be done by index unnormalized tokens and maintain a query expansion list of multiple vocabulary entries to consider for a certain query term. A query term is then effectively a disjunction of several postings list. Also, an alternative is to do the expansion during index construction. </span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">3) deal with stop words. In modern search engines this is not a good idea, so I think we can just discard it.</span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">4) dividing the queries. User input might be long sentence and contains ‘stop words’. Search engine achieves better return result by doing multiple queries based on the dividing of the original query string. How to divide the query is the most difficult. Here’s some idea. </span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"> a. documents contain more consecutive matching words rank higher.</span><br><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"> b. different words have different weight (e.g., tf-idf), documents with bigger weight rank higher.</span><br>
<span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"> c. use k-gram to divide long query and limit the query times.</span><br><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"></span><br>
<font face="Arial"><span style="font-size:15px;white-space:pre-wrap"><br></span></font><span style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Thank you.</span></span><span class="HOEnZb"><font color="#888888"><br clear="all">
<div><br></div>-- <br>Weixian Zhou<br>Department of Computer Science and Engineering<br>University at Buffalo, SUNY<br><br>
</font></span></div><br><br clear="all"><div><br></div>-- <br>Weixian Zhou<br>Department of Computer Science and Engineering<br>University at Buffalo, SUNY<br><br>