<div dir="ltr"><div><div>hi, guys:<br><br></div>I think norm(t, d) in Lucene can used to caculate the number which is similar to doc length(see norm(t,d) in <a href="http://lucene.apache.org/core/3_5_0/api/all/org/apache/lucene/search/Similarity.html#formula_norm">http://lucene.apache.org/core/3_5_0/api/all/org/apache/lucene/search/Similarity.html#formula_norm</a> ).<br>
<br>And this feature is applied into this pull request(<a href="https://github.com/xapian/xapian/pull/25">https://github.com/xapian/xapian/pull/25</a>). Here's the informations about new features and prerformance test:<br>
<br><div class="">
<div class="">
<p>This is a patch of Lucene 3.6.2 backend, it is just
support Lucene3.6.2, and not fully tested, I send this patch for
wandering if it works for the idea <a href="http://trac.xapian.org/wiki/ProjectIdeas#BackendforLuceneformatindexes">http://trac.xapian.org/wiki/ProjectIdeas#BackendforLuceneformatindexes</a>.<br>
until now, fewer features supported, includes:<br>
1. Single term search.<br>
2. 'AND' search supported, but performance needed to be optimize.<br>
3. Multiple segments.<br>
4. Doc length. Using .nrm instead.</p>
<p>Additonally:<br>
1. xxx_lower_bound, xxx_upper_bound, total doc length are not supported.
These datas are not exsits in Lucene backend, I'v used constant to
instead, so the search results may be not good.<br>
2. Compound file is not suppoted. so Compound file must be disable where doing index.</p>
<p>I've built a performance test of 1,000,000 documents(actually, I've
download a single file from wiki, which include 1,000,000 lines, I'v
treat one line as a document) from wiki. When doing single term seach,
performance of Lucene backend is as fast as xapian Chert. <br>
Test environment, OS: Vitual machine Ubuntu, CPU: 1 core, MEM: 800M.<br>
242 terms, doing single term seach per term, cacultes the total time
used for these 242 searches(results are fluctuant, so I give 10 results
per backend):<br>
1. backend Lucene<br>
1540ms, 1587ms, 1516ms, 1706ms, 1690ms, 1597ms, 1376ms, 1570ms, 1218ms, 1551ms<br>
2. backend Chert<br>
1286ms, 1626ms, 1575ms, 1771ms, 1661ms, 1662ms, 1808ms, 1341ms, 1688ms, 1809ms</p>
<p>Code for testing is quest.cc, you can look this file for details.</p>
<p>Code for Lucene indexing like this(And Xapian indexing used example/simpleindex.cc):</p>
<pre><code> IndexWriter indexWriter = new IndexWriter(directory, new EnglishAnalyzer(Version.LUCENE_36),
IndexWriter.MaxFieldLength.UNLIMITED);
indexWriter.setUseCompoundFile(false); //CompoundFile must be disable
int lineId = 0;
while (br.ready()) { //read lines from input file, each line as a document
lineId++;
String origLine = br.readLine();
origLine = origLine.trim();
Document doc = new Document();
doc.add(new Field("data", origLine, Field.Store.YES, Field.Index.ANALYZED));
doc.add(new Field("dataorigin", origLine, Field.Store.YES,
Field.Index.NOT_ANALYZED));
doc.add(new Field("lid", String.valueOf(lineId), Field.Store.YES,
Field.Index.NOT_ANALYZED));
indexWriter.addDocument(doc);
}
</code></pre>
</div>
</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/17 Richard Boulton <span dir="ltr"><<a href="mailto:richard@tartarus.org" target="_blank">richard@tartarus.org</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ah, a quick follow-on from that: read <a href="http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/Similarity.html" target="_blank">http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/Similarity.html</a><div>
<br>There's a per-document "norm" which can be stored, which BM25Similarity uses to store the document length. Additional factors can be stored in DocValuesFields (which are very similar to document values in Xapian, in that they're stored in separate sequences, though are a bit more flexible).</div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On 17 June 2013 16:06, Richard Boulton <span dir="ltr"><<a href="mailto:richard@tartarus.org" target="_blank">richard@tartarus.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You might want to look at how Lucene has implemented document length lookup for the BM25Similarity class (added in Lucene 4.0):<div>
<br></div><div><a href="http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/BM25Similarity.html" target="_blank">http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/BM25Similarity.html</a><br>
</div><div><br></div><div>I assumed they're using a document payload for storing the lengths, but haven't looked into it.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>
On 17 June 2013 14:28, jiangwen jiang <span dir="ltr"><<a href="mailto:jiangwen127@gmail.com" target="_blank">jiangwen127@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote><u><i>Or do you mean that it's one number per document whereas the other stats</i><br>
<i>are per database, so it's harder to store it?</i></u><div>
<br></div></div><div>yes, I mean this. It's a huge data. If a new doclength list(contains all the doclength in a list, like chert)</div><div>is added by myself, I am concern about:</div><div>1. This doclength list may be the bottlenect in this backend, <a href="http://trac.xapian.org/ticket/326" target="_blank">http://trac.xapian.org/ticket/326</a></div>
<div>2. Change too much above Lucene file format, then it's hard to compare performance between Xapian and Lucene</div><div><br></div><div>Some ideas:</div><div>1. Using rank algorithm without doclength, such as BM25Weight or TradWeight without doclength, or tfidfWeight.</div>
<div> If ranking results will be not good without doclength?</div><div>2. Stores doclength in .prx payload when doing Lucene indexing. </div><div> <a href="https://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/Payload.html" target="_blank">https://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/Payload.html</a></div>
<div> <a href="http://searchhub.org/2009/08/05/getting-started-with-payloads/" target="_blank">http://searchhub.org/2009/08/05/getting-started-with-payloads/</a></div><div> But this method has obvious drawback, it's not for general Lucene index data, if doclength is not stored, this method</div>
<div> doesn't works</div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
</blockquote></div>Any suggestions?</div><div><br></div><div>Regards</div>
<br></div></div><div>_______________________________________________<br>
Xapian-devel mailing list<br>
<a href="mailto:Xapian-devel@lists.xapian.org" target="_blank">Xapian-devel@lists.xapian.org</a><br>
<a href="http://lists.xapian.org/mailman/listinfo/xapian-devel" target="_blank">http://lists.xapian.org/mailman/listinfo/xapian-devel</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>