[Xapian-discuss] 100% relevant

Wed Aug 18 12:04:10 BST 2004

> > I've asked Ben to run "make check" on the actual compiled directory
> > which the binary was installed from, to see if there's something funny
> > with that particular version.
> 
> Seems unlikely, but worth checking.

Ben's tried this, and it passes the tests fine.

> Hmm, actually there's an easy way to see if this happens from C++ too.
> Can you try querying the database with the quest utility from
> xapian-examples?
> 
> quest -d /path/to/database 'treatment OR centre'

Good idea, I'm doing this in the recipe, see below.

> > I can post up the script I'm using to create the database.  I've tried
> > with a clean one with only a small amount of data, and I'm getting the
> > same problem.
> 
> Probably the next place to look.

OK, I have a recipe which partly reproduces it, although I'm getting
different problems on different machines.

Find attached testindex.pl, which creates a Xapian database from a
MySQL table.  The table is a combined fragment of a real table, and
you can download it here:
    http://www.theyworkforyou.com/pwdata/scratch/twfy-xaptest-fragment.sql.bz2
Also attached is search.pl, a command line Xapian-querying test program.

The recipe is:
    1. Load the table into a mysql database
    2. Edit testindex.pl to point to that database
    3. Run testindex.pl, specifying a folder for the new Xapian database
    4. Then use delve and/or search.pl to query it.

Results I'm getting (see attached reproduce.txt for details):

baked (the theyworkforyou.com server):  delve reports
"Error opening database: Couldn't detect type of database" when run
with a query.  search.pl gives results, but reports 0 weights.
Perl v5.8.2, Search::Xapian 0.8.0.2

owl (my laptop): delve reports 
"Error opening database: Couldn't detect type of database" when run
with a query.  search.pl gives reasonable results, with variable
weights.
Perl v5.8.3, Search::Xapian 0.8.0.2

I reckon this is some kind of Perl/Xapian installation issue, and
suspect that some error reporting somewhere could be improved to make
it clearer what I'm doing wrong.

Questions:
1. Should I upgrade to Search::Xapian 0.8.0.4, which looks like the
latest version on CPAN?  Is there a version called 0.8.1 yet?
2. Why is delve saying "Couldn't detect type of database"?  Even
though it can still count the documents in it, and search.pl can
return some results from it?
3. Anything I can do to test the confiuration?  It's possible an older
version of Xapian is being loaded in, or a different Perl module to
the one I think.  But I don't know a nice way of checking this.

Francis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testindex.pl
Type: text/x-perl
Size: 3138 bytes
Desc: not available
Url : http://lists.tartarus.org/pipermail/xapian-discuss/attachments/20040818/7bb113b0/testindex.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: search.pl
Type: text/x-perl
Size: 813 bytes
Desc: not available
Url : http://lists.tartarus.org/pipermail/xapian-discuss/attachments/20040818/7bb113b0/search.bin
-------------- next part --------------
On baked (the theyworkforyou.com server):

$ rm -fr ~/scratch/testxapdb
$ ~/fawkes/search/index.pl ~/scratch/testxapdb lastmonth
xapian indexing wrans 2004-07-19
xapian indexing debate 2004-07-19
xapian indexing debate 2004-07-20
xapian indexing wrans 2004-07-20
xapian indexing wrans 2004-07-21
xapian indexing debate 2004-07-21
xapian indexing debate 2004-07-22
xapian indexing wrans 2004-07-22

exiting before extra checks
$ ~/src/xapian-examples-0.8.1/delve ~/scratch/testxapdb/ 'treatment OR centre'
Error opening database: Couldn't detect type of database
$ ~/src/xapian-examples-0.8.1/delve ~/scratch/testxapdb/
number of documents = 3621
average document length = 3.38729e+06
$ ~/fawkes/search/search.pl ~/scratch/testxapdb/ treatment centre
82
wrans/2004-07-19.3W.2 -> 100% 0w
wrans/2004-07-19.28W.8 -> 100% 0w
...

Notice that delve doesn't like the database at all this time, and that
search.pl is reporting 100% relevancy and zero weights (the 0w).

Now extracting data so can test it elsewhere.
mysql> create table tmpindextest select body, person_id, hdate, gid, major from epobject, hansard
             left join member on hansard.speaker_id = member.member_id
             where epobject.epobject_id = hansard.epobject_id and
             hdate > date_sub(curdate(), interval 1 month);
Modified index.pl to make simpler testindex.pl which uses tmpindextest.

Checking still goes wrong on baked:

$ rm -fr ~/scratch/testxapdb
$ ~/fawkes/search/testindex.pl ~/scratch/testxapdb lastmonth
xapian indexing wrans 2004-07-19
xapian indexing debate 2004-07-19
xapian indexing debate 2004-07-20
xapian indexing wrans 2004-07-20
xapian indexing wrans 2004-07-21
xapian indexing debate 2004-07-21
xapian indexing debate 2004-07-22
xapian indexing wrans 2004-07-22
$ ~/src/xapian-examples-0.8.1/delve ~/scratch/testxapdb/ 'treatment OR centre'
Error opening database: Couldn't detect type of database
$ ~/src/xapian-examples-0.8.1/delve ~/scratch/testxapdb/
number of documents = 3621
average document length = 3.375e+06
$ ~/fawkes/search/search.pl ~/scratch/testxapdb/ treatment centre
82
wrans/2004-07-19.3W.2 -> 100% 0w
wrans/2004-07-19.28W.8 -> 100% 0w
wrans/2004-07-19.28W.11 -> 100% 0w
wrans/2004-07-19.29W.5 -> 100% 0w
wrans/2004-07-19.32W.0 -> 100% 0w
...

Problem still happens.  Now dump the data and try on owl.
$ mysqldump -u fawkes -p fawkes tmpindextest > ~/scratch/twfy-xaptest-fragment.sql

On owl (my laptop):
$ mysql fawkes < twfy-xaptest-fragment.sql
$ rm -fr ~/scratch/testxapdb
$ ~/devel/fawkes/search/testindex.pl ~/scratch/testxapdb lastmonth
xapian indexing wrans 2004-07-19
xapian indexing debate 2004-07-19
xapian indexing debate 2004-07-20
xapian indexing wrans 2004-07-20
xapian indexing wrans 2004-07-21
xapian indexing debate 2004-07-21
xapian indexing debate 2004-07-22
xapian indexing wrans 2004-07-22

$ /usr/local/src/xapian-0.8.1/xapian-examples-0.8.1/delve ~/scratch/testxapdb/ 'treatment OR centre'
Error opening database: Couldn't detect type of database
$ /usr/local/src/xapian-0.8.1/xapian-examples-0.8.1/delve ~/scratch/testxapdb/
number of documents = 3621
average document length = 2.82431e+06
$ ~/devel/fawkes/search/search.pl ~/scratch/testxapdb/ treatment centre
5
debate/2004-07-22.525.0 -> 100% 17.6695338542631w
debate/2004-07-19.127.2 -> 99% 17.6695338124996w
debate/2004-07-19.51.0 -> 99% 17.6695337515089w
debate/2004-07-19.120.2 -> 99% 17.6695211748926w
debate/2004-07-22.562.0 -> 99% 17.6695119127936w