notmuch: Xapian exception during database creation

[David Edmondson]

On Friday, 2017-12-29 at 22:23:01 UTC, Olly Betts wrote:

>> After the failure the database appears to be okay:
>> 
>> > agrajag-testing ~/s/notmuch % xapian-check ~/Maildir/.notmuch/xapian
>> > ...
>> > position table structure checked OK
>
> This seems to be for an almost empty database (2 items in the postlist
> table and nothing anywhere else) which doesn't really seem consistent
> with the amount of data notmuch reports as having processed.  Are you
> setting XAPIAN_FLUSH_THRESHOLD very high?

No, I didn't set any specific value.

> You can look at the low level entries in the postlist table with:
>
> xapian-inspect ~/Maildir/.notmuch/xapian/postlist.glass
>
> (You'll need to build xapian-core from source to get xapian-inspect,
> as it's really a tool for developers).
>
> I'd guess the two entries are user metadata entries notmuch has added
> (keys starting \x00\xc0) - otherwise I'd expect entries in other tables
> too.
>
>> Unfortunately I can't share the dataset, as it contains mail belonging
>> to my employer :-(
>> 
>> Any suggestions on how to go about debugging this?
>
> Building xapian-core configured with --enable-assertions is worth
> trying, as it might catch the problem earlier.

I'll try these things, thanks.

> You could also try running it under valgrind, though that may take some
> time if it normally takes 1h21m.

valgrind complained about a couple of things in debugger.cc, but they
seem both innocuous and unrelated.

> If that doesn't help, either we need to find a way to reproduce this
> which isn't using sensitive data, or add more consistency checks to the
> code to try to pin down what's going on.

David Bremner had a dataset that produced this same failure, I
believe. I don't know the origin or status of that dataset - David?

Just for completeness, I forced notmuch to build a chert database and
the initial scan ran through to completion without any problems.

dme.
-- 
I'm not the reason you're looking for redemption.