From arpitjain099 at gmail.com  Mon Aug 10 20:58:05 2026
From: arpitjain099 at gmail.com (Arpit Jain)
Date: Tue, 11 Aug 2026 04:58:05 +0900
Subject: MSet::snippet() returns unescaped text when hi_start and hi_end are
 both empty
Message-ID: <CALUkFOM3sBUjFR4VN7V0jrqFFCSv9SweCB18E8P3TmWJyHDvcw@mail.gmail.com>

Hi Xapian developers,

I think Xapian::MSet::snippet() in xapian-core breaks its documented
HTML-escaping guarantee on one fast path: when hi_start and hi_end are both
empty and the text already fits inside length, it returns the caller's text
byte for byte with no escaping. I built and ran a proof of concept against
the released 2.0.0 library; the output is below.

Summary
-------
All of the escaping in snippet generation lives in append_escaping_xml(),
which is only reached from SnipPipe::drain(). The branch at the top of
MSet::Internal::snippet() returns before drain() ever runs, so on that path
nothing is escaped, even though the header docs promise unconditionally
that the returned text is safe for HTML. The result is inconsistent in a
way that is easy to miss: the same call escapes correctly as soon as the
text is longer than length, so short documents come back raw and long ones
do not.

Where it is
-----------
xapian-core/queryparser/termgenerator_internal.cc:836-839, at tag v2.0.0
and on git master:

if (hi_start.empty() && hi_end.empty() && text.size() <= length) {
// Too easy!
return string{text};
}

The guarantee it contradicts is at xapian-core/include/xapian/mset.h:404,
in the doc comment for MSet::snippet(): "The returned text is escaped to
make it suitable for use in HTML (though beware that in upstream releases
1.4.5 and earlier this escaping was sometimes incomplete)". There is no
exception there for empty markers.

What happens
------------
An application that indexes untrusted content and renders results with
snippet() using empty hi_start and hi_end, trusting the documented
escaping, emits the attacker's markup into the page for every document
short enough to hit the branch. That is stored XSS in the embedding
application, and the attacker needs nothing beyond the normal
content-submission path.

To be straight about the limits: this is a library contract violation
rather than a hole in Xapian itself, it only bites callers who pass empty
markers and skip their own escaping, and I have not gone hunting for a
specific downstream that does that. Omega is unaffected because it uses
non-empty markers. What makes it worth fixing anyway is that the
documentation tells callers they do not need to escape, and the escaping
CVE-2018-0499 added in 2018 sits below the branch that skips it.

Proof of concept
----------------
Against xapian-core 2.0.0 (Homebrew bottle, arm64 macOS). The driver uses
the public API only: index one document, run a real query through
QueryParser and Enquire, then call snippet() three ways on the same MSet.
Includes and the main() wrapper elided:

Xapian::WritableDatabase db("/tmp/xsnip/db",
Xapian::DB_CREATE_OR_OVERWRITE);
Xapian::Stem stemmer("english");
Xapian::TermGenerator tg;
tg.set_stemmer(stemmer);
Xapian::Document doc;
tg.set_document(doc);
std::string text = "<script>alert(1)</script> hello world";
tg.index_text(text);
doc.set_data(text);
db.add_document(doc);
db.commit();

Xapian::Enquire enq(db);
Xapian::QueryParser qp;
qp.set_stemmer(stemmer);
qp.set_stemming_strategy(Xapian::QueryParser::STEM_SOME);
enq.set_query(qp.parse_query("hello"));
Xapian::MSet mset = enq.get_mset(0, 10);
unsigned flags = Xapian::MSet::SNIPPET_BACKGROUND_MODEL |
Xapian::MSet::SNIPPET_EXHAUSTIVE;

std::cout << "version : " << Xapian::version_string() << "\n";
std::cout << "input : " << text << "\n";
std::cout << "default markers : " << mset.snippet(text, 500, stemmer) <<
"\n";
std::cout << "empty markers : " << mset.snippet(text, 500, stemmer, flags,
"", "") << "\n";
std::cout << "empty, len=20 : " << mset.snippet(text, 20, stemmer, flags,
"", "") << "\n";

Built with g++ -std=c++17 t.cc -o t $(xapian-config --cxxflags --libs),
then run. Observed output:

version : 2.0.0
input : <script>alert(1)</script> hello world
default markers : &lt;script&gt;alert(1)&lt;/script&gt; <b>hello</b> world
empty markers : <script>alert(1)</script> hello world
empty, len=20 : ...hello world

Line 3 is the guarded path and escapes as documented. Line 4 is the same
MSet, same text, same flags, empty markers, and the script tag comes back
raw. Line 5 is that same empty-marker call with length cut to 20 so the
text no longer fits, which skips the branch and escapes again.

Suggested fix
-------------
Either drop the fast path so everything goes through SnipPipe::drain(), or
keep it and run append_escaping_xml() over text into the return string
instead of returning string{text}, which preserves the performance win. If
the intent really is that empty markers mean "give me the raw text", I
would say so in the mset.h comment, but I would still argue against leaving
the code as it stands, because the current rule is not "empty markers mean
raw", it is "empty markers mean raw, but only when the text is short
enough", and no caller can reason about that.

Why I don't think it's a duplicate
----------------------------------
The escaping was added by commit c1986aff, "Add missing XML escaping in
MSet::snippet()", whose message is "We were escaping in some cases, but not
all". That is the CVE-2018-0499 fix. It introduced append_escaping_xml(),
applied it at three sites all inside SnipPipe::drain(), and added a test in
tests/api_snippets.cc. It did not touch MSet::Internal::snippet(). The
empty-marker branch returns before drain() is ever constructed, so the 2018
fix could not have covered it, and the run above shows it still returns raw
text on 2.0.0. This is the leftover "some cases, but not all" case, not a
rediscovery of the fixed ones.

Severity and classification (my read, your call)
------------------------------------------------
Medium. CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:L/I:L/A:N. CWE-116, with CWE-79
as the downstream effect. AC is High because the impact depends on a caller
configuration the attacker does not control, and scope is Changed because
the consequence lands in the embedding application's browser context rather
than in Xapian.

Affected versions
-----------------
Verified by reading the source at v1.4.6, v1.4.31, v1.5.2 and v2.0.0, and
on git master; the line is at termgenerator_internal.cc:762-765 in v1.4.31.
The proof of concept was executed against the released 2.0.0 library. I
have not pinned down the first release containing the branch, only that it
predates the 2018 escaping fix.

Tooling
-------
I found this by starting from CVE-2018-0499, reading what its fix commit
actually changed, then looking for paths in snippet() that return without
going through drain(). I used AI assistance while investigating, but I
wrote and compiled the proof of concept myself and the output above is what
it printed against xapian-core 2.0.0, so this is empirically verified
rather than inferred from source. If you publish an advisory or request a
CVE for this, my GitHub handle is arpitjain099. Happy to send a patch for
whichever fix shape you prefer.

Thanks,
Arpit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20260811/95f6beec/attachment.htm>

From arpitjain099 at gmail.com  Tue Aug 11 11:47:04 2026
From: arpitjain099 at gmail.com (Arpit Jain)
Date: Tue, 11 Aug 2026 19:47:04 +0900
Subject: MSet::snippet() returns unescaped text when hi_start and hi_end are
 both empty
Message-ID: <CALUkFOOLtO=nAry7XbS++Duohw_5KAdhx-uGmOvjXPBFTwBAFw@mail.gmail.com>

Hi Xapian developers,

Since Xapian publishes no private security contact, I am writing to the
list rather than putting this anywhere more public; my name is Arpit Jain
and I work on open-source supply-chain security.

I think Xapian::MSet::snippet() in xapian-core breaks its documented
HTML-escaping guarantee on one fast path: when hi_start and hi_end are both
empty and the text already fits inside length, it returns the caller's text
byte for byte with no escaping. I built and ran a proof of concept against
the released 2.0.0 library; the output is below.

Summary
-------
All of the escaping in snippet generation lives in append_escaping_xml(),
which is only reached from SnipPipe::drain(). The branch at the top of
MSet::Internal::snippet() returns before drain() ever runs, so on that path
nothing is escaped, even though the header docs promise unconditionally
that the returned text is safe for HTML. The result is inconsistent in a
way that is easy to miss: the same call escapes correctly as soon as the
text is longer than length, so short documents come back raw and long ones
do not.

Where it is
-----------
xapian-core/queryparser/termgenerator_internal.cc:836-839, at tag v2.0.0
and on git master:

if (hi_start.empty() && hi_end.empty() && text.size() <= length) {
// Too easy!
return string{text};
}

The guarantee it contradicts is at xapian-core/include/xapian/mset.h:404,
in the doc comment for MSet::snippet(): "The returned text is escaped to
make it suitable for use in HTML (though beware that in upstream releases
1.4.5 and earlier this escaping was sometimes incomplete)". There is no
exception there for empty markers.

What happens
------------
An application that indexes untrusted content and renders results with
snippet() using empty hi_start and hi_end, trusting the documented
escaping, emits the attacker's markup into the page for every document
short enough to hit the branch. That is stored XSS in the embedding
application, and the attacker needs nothing beyond the normal
content-submission path.

To be straight about the limits: this is a library contract violation
rather than a hole in Xapian itself, it only bites callers who pass empty
markers and skip their own escaping, and I have not gone hunting for a
specific downstream that does that. Omega is unaffected because it uses
non-empty markers. What makes it worth fixing anyway is that the
documentation tells callers they do not need to escape, and the escaping
CVE-2018-0499 added in 2018 sits below the branch that skips it.

Proof of concept
----------------
Against xapian-core 2.0.0 (Homebrew bottle, arm64 macOS). The driver uses
the public API only: index one document, run a real query through
QueryParser and Enquire, then call snippet() three ways on the same MSet.
Includes and the main() wrapper elided:

Xapian::WritableDatabase db("/tmp/xsnip/db",
Xapian::DB_CREATE_OR_OVERWRITE);
Xapian::Stem stemmer("english");
Xapian::TermGenerator tg;
tg.set_stemmer(stemmer);
Xapian::Document doc;
tg.set_document(doc);
std::string text = "<script>alert(1)</script> hello world";
tg.index_text(text);
doc.set_data(text);
db.add_document(doc);
db.commit();

Xapian::Enquire enq(db);
Xapian::QueryParser qp;
qp.set_stemmer(stemmer);
qp.set_stemming_strategy(Xapian::QueryParser::STEM_SOME);
enq.set_query(qp.parse_query("hello"));
Xapian::MSet mset = enq.get_mset(0, 10);
unsigned flags = Xapian::MSet::SNIPPET_BACKGROUND_MODEL |
Xapian::MSet::SNIPPET_EXHAUSTIVE;

std::cout << "version : " << Xapian::version_string() << "\n";
std::cout << "input : " << text << "\n";
std::cout << "default markers : " << mset.snippet(text, 500, stemmer) <<
"\n";
std::cout << "empty markers : " << mset.snippet(text, 500, stemmer, flags,
"", "") << "\n";
std::cout << "empty, len=20 : " << mset.snippet(text, 20, stemmer, flags,
"", "") << "\n";

Built with g++ -std=c++17 t.cc -o t $(xapian-config --cxxflags --libs),
then run. Observed output:

version : 2.0.0
input : <script>alert(1)</script> hello world
default markers : &lt;script&gt;alert(1)&lt;/script&gt; <b>hello</b> world
empty markers : <script>alert(1)</script> hello world
empty, len=20 : ...hello world

Line 3 is the guarded path and escapes as documented. Line 4 is the same
MSet, same text, same flags, empty markers, and the script tag comes back
raw. Line 5 is that same empty-marker call with length cut to 20 so the
text no longer fits, which skips the branch and escapes again.

Suggested fix
-------------
Either drop the fast path so everything goes through SnipPipe::drain(), or
keep it and run append_escaping_xml() over text into the return string
instead of returning string{text}, which preserves the performance win. If
the intent really is that empty markers mean "give me the raw text", I
would say so in the mset.h comment, but I would still argue against leaving
the code as it stands, because the current rule is not "empty markers mean
raw", it is "empty markers mean raw, but only when the text is short
enough", and no caller can reason about that.

Why I don't think it's a duplicate
----------------------------------
The escaping was added by commit c1986aff, "Add missing XML escaping in
MSet::snippet()", whose message is "We were escaping in some cases, but not
all". That is the CVE-2018-0499 fix. It introduced append_escaping_xml(),
applied it at three sites all inside SnipPipe::drain(), and added a test in
tests/api_snippets.cc. It did not touch MSet::Internal::snippet(). The
empty-marker branch returns before drain() is ever constructed, so the 2018
fix could not have covered it, and the run above shows it still returns raw
text on 2.0.0. This is the leftover "some cases, but not all" case, not a
rediscovery of the fixed ones.

Severity and classification (my read, your call)
------------------------------------------------
Medium. CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:L/I:L/A:N. CWE-116, with CWE-79
as the downstream effect. AC is High because the impact depends on a caller
configuration the attacker does not control, and scope is Changed because
the consequence lands in the embedding application's browser context rather
than in Xapian.

Affected versions
-----------------
Verified by reading the source at v1.4.6, v1.4.31, v1.5.2 and v2.0.0, and
on git master; the line is at termgenerator_internal.cc:762-765 in v1.4.31.
The proof of concept was executed against the released 2.0.0 library. I
have not pinned down the first release containing the branch, only that it
predates the 2018 escaping fix.

Tooling
-------
I found this by starting from CVE-2018-0499, reading what its fix commit
actually changed, then looking for paths in snippet() that return without
going through drain(). I used AI assistance while investigating, but I
wrote and compiled the proof of concept myself and the output above is what
it printed against xapian-core 2.0.0, so this is empirically verified
rather than inferred from source. If you publish an advisory or request a
CVE for this, my GitHub handle is arpitjain099. Happy to send a patch for
whichever fix shape you prefer.

Thanks,
Arpit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20260811/1cd039e1/attachment-0001.htm>

From olly at survex.com  Wed Aug 12 00:51:40 2026
From: olly at survex.com (Olly Betts)
Date: Wed, 12 Aug 2026 00:51:40 +0100
Subject: MSet::snippet() returns unescaped text when hi_start and hi_end
 are both empty
In-Reply-To: <CALUkFOOLtO=nAry7XbS++Duohw_5KAdhx-uGmOvjXPBFTwBAFw@mail.gmail.com>
References: <CALUkFOOLtO=nAry7XbS++Duohw_5KAdhx-uGmOvjXPBFTwBAFw@mail.gmail.com>
Message-ID: <anu1jEVLVcY1+hgx@survex.com>

On Tue, Aug 11, 2026 at 07:47:04PM +0900, Arpit Jain wrote:
> Since Xapian publishes no private security contact, I am writing to the
> list rather than putting this anywhere more public; my name is Arpit Jain
> and I work on open-source supply-chain security.

FWIW, this list has multiple external public archives.  Probably our bug
tracker is actually less public than this list.

> I think Xapian::MSet::snippet() in xapian-core breaks its documented
> HTML-escaping guarantee on one fast path: when hi_start and hi_end are both
> empty and the text already fits inside length, it returns the caller's text
> byte for byte with no escaping.

Indeed - thanks for reporting.  As you identified, it's effectively a
missed case from CVE-2018-0499.

I've pushed fixes to main and RELEASE/1.4.  I'm already working on new
releases so this should hopefully be in releases soon.

> Severity and classification (my read, your call)
> ------------------------------------------------
> Medium. CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:L/I:L/A:N. CWE-116, with CWE-79
> as the downstream effect. AC is High because the impact depends on a caller
> configuration the attacker does not control, and scope is Changed because
> the consequence lands in the embedding application's browser context rather
> than in Xapian.

Sorry, I don't know what any of that means!

I'd expect that it's uncommon to pass empty hi_start and hi_end,
especially when used in a web context (since end users will expect
search terms to be highlighted in snippets because that's (a) helpful
and (b) what pretty much every search system does).

I used Debian codesearch to find code in software packaged by Debian
which uses MSet::snippet() (unfortunately there are a lot of unrelated
matches as well):

https://codesearch.debian.net/search?q=(%5C.%7C-%3E)%5Cb*snippet%5Cb*%5C(%5B%5E')0-9%5D&literal=0

All the calls I found use non-empty hi_start and hi_end.  That's likely
a subset of users of this API but at least indicative.

Cheers,
    Olly


From arpitjain099 at gmail.com  Wed Aug 12 06:02:20 2026
From: arpitjain099 at gmail.com (Arpit Jain)
Date: Wed, 12 Aug 2026 14:02:20 +0900
Subject: MSet::snippet() returns unescaped text when hi_start and hi_end
 are both empty
In-Reply-To: <anu1jEVLVcY1+hgx@survex.com>
References: <CALUkFOOLtO=nAry7XbS++Duohw_5KAdhx-uGmOvjXPBFTwBAFw@mail.gmail.com>
 <anu1jEVLVcY1+hgx@survex.com>
Message-ID: <CALUkFON+FP3sMXzH+a526Y4L+n21WrEAugs7n3vYLWnpzOH9xg@mail.gmail.com>

Thanks for the quick fix, and for the correction about the list. I had that
backwards: I treated it as the private option because there was no security
contact, when it has public archives and the bug tracker would have been
the less exposed route. I will use the tracker for Xapian in future.

On the severity block, I will drop it. Plain version: this only bites a
caller who passes empty hi_start and hi_end and then renders the snippet as
HTML, and your codesearch is good evidence that essentially nobody does the
first part. So the real-world exposure is low and I would not argue for
treating it as more than a correctness fix.

Thanks again,
Arpit

On Wed, Aug 12, 2026 at 8:51?AM Olly Betts <olly at survex.com> wrote:

> On Tue, Aug 11, 2026 at 07:47:04PM +0900, Arpit Jain wrote:
> > Since Xapian publishes no private security contact, I am writing to the
> > list rather than putting this anywhere more public; my name is Arpit Jain
> > and I work on open-source supply-chain security.
>
> FWIW, this list has multiple external public archives.  Probably our bug
> tracker is actually less public than this list.
>
> > I think Xapian::MSet::snippet() in xapian-core breaks its documented
> > HTML-escaping guarantee on one fast path: when hi_start and hi_end are
> both
> > empty and the text already fits inside length, it returns the caller's
> text
> > byte for byte with no escaping.
>
> Indeed - thanks for reporting.  As you identified, it's effectively a
> missed case from CVE-2018-0499.
>
> I've pushed fixes to main and RELEASE/1.4.  I'm already working on new
> releases so this should hopefully be in releases soon.
>
> > Severity and classification (my read, your call)
> > ------------------------------------------------
> > Medium. CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:L/I:L/A:N. CWE-116, with
> CWE-79
> > as the downstream effect. AC is High because the impact depends on a
> caller
> > configuration the attacker does not control, and scope is Changed because
> > the consequence lands in the embedding application's browser context
> rather
> > than in Xapian.
>
> Sorry, I don't know what any of that means!
>
> I'd expect that it's uncommon to pass empty hi_start and hi_end,
> especially when used in a web context (since end users will expect
> search terms to be highlighted in snippets because that's (a) helpful
> and (b) what pretty much every search system does).
>
> I used Debian codesearch to find code in software packaged by Debian
> which uses MSet::snippet() (unfortunately there are a lot of unrelated
> matches as well):
>
>
> https://codesearch.debian.net/search?q=(%5C.%7C-%3E)%5Cb*snippet%5Cb*%5C(%5B%5E')0-9%5D&literal=0
>
> All the calls I found use non-empty hi_start and hi_end.  That's likely
> a subset of users of this API but at least indicative.
>
> Cheers,
>     Olly
>


-- 
Thanks,
Arpit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20260812/7fd16687/attachment.htm>

From olly at survex.com  Thu Aug 13 00:53:16 2026
From: olly at survex.com (Olly Betts)
Date: Thu, 13 Aug 2026 00:53:16 +0100
Subject: MSet::snippet() returns unescaped text when hi_start and hi_end
 are both empty
In-Reply-To: <CALUkFON+FP3sMXzH+a526Y4L+n21WrEAugs7n3vYLWnpzOH9xg@mail.gmail.com>
References: <CALUkFOOLtO=nAry7XbS++Duohw_5KAdhx-uGmOvjXPBFTwBAFw@mail.gmail.com>
 <anu1jEVLVcY1+hgx@survex.com>
 <CALUkFON+FP3sMXzH+a526Y4L+n21WrEAugs7n3vYLWnpzOH9xg@mail.gmail.com>
Message-ID: <an0HbNpJUq8SRCC5@survex.com>

On Wed, Aug 12, 2026 at 02:02:20PM +0900, Arpit Jain wrote:
> Thanks for the quick fix, and for the correction about the list. I had that
> backwards: I treated it as the private option because there was no security
> contact, when it has public archives and the bug tracker would have been
> the less exposed route. I will use the tracker for Xapian in future.

I think it was a reasonable assumption.

Perhaps we should create a security contact email address, but we're
averaging a security fix every 8-9 years so an open address will
inevitably get far more spam, non-security-related messages, etc than
actual reports.  Even ignoring the developer time that wastes, it means
there's a significant risk of reports being missed amongst the noise.

Is there a good approach to this that other projects with very
infrequent security reports use?

> On the severity block, I will drop it. Plain version: this only bites a
> caller who passes empty hi_start and hi_end and then renders the snippet as
> HTML,

The text supplied to snippet() would also need to be under an attacker's
control (or they'd need to find existing content in the system which
happened to be short enough but contain something suitable).  Sometimes
it is (e.g. in a webmail frontend) but often it isn't so that further
reduces where this could be exploited.

> and your codesearch is good evidence that essentially nobody does the
> first part. So the real-world exposure is low and I would not argue for
> treating it as more than a correctness fix.

I agree real-world exposure is low, but it's hard to know how the API
gets used (and codesearch only shows us a subset), so for people
maintaining packages of xapian-core my recommendation would be to apply
the patch (or package a version with the fix) since it's a simple patch
with very low risk of unwanted side effects.

At least for the Debian package (which I happen to also maintain) I'm
intending to submit the patch for the stable release via the security
queue, which gets to more users sooner than a stable update would.

Cheers,
    Olly