[Xapian-tickets] [Xapian] #156: Testsuite should allow tests to mark themselves as known failures

[Xapian]

#156: Testsuite should allow tests to mark themselves as known failures
------------------------+------------------------------
 Reporter:  richard     |             Owner:  olly
     Type:  defect      |            Status:  reopened
 Priority:  normal      |         Milestone:
Component:  Test Suite  |           Version:  SVN trunk
 Severity:  normal      |        Resolution:
 Keywords:              |        Blocked By:
 Blocking:              |  Operating System:  All
------------------------+------------------------------

Comment (by olly):

 If I read the patch right, an testcase marked "XFAIL()" which passes
 results in the test run failing overall (which I think makes sense and
 seems to align with at least some other test harnesses which support
 "XFAIL").  Other harnesses seem to use "XPASS" for "test marked XFAIL()
 which passed", which seems like a convention we probably should follow.

 Failing on "XPASS" means "XFAIL()" isn't suitable for flagging a flaky
 test, but if we really want something for that, then a separate "FLAKY()"
 annotation of some sort would make more sense.  I'm not sure we want to go
 there (making flaky tests not flaky would be better), and if we do we
 should look at what other harnesses do before we just invent something
 different.  Somewhat related is that we already skip tests which use
 timing to determine `O()` behaviour if `$AUTOMATED_TESTING` is set - these
 can be a little bit flaky in the face of variable load (and that's pretty
 much inherent in what they do), but sufficiently rarely that it really
 only causes a problem for automated builds.  This skipping is automated by
 the timing machinery in the test harness.

--
Ticket URL: <https://trac.xapian.org/ticket/156#comment:6>
Xapian <https://xapian.org/>
Xapian