[Xapian-tickets] [Xapian] #156: Testsuite should allow tests to mark themselves as known failures

[Xapian]

#156: Testsuite should allow tests to mark themselves as known failures
------------------------+------------------------------
 Reporter:  richard     |             Owner:  olly
     Type:  defect      |            Status:  reopened
 Priority:  normal      |         Milestone:
Component:  Test Suite  |           Version:  SVN trunk
 Severity:  normal      |        Resolution:
 Keywords:              |        Blocked By:
 Blocking:              |  Operating System:  All
------------------------+------------------------------
Changes (by olly):

 * status:  closed => reopened
 * resolution:  wontfix =>


Old description:

> The idea of this is that if you find a bug, and have produced a test case
> to
> exhibit the bug, it is helpful to check that test case into SVN so that
> others
> can see the problem and work on it, but it is not desirable to break SVN
> HEAD.
> Therefore, it is helpful to be able to mark the test as a known failure,
> and
> commit it.
>
> The testsuite will report the known failures (in yellow, if colour output
> is
> supported), but still report a success status.  If a test which is marked
> as a
> known failure actually passes, the testsuite will report this as an
> error; this
> should ensure that the known failure marker is removed once the problem
> being
> tested is fixed.

New description:

 The idea of this is that if you find a bug, and have produced a test case
 to
 exhibit the bug, it is helpful to check that test case into SVN so that
 others
 can see the problem and work on it, but it is not desirable to break SVN
 HEAD.
 Therefore, it is helpful to be able to mark the test as a known failure,
 and
 commit it.

 The testsuite will report the known failures (in yellow, if colour output
 is
 supported), but still report a success status.  If a test which is marked
 as a
 known failure actually passes, the testsuite will report this as an error;
 this
 should ensure that the known failure marker is removed once the problem
 being
 tested is fixed.

--

Comment:

 I'm wondering if we should revisit this.

 Since the switch to git, the style of development is now more branch-
 based, and there's a lot to be said for being able to add a testcase on a
 branch as `XFAIL` in one commit, then in a separate commit (or commits)
 fix the bug and drop the `XFAIL`.  With CI, a reviewer can then see that
 the testcase was actually failing before the fix was applied (avoiding the
 easy pitfall of adding a regression test which actually passed before the
 fix too).

 > And that the testsuite change sometimes turns out not to be quite right
 anyway in my experience...

 With git you can rewrite history prior to submitting the changes for
 review, and fix-up the commit with the testsuite change.

 The patch looks plausible (though I haven't tried to apply it to current
 sources and I suspect it'll need a little adjustment after this much time
 - at least the numeric range fix needs dropping, as we fixed that years
 ago).

 I think it'd be nice to allow an informational message in `KNOWN_FAILURE`
 (e.g. something like `KNOWN_FAILURE("NumberVRP assumes numbers sort as
 their string values");` for the example bug in the patch).  The current
 `bool` flag could just be a `const char*` which defaulted to `NULL`,
 assuming a static message is sufficient.

--
Ticket URL: <https://trac.xapian.org/ticket/156#comment:4>
Xapian <https://xapian.org/>
Xapian