Ticket #6129 (closed Feature Requests: fixed)
Include warning counts in reports
|Reported by:||olaf.meeuwissen@…||Owned by:||rogeeff|
|Milestone:||To Be Determined||Component:||test|
I mentioned this on the mailing list before, but I would like to be able to report the number of times assertions in the BOOST_WARN "family" have failed.
For test cases, the (XML) report includes information similar to:
result="passed" assertions_passed="x" assertions_failed="x" expected_failures="x"
and for test suites
result="passed" assertions_passed="x" assertions_failed="x" expected_failures="x" test_cases_passed="x" test_cases_failed="x" test_cases_skipped="x" test_cases_aborted="x"
I guess I would like to see a count of something like assertions_ignored added to the test cases and an additional ignored value for their result attribute. This ignored value is used for test cases that fail one or more of their BOOST_WARN type assertions but pass all of their BOOST_REQUIRE and BOOST_CHECK assertions. If all assertions pass, the test case passes. If a BOOST_CHECK assertion fails the test case fails. If a BOOST_REQUIRE assertion fails, the test case is aborted and fails.
For the test suites, the same story goes for the assertions_ignored and ignored additions, where an ignored result is assigned if one or more test cases with that result were encountered and none of the test cases failed. If all test cases passed, the test suite passes. If one or more test cases failed, the test suite fails. In addition to the existing test_cases_* counts, I'd like to see a test_cases_ignored count.
Summarizing, you'd get something like this:
- counts of assertions passed, ignored or failed
- a value of passed, ignored or failed for the result attribute
- counts of test cases passed, ignored, failed, skipped or aborted
- counts of test suites passed, ignored, failed or skipped?
Please note the distinction between skipped (was not even started so no assertions have been evaluated) and ignored (ran to completion with one or more failed BOOST_WARN assertions).
AFAIK, test suites cannot be aborted. Can they?
In reviewing my ticket, I noticed that skipped and aborted should perhaps be added as test case result values as well. That way, the report includes everything you ever wanted to know about all test cases' and test suites' states.
Adding the output of failed assertions would be nice too, but that's outside the scope of this ticket.