Description
CI runs with any failed/expired tests should result in a fail state for the action as a whole.
This is so that we a) gain this information at a high-level and in future can trigger some automation from it (e.g, alerting a Slack channel or sending an email to stakeholders), and b) gain a quick heads-up that a human needs to go review the results file and determine the test fails/write up the required bug reports, etc.
Expected Change
The overview of each CI page tells us that a test IS a fail (by using the red X icons), such as:
However, that the fail message is not the generic "error code 1" message shown there; instead, it's an informative/automatable "There were test fails in this run" message - i.e. the first line shown here:
Description
CI runs with any failed/expired tests should result in a fail state for the action as a whole.
This is so that we a) gain this information at a high-level and in future can trigger some automation from it (e.g, alerting a Slack channel or sending an email to stakeholders), and b) gain a quick heads-up that a human needs to go review the results file and determine the test fails/write up the required bug reports, etc.
Expected Change
The overview of each CI page tells us that a test IS a fail (by using the red X icons), such as:
However, that the fail message is not the generic "error code 1" message shown there; instead, it's an informative/automatable "There were test fails in this run" message - i.e. the first line shown here:
