test_runner: support custom message for expectFailure by Han5991 · Pull Request #61563 · nodejs/node

Han5991 · 2026-01-28T14:50:42Z

Summary

This PR enhances the expectFailure option in the test runner to accept different types of values, enabling both custom failure labels and robust error validation. This implementation is referenced from and inspired by nodejs/test-runner#10.

Changes

The expectFailure option now supports the following types:

String: Treated as a failure label (reason).

test('bug', { expectFailure: 'Investigating' }, ...);

RegExp / Function / Error Class: Treated as a matcher to validate the thrown error (similar to assert.throws).

test('regex', { expectFailure: /error message/ }, ...);
test('class', { expectFailure: RangeError }, ...);

Object:
- If it contains label or match properties, it's treated as a configuration object.
```
test('config', {
  expectFailure: {
    label: 'Known issue',
    match: /specific error/
  }
}, ...);
```
- Otherwise, it's treated as an object matcher (properties matching).

Inheritance:

Tests now inherit expectFailure from their parent suite. This allows marking an entire suite as expected to fail.

test('suite', { expectFailure: true }, async (t) => {
  await t.test('subtest', () => { throw new Error(); }); // Passes (expected failure)
});

References

Based on concepts from: doc(proposal): expectFailure label and/or matcher test-runner#10

Resolves: #61570

nodejs-github-bot · 2026-01-28T14:51:15Z

Review requested:

@nodejs/test_runner

doc/api/test.md

JakobJingleheimer

We already have a plan for the value of expectFail: it will soon accept a regular expression to match against.

Han5991 · 2026-01-28T15:25:42Z

We already gave a plan for the value of expectFail: it will soon accept a regular expression to match against.

@JakobJingleheimer
Ah, thanks for the context! I missed that there was an existing plan for error matching.

In that case, I'd be happy to pivot this PR to implement the expectFailure validation logic (accepting a string/regex to match the error) instead of just a message. Does that sound good, or is there someone else already working on it?"

JakobJingleheimer · 2026-01-28T15:33:11Z

@vassudanagunta you were part of the original discussion; did you happen to start an implementation?

To my knowledge though, no-one has started.

I had planned to pick it up next week, but if you would like to do, go ahead.

If you do, I think it would probably be better to start a new PR than to pivot this one. So open a draft and I'll add it to the test-runner team's kanban board so it gets proper visibility.

codecov · 2026-01-28T16:01:01Z

Codecov Report

❌ Patch coverage is 88.60759% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.77%. Comparing base (3819c7f) to head (891ad18).
⚠️ Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/test_runner/test.js	88.46%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #61563      +/-   ##
==========================================
+ Coverage   89.74%   89.77%   +0.02%     
==========================================
  Files         675      675              
  Lines      204642   204749     +107     
  Branches    39322    39358      +36     
==========================================
+ Hits       183657   183811     +154     
+ Misses      13257    13235      -22     
+ Partials     7728     7703      -25

Files with missing lines	Coverage Δ
lib/internal/test_runner/reporter/tap.js	`98.23% <100.00%> (ø)`
lib/internal/test_runner/test.js	`96.84% <88.46%> (-0.49%)`	⬇️

... and 39 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

vassudanagunta · 2026-01-28T18:23:39Z

@JakobJingleheimer nope, haven't started this, though I had long ago implemented it in node-test-extra (not yet released).

I think it's important to get the requirements nailed. IMHO, #61570.

JakobJingleheimer · 2026-01-28T20:10:19Z

As I said, let's put together a proposal in the nodejs/test_runner repo 🙂

vassudanagunta · 2026-01-28T22:00:28Z

As I said

?

I should start a discussion in that repo?

ljharb · 2026-01-28T22:06:07Z

reviewed before reading the discussion; imo a string should work as in this PR whether or not it also supports accepting a regex.

JakobJingleheimer · 2026-01-28T22:31:51Z

It could do. My concern is supporting this without considering the intended regex feature accidentally precluding that intended feature, or inadvertently creating a breaking change, or creating heavily conflicting PRs (very frustrating for the implementators).

I think we can likely get both; we can easily avoid those problems with a quick proposal so everyone is on the same page 🙂

I should start a discussion in that repo?

Please start a proposal like the ones already in that repo 🙂 https://github.com/nodejs/test-runner/tree/main/proposals we can discuss it in that PR

ljharb · 2026-01-28T22:33:40Z

conflicts fair; as long as the "should expect failure" uses truthiness (does an empty string count as true or false, though?), i can't foresee any semantic collision.

Han5991 · 2026-01-29T02:26:17Z

@ljharb @vassudanagunta

I've opened a proposal PR in the test-runner repository as suggested by @JakobJingleheimer.
It would be great to continue the discussion on the spec details there:
nodejs/test-runner#10

Han5991 · 2026-01-30T02:17:37Z

doc/api/test.md

+  expectFailure: {
+    with: /error message/,
+    message: 'reason for failure',
+  },


Updated the proposal based on the feedback:

Added support for Direct Matchers (e.g., expectFailure: /error/).

Clarified that with is for validation and message is for reasoning.

Noted that expectFailure: 'reason' and { message: 'reason' } are equivalent.

It should all happen in the proposal (I believe I suggested closing this PR in the interim; it can always be re-opened).

I've included it in the proposal. Could you please review it?

nodejs/test-runner#10

Yes, I shall tomorrow when I'm back from holiday.

Okay, I'll close this for now as requested until the proposal is passed. I'll reopen it once we are ready

vassudanagunta

I think this needs to be documented a little better for the user.

vassudanagunta · 2026-02-04T06:53:52Z

doc/api/test.md

That minimum changes to fix the incorrect English is to move "currently" before "returns", and to refer to "test-cases" plural.

But the entire sentence is unnecessarily complex and confusing, needing a rewrite. I'd recommend:

Suggested change

In each of the following, `doTheThing()` fails to return `true`,

vassudanagunta · 2026-02-04T06:55:21Z

doc/api/test.md

Suggested change

but since the tests are flagged `expectFailure`, they pass.

doc/api/test.md

+it('should do the thing', { expectFailure: 'feature not implemented' }, () => {
+  assert.strictEqual(doTheThing(), true);
+});
+


vassudanagunta · 2026-02-04T07:33:10Z

doc/api/test.md

+  assert.strictEqual(doTheThing(), true);
+});
+
+it('should fail with specific error', {


Finally, I think this example with the {match, label} value should be broken out as the last example with its own explanation, e.g. "To supply both a reason and specific error for assertFailure..."

vassudanagunta · 2026-02-04T07:43:03Z

doc/api/test.md

    thread. If `false`, only one test runs at a time.
    If unspecified, subtests inherit this value from their parent.
    **Default:** `false`.
+  * `expectFailure` {boolean|string|Object} If truthy, the test is expected to


Missing: that you can specify a specific error directly, without a wrapping {match: ...} object. And probably should reference the specific types with a link to assert.throws as I suggested above.

If it seems redundant to do that in two places, the I guess provide all the details in this section and link to here from Expecting tests to fail.

JakobJingleheimer · 2026-02-04T18:26:45Z

@Han5991 the proposal isn't finished / accepted yet (still hasn't been reviewed by the rest of the test-runner team), so I think it's premature to resume this (the proposal isn't a requirement, but I think it's a good idea and will reduce churn, needless re-reviews, etc—and indeed, there was just earlier today another adjustment to align terms). I do appreciate the enthusiasm 😁

It's added to the team's agenda, so it'll get raised at the next meeting.

Han5991 · 2026-02-04T21:49:20Z

Thanks for letting me know. I'll leave it as a draft until the proposal is finalized.

Resolved by proposal

JakobJingleheimer · 2026-02-09T20:05:36Z

@Han5991 the proposal is all set, so full-steam ahead 🙂 (or whatever pace you want)

Han5991 · 2026-02-09T22:39:27Z

@JakobJingleheimer
Thank you for taking the time to let me know!

vassudanagunta · 2026-02-12T18:09:08Z

Should add "resolves #61570" to PR description or commit message.

vassudanagunta

Some new examples do not jive with the explanatory text. Can reduce the number of examples and shorten the doc without losing any explanatory power.

vassudanagunta · 2026-02-12T18:32:25Z

doc/api/test.md

+it('should fail with regex', { expectFailure: /error message/ }, () => {
+  assert.strictEqual(doTheThing(), true);
+});
+
+it('should fail with function', {
+  expectFailure: (err) => err.code === 'ERR_CODE',
+}, () => {
+  assert.strictEqual(doTheThing(), true);
+});
+
+it('should fail with object matcher', {
+  expectFailure: { code: 'ERR_CODE' },
+}, () => {
+  assert.strictEqual(doTheThing(), true);
+});


These last three examples do not jive with:

In each of the following, doTheThing() fails to return true, but since the tests are flagged expectFailure, they pass.

They will not pass because the regex, matching function and matching object do not match the test failure. Example code should work as described, otherwise it will confuse the reader.

I recommend following my original suggestion made last week (above): Give examples for the error matching options separately, after explaining:

Each of the following tests will fail despite being flagged expectFailure because the failure was not the expected one.

i.e. remove the above three examples. It is sufficient to have the examples of expectFailure test fails below. The reader will get it.

Thank you for your correction request. I have now reflected it.

doc/api/test.md

vassudanagunta

Looks like suite-level support for expectFailure was missed the first time around. Should fix this now... I think it will be easy.

vassudanagunta · 2026-02-13T13:14:43Z

doc/api/test.md


-This flips the pass/fail reporting for a specific test or suite: A flagged test/test-case must throw
-in order to "pass"; a test/test-case that does not throw, fails.
+This flips the pass/fail reporting for a specific test or suite: a flagged test


AFAICT, expectFailure has not been implemented for suite. Per above and the doc for suite options, it should. I think this was missed in the initial expectFailure implementation.

vassudanagunta · 2026-02-13T13:18:37Z

lib/internal/test_runner/test.js

    this.expectedAssertions = plan;
    this.cancelled = false;
-    this.expectFailure = expectFailure !== undefined && expectFailure !== false;
+    this.expectFailure = parseExpectFailure(expectFailure);


Supporting suite.expectFailure might be as simple as:

Suggested change

this.expectFailure = parseExpectFailure(expectFailure);

this.expectFailure = parseExpectFailure(expectFailure) || this.parent?.expectFailure;

vassudanagunta · 2026-02-13T13:22:45Z

test/parallel/test-runner-xfail.js

Should add test coverage for suite level expectFailure.

vassudanagunta · 2026-02-13T13:25:04Z

doc/api/test.md


-In the following, `doTheThing()` returns _currently_ `false` (`false` does not equal `true`, causing
-`strictEqual` to throw, so the test-case passes).
+In each of the following, `doTheThing()` fail to return `true`, but since the


Suggested change

In each of the following, `doTheThing()` fail to return `true`, but since the

In each of the following, `doTheThing()` fails to return `true`, but since the

vassudanagunta · 2026-02-13T13:29:07Z

doc/api/test.md

+
+// To supply both a reason and specific error for `expectFailure`, use { label, match }.


Suggested change

// To supply both a reason and specific error for `expectFailure`, use { label, match }.

```

To supply both a reason and specific error for `expectFailure`,

use `{ label, match }`.

```js

would be more readable, and is consistent with the rest of the docs.

Normalize expectFailure values to support string labels, direct matchers, and { label, match } objects while validating errors via assert.throws. Allow tests to inherit the 'expectFailure' option from their parent suite. Update TAP reporting and tests, including function matcher coverage and unexpected-pass behavior, and clarify docs for direct matcher usage. Resolves: nodejs#61570

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jan 28, 2026

aduh95 reviewed Jan 28, 2026

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

JakobJingleheimer previously requested changes Jan 28, 2026

View reviewed changes

ljharb approved these changes Jan 28, 2026

View reviewed changes

Han5991 force-pushed the test-runner/support-getxfail-message branch from 1af3584 to 13aedaa Compare January 28, 2026 23:26

Han5991 requested a review from aduh95 January 28, 2026 23:28

Han-sangwook mentioned this pull request Jan 29, 2026

feat: proposal for expectFailure enhancements nodejs/test-runner#9

Closed

Han5991 mentioned this pull request Jan 29, 2026

doc(proposal): expectFailure label and/or matcher nodejs/test-runner#10

Merged

avivkeller approved these changes Jan 29, 2026

View reviewed changes

vassudanagunta reviewed Jan 29, 2026

View reviewed changes

Han5991 marked this pull request as draft January 31, 2026 07:56

Han5991 closed this Jan 31, 2026

Han5991 reopened this Feb 3, 2026

Han5991 marked this pull request as ready for review February 3, 2026 22:45

Han5991 requested a review from JakobJingleheimer February 3, 2026 22:45

Han5991 force-pushed the test-runner/support-getxfail-message branch from 061f049 to 346ec8f Compare February 4, 2026 00:49

vassudanagunta reviewed Feb 4, 2026

View reviewed changes

Han5991 marked this pull request as draft February 4, 2026 21:42

Han5991 force-pushed the test-runner/support-getxfail-message branch from 7946173 to 48e85f0 Compare February 9, 2026 22:30

Han5991 marked this pull request as ready for review February 9, 2026 22:30

Han5991 force-pushed the test-runner/support-getxfail-message branch from 48e85f0 to a5eaf98 Compare February 9, 2026 22:32

Han5991 force-pushed the test-runner/support-getxfail-message branch 2 times, most recently from c2bebdf to 7145647 Compare February 10, 2026 01:31

vassudanagunta reviewed Feb 12, 2026

View reviewed changes

Han5991 force-pushed the test-runner/support-getxfail-message branch 2 times, most recently from 09def7f to 8fa0720 Compare February 12, 2026 22:04

Han5991 requested a review from vassudanagunta February 12, 2026 22:04

vassudanagunta reviewed Feb 13, 2026

View reviewed changes

Han5991 force-pushed the test-runner/support-getxfail-message branch from 8fa0720 to 891ad18 Compare February 13, 2026 16:06


	In each of the following, `doTheThing()` fails to return `true`,

	this.expectFailure = parseExpectFailure(expectFailure);
	this.expectFailure = parseExpectFailure(expectFailure) \|\| this.parent?.expectFailure;


		// To supply both a reason and specific error for `expectFailure`, use { label, match }.

Uh oh!

Conversation

Han5991 commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

References

Uh oh!

nodejs-github-bot commented Jan 28, 2026

Uh oh!

Uh oh!

JakobJingleheimer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Han5991 commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

codecov bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vassudanagunta commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

vassudanagunta commented Jan 28, 2026

Uh oh!

ljharb commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

ljharb commented Jan 28, 2026

Uh oh!

Han5991 commented Jan 29, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

JakobJingleheimer Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vassudanagunta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

vassudanagunta Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JakobJingleheimer commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Han5991 commented Feb 4, 2026

Uh oh!

JakobJingleheimer commented Feb 9, 2026

Han5991 commented Jan 28, 2026 •

edited

Loading

JakobJingleheimer left a comment •

edited

Loading

codecov bot commented Jan 28, 2026 •

edited

Loading

JakobJingleheimer Jan 31, 2026 •

edited

Loading

vassudanagunta Feb 4, 2026 •

edited

Loading

JakobJingleheimer commented Feb 4, 2026 •

edited

Loading

vassudanagunta commented Feb 12, 2026 •

edited

Loading

vassudanagunta Feb 12, 2026 •

edited

Loading