From 3d8c0ba795f120fc52d0705e05948b4de5037f68 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Thu, 10 Jul 2025 23:22:43 +0200 Subject: [PATCH 01/22] feat: add assertion options to RuleTester --- .../README.md | 241 ++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 designs/2025-rule-tester-assertion-options/README.md diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md new file mode 100644 index 00000000..2ec7b49b --- /dev/null +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -0,0 +1,241 @@ +- Repo: https://github.com/eslint/eslint +- Start Date: 2025-07-10 +- RFC PR: (leave this empty, to be filled in later) +- Authors: ST-DDT + +# Rule Tester Assertions Options + +## Summary + +Add options that control which assertions are required for each `RuleTester` test case. + +## Motivation + +In most eslint(-plugin)s' rules the error assertions are different from each other. +Adding options that could be set/shared when configuring the rule tester would ensure a common base level of assertions is met throughout the (plugin's) project. +The options could be defined on two levels. On `RuleTester`'s `constructor` effectively impacting all tests, or on the test `run` method itself, so only that set is affected. + +## Detailed Design + +### Variant 1 - Constructor based options + +````ts +new RuleTester(testerConfig: {...}, assertionOptions: { + /** + * Require message assertion for each invalid test case. + * + * @default false + */ + requireMessage: boolean; + /** + * Require full location assertions for each invalid test case. + * + * @default false + */ + requireLocation: boolean; +} = {}); +```` + +### Variant 2 - Test method based options + +````ts +ruleTester.run("rule-name", rule, tests, assertionOptions: { + /** + * Require message assertion for each invalid test case. + * + * @default false + */ + requireMessage: boolean; + /** + * Require full location assertions for each invalid test case. + * + * @default false + */ + requireLocation: boolean; + /** + * Require and expect only the given test scenarios. + * This allows omitting certain scenarios from this run with the current options. + * + * @default ["valid","invalid"] + */ + requiredScenarios: ReadonlyArray<'valid' | 'invalid'>; +}); +```` + +### Shared Logic + +If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. +This can be done either by providing a only message, or by using the `message` property of the error object in the assertion (Same as the current behavior). +We could enable this property by default, but it would be a breaking change. +If we add a `requireMessageId` option, it would be mutually exclusive with `requireMessage`, and the invalid test case cannot consist of an error count or message assertion only, but must also include a messageId assertion. +Alternatively, we could alter the `requireMessage` option to `false | true | "message" | "messageId"` (`true` => `"message"`). + +````ts +ruleTester.run("rule-name", rule, { + invalid: [ + { + code: "const a = 1;", + errors: 1, // ❌ + }, + { + code: "const a = 2;", + errors: [ + "Error message here.", // ✅ + ] + }, + { + code: "const a = 3;", + errors: [ + { + message: "Error message here.", // ✅ + } + ] + } + ] +}, { + requireMessage: true +}); +```` + +If `requireLocation` is set to `true`, the invalid test case cannot consist of an error count or errorMessage assertion only, but must also include a full location assertion. +We could enable this property by default, but it would be a breaking change. + +````ts +ruleTester.run("rule-name", rule, { + invalid: [ + { + code: "const a = 1;", + errors: 1, // ❌ + }, + { + code: "const a = 2;", + errors: [ + "Error message here.", // ❌ + ] + }, + { + code: "const a = 3;", + errors: [ + { + line: 1, // ❌ + column: 1, + + } + ] + }, + { + code: "const a = 4;", + errors: [ + { + line: 1, // ✅ + column: 1, + endLine: 1, + endColumn: 12, + } + ] + } + ] +}, { + requireLocation: true +}); +```` + +If `requiredScenarios` is set, the `run` will only require and expect the given scenarios. +This can only be used for the `run` method, not the constructor, because there should always be at least one valid and one invalid test case. +The `requiredScenarios` option can be used to omit certain scenarios from the run, e.g. if the user wants to import a set of tests from a different source, that may have other assertion requirements or haven't achieved the quality needed to require the same assertion strictness. + +````ts +ruleTester.run("rule-name", rule, { + valid: [...], // ✅ + invalid: [...], +}, { + // Default is ["valid", "invalid"] +}); +ruleTester.run("rule-name", rule, { + valid: [...], // ❌ +}, { + // Default is ["valid", "invalid"] +}); +ruleTester.run("rule-name", rule, { + valid: [...], // ✅ + invalid: [...], +}, { + requiredScenarios: ["valid", "invalid"] +}); +ruleTester.run("rule-name", rule, { + valid: [...], // ✅ +}, { + requiredScenarios: ["valid"] +}); +ruleTester.run("rule-name", rule, { + valid: [...], // ❌ + invalid: [...], +}, { + requiredScenarios: ["invalid"] +}); +```` + +## Documentation + +This RFC will be documented in the RuleTester documentation, explaining the new options and how to use them. +So mainly here: https://eslint.org/docs/latest/integrate/nodejs-api#ruletester +Additionally, we should write a short blog post to announce for plugin maintainers, to raise awareness of the new options and encourage them to use them in their tests. + +## Drawbacks + +This proposal adds slightly more complexity to the RuleTester logic, as it needs to handle the new options and enforce the assertions based on them. +Currently, the RuleTester logic is already deeply nested, so adding more options may make it harder to read and maintain. + +Additionally, since we add the options as a second parameter it might interfere with future additions to the parameters. +This could by eleviated by renaming the parameter from `assertionOptions` to `options` (either from the start or when the need for different type of options arises). + +If we enable the `requireMessage` and `requireLocation` options by default, it would be a breaking change for existing tests that do not follow these assertion requirements yet. + +## Backwards Compatibility Analysis + +This change should not affect existing ESLint users or plugin developers, as it only adds new options to the RuleTester and does not change any existing behavior. +If we enable the `requireMessage` and `requireLocation` options by default, it would be a breaking change for existing tests that do not follow these assertion requirements yet. +If we want to enable them by default, we should do so in a major release and communicate the upcoming change to the users early via blog post. + +## Alternatives + +As an alternative to this proposal, we could add a eslint rule that applies the same assertions, but uses the central eslint config. +While this would apply the same assertions for all rule testers, it would be a lot more complex to implement and maintain, +it requires identifying the RuleTester calls in the codebase and might run into issues if the assertions aren't specified inline but via a variable or transformation. + +## Open Questions + +1. Is there a need for disabling scenarios like `valid` or `invalid`? +2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? +3. Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change) +4. Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions? +5. Should we add a `strict` option that enables all assertion options by default? + +## Help Needed + +English is not my first language, so I would appreciate help with the wording and grammar of this RFC. +I'm able to implement this RFC, if we decide to go with options instead of a new rule. + +## Frequently Asked Questions + +### Why + +Because it is easy to miss a missing assertion in RuleTester test cases, especially when many new invalid test cases are added. + +## Related Discussions + +The idea was initially sparked by this comment: vuejs/eslint-plugin-vue#2773 (comment) + +- https://github.com/vuejs/eslint-plugin-vue/pull/2773#discussion_r2176359714 + +> It might be helpful to include more detailed error information, such as line, column, endLine, and endColumn... +> [...] +> Let’s make the new cases more detailed first. 😊 + +The first steps have been taken in: eslint/eslint#19904 - feat: output full actual location in rule tester if different + +- https://github.com/eslint/eslint/pull/19904 + +This lead to the issue that this RFC is based on: eslint/eslint#19921 - Change Request: Add options to rule-tester requiring certain assertions + +- https://github.com/eslint/eslint/issues/19921 From e25685f3825ab90d84270bd1092dcc38ec8ef55a Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sat, 12 Jul 2025 01:21:13 +0200 Subject: [PATCH 02/22] docs: address soem feedback --- .../README.md | 31 ++++++++++++------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 2ec7b49b..ec5ae85e 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -23,12 +23,16 @@ The options could be defined on two levels. On `RuleTester`'s `constructor` effe new RuleTester(testerConfig: {...}, assertionOptions: { /** * Require message assertion for each invalid test case. + * If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. + * If `'message'`, errors must extend `string | Array<{ message: string }>`. + * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. * * @default false */ - requireMessage: boolean; + requireMessage: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. + * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. * * @default false */ @@ -42,12 +46,16 @@ new RuleTester(testerConfig: {...}, assertionOptions: { ruleTester.run("rule-name", rule, tests, assertionOptions: { /** * Require message assertion for each invalid test case. + * If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. + * If `'message'`, errors must extend `string | Array<{ message: string }>`. + * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. * * @default false */ - requireMessage: boolean; + requireMessage: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. + * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. * * @default false */ @@ -65,9 +73,7 @@ ruleTester.run("rule-name", rule, tests, assertionOptions: { ### Shared Logic If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. -This can be done either by providing a only message, or by using the `message` property of the error object in the assertion (Same as the current behavior). -We could enable this property by default, but it would be a breaking change. -If we add a `requireMessageId` option, it would be mutually exclusive with `requireMessage`, and the invalid test case cannot consist of an error count or message assertion only, but must also include a messageId assertion. +This can be done either by providing only a `string` message, or by using the `message` property of the error object in the assertion (Same as the current behavior). Alternatively, we could alter the `requireMessage` option to `false | true | "message" | "messageId"` (`true` => `"message"`). ````ts @@ -97,8 +103,7 @@ ruleTester.run("rule-name", rule, { }); ```` -If `requireLocation` is set to `true`, the invalid test case cannot consist of an error count or errorMessage assertion only, but must also include a full location assertion. -We could enable this property by default, but it would be a breaking change. +If `requireLocation` is set to `true`, the invalid test case's error validation must include all location assertions such as `line`, `column`, `endLine`, and `endColumn`. ````ts ruleTester.run("rule-name", rule, { @@ -186,16 +191,20 @@ Additionally, we should write a short blog post to announce for plugin maintaine This proposal adds slightly more complexity to the RuleTester logic, as it needs to handle the new options and enforce the assertions based on them. Currently, the RuleTester logic is already deeply nested, so adding more options may make it harder to read and maintain. +Due to limitations of the `RuleTester`, the error message does not point to the correct location of the test case. +While this is true already, it gets slightly more relevant, since now it might report missing properties, that cannot be searched for in the test file. + +See also: https://github.com/eslint/eslint/issues/19936 + Additionally, since we add the options as a second parameter it might interfere with future additions to the parameters. This could by eleviated by renaming the parameter from `assertionOptions` to `options` (either from the start or when the need for different type of options arises). If we enable the `requireMessage` and `requireLocation` options by default, it would be a breaking change for existing tests that do not follow these assertion requirements yet. +It is not planned to enable `requireMessage` or `requireLocation` by default. ## Backwards Compatibility Analysis This change should not affect existing ESLint users or plugin developers, as it only adds new options to the RuleTester and does not change any existing behavior. -If we enable the `requireMessage` and `requireLocation` options by default, it would be a breaking change for existing tests that do not follow these assertion requirements yet. -If we want to enable them by default, we should do so in a major release and communicate the upcoming change to the users early via blog post. ## Alternatives @@ -207,8 +216,8 @@ it requires identifying the RuleTester calls in the codebase and might run into 1. Is there a need for disabling scenarios like `valid` or `invalid`? 2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? -3. Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change) -4. Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions? +3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No +4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` 5. Should we add a `strict` option that enables all assertion options by default? ## Help Needed From 641dabb7a6ad70dbc981a2618c3065d5f052f5ff Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sat, 12 Jul 2025 01:36:33 +0200 Subject: [PATCH 03/22] docs: address some feedback --- .../README.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index ec5ae85e..99d9b325 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -72,28 +72,30 @@ ruleTester.run("rule-name", rule, tests, assertionOptions: { ### Shared Logic -If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. -This can be done either by providing only a `string` message, or by using the `message` property of the error object in the assertion (Same as the current behavior). -Alternatively, we could alter the `requireMessage` option to `false | true | "message" | "messageId"` (`true` => `"message"`). +If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. (See below) +This can be done either by providing only a `string` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). +If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. +If `'message'`, errors must extend `string | Array<{ message: string }>`. +If `'messageId'`, errors must extend `Array<{ messageId: string }>`. ````ts ruleTester.run("rule-name", rule, { invalid: [ { code: "const a = 1;", - errors: 1, // ❌ + errors: 1, // ❌ Message isn't being checked here }, { code: "const a = 2;", errors: [ - "Error message here.", // ✅ + "Error message here.", // ✅ we only check the error message ] }, { code: "const a = 3;", errors: [ { - message: "Error message here.", // ✅ + message: "Error message here.", // ✅ we check the error message and potentially other properties } ] } @@ -110,19 +112,19 @@ ruleTester.run("rule-name", rule, { invalid: [ { code: "const a = 1;", - errors: 1, // ❌ + errors: 1, // ❌ Location isn't checked here }, { code: "const a = 2;", errors: [ - "Error message here.", // ❌ + "Error message here.", // ❌ Location isn't checked here ] }, { code: "const a = 3;", errors: [ { - line: 1, // ❌ + line: 1, // ❌ Location isn't fully checked here column: 1, } @@ -132,7 +134,7 @@ ruleTester.run("rule-name", rule, { code: "const a = 4;", errors: [ { - line: 1, // ✅ + line: 1, // ✅ All location properties have been checked column: 1, endLine: 1, endColumn: 12, From 05a971e134c7eb3e1ea34cce2ab0a7aed4991df0 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Thu, 17 Jul 2025 23:22:06 +0200 Subject: [PATCH 04/22] docs: improve requiredScenarios documentation --- .../README.md | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 99d9b325..42f8575f 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -147,9 +147,10 @@ ruleTester.run("rule-name", rule, { }); ```` +Having this option is optional. If `requiredScenarios` is set, the `run` will only require and expect the given scenarios. -This can only be used for the `run` method, not the constructor, because there should always be at least one valid and one invalid test case. -The `requiredScenarios` option can be used to omit certain scenarios from the run, e.g. if the user wants to import a set of tests from a different source, that may have other assertion requirements or haven't achieved the quality needed to require the same assertion strictness. +This can only be used for the `run` method, not the constructor, because there should always be at least one valid and one invalid test case (potentially spread over multiple run calls). +The `requiredScenarios` option can be used to disable having to provide `valid`/`invalid` scenarios. ````ts ruleTester.run("rule-name", rule, { @@ -182,6 +183,27 @@ ruleTester.run("rule-name", rule, { }); ```` +This option is meant to be combined with the other options, to make sure each test set applies the best applicable assertion coverage. + +````ts +ruleTester.run("rule-name", rule, { + valid: [...], + invalid: [...], +}; +ruleTester.run("rule-name", rule, { + invalid: [...], +}, { + requiredScenarios: ["invalid"], + requireMessage: true, +}); +ruleTester.run("rule-name", rule, { + invalid: [...], +}, { + requiredScenarios: ["invalid"], + requireLocation: true, +}); +```` + ## Documentation This RFC will be documented in the RuleTester documentation, explaining the new options and how to use them. From 19762beea57ccd80e1030ea94e9ddcaf3e1f08c4 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Fri, 18 Jul 2025 00:24:11 +0200 Subject: [PATCH 05/22] docs: add implementation hint --- .../README.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 42f8575f..04a74717 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -72,6 +72,8 @@ ruleTester.run("rule-name", rule, tests, assertionOptions: { ### Shared Logic +#### requireMessage + If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. (See below) This can be done either by providing only a `string` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. @@ -105,6 +107,8 @@ ruleTester.run("rule-name", rule, { }); ```` +#### requireLocation + If `requireLocation` is set to `true`, the invalid test case's error validation must include all location assertions such as `line`, `column`, `endLine`, and `endColumn`. ````ts @@ -147,6 +151,8 @@ ruleTester.run("rule-name", rule, { }); ```` +#### requiredScenarios + Having this option is optional. If `requiredScenarios` is set, the `run` will only require and expect the given scenarios. This can only be used for the `run` method, not the constructor, because there should always be at least one valid and one invalid test case (potentially spread over multiple run calls). @@ -204,6 +210,117 @@ ruleTester.run("rule-name", rule, { }); ```` +## Implementation Hint + +````patch +diff --git a/lib/rule-tester/rule-tester.js b/lib/rule-tester/rule-tester.js +index dbd8c274f..f02d34d95 100644 +--- a/lib/rule-tester/rule-tester.js ++++ b/lib/rule-tester/rule-tester.js +@@ -558,13 +558,22 @@ class RuleTester { + * valid: (ValidTestCase | string)[], + * invalid: InvalidTestCase[] + * }} test The collection of tests to run. ++ * @param {{ ++ * requireMessage: boolean | "message" | "messageId", ++ * requireLocation: boolean ++ * requiredScenarios: ["valid" | "invalid"], ++ * }} [options] Options for the test. + * @throws {TypeError|Error} If `rule` is not an object with a `create` method, + * or if non-object `test`, or if a required scenario of the given type is missing. + * @returns {void} + */ +- run(ruleName, rule, test) { +- const testerConfig = this.testerConfig, ++ run(ruleName, rule, test, options = {}) { ++ const { ++ requireMessage = false, ++ requireLocation = false, + requiredScenarios = ["valid", "invalid"], ++ } = options; ++ const testerConfig = this.testerConfig, + scenarioErrors = [], + linter = this.linter, + ruleId = `rule-to-test/${ruleName}`; +@@ -1092,6 +1101,10 @@ class RuleTester { + } + + if (typeof item.errors === "number") { ++ assert.ok( ++ !requireMessage && !requireLocation, ++ "Invalid cases must have 'errors' value as an array", ++ ); + if (item.errors === 0) { + assert.fail( + "Invalid cases must have 'error' value greater than 0", +@@ -1137,6 +1150,10 @@ class RuleTester { + + if (typeof error === "string" || error instanceof RegExp) { + // Just an error message. ++ assert.ok( ++ requireMessage !== 'messageId' && !requireLocation, ++ "Invalid cases must have 'errors' value as an array of objects e.g. { message: 'error message' }", ++ ); + assertMessageMatches(message.message, error); + assert.ok( + message.suggestions === void 0, +@@ -1157,6 +1174,10 @@ class RuleTester { + }); + + if (hasOwnProperty(error, "message")) { ++ assert.ok( ++ requireMessage !== "messageId", ++ "The test run requires 'messageId' but the error (also) uses 'message'.", ++ ); + assert.ok( + !hasOwnProperty(error, "messageId"), + "Error should not specify both 'message' and a 'messageId'.", +@@ -1165,11 +1186,16 @@ class RuleTester { + !hasOwnProperty(error, "data"), + "Error should not specify both 'data' and 'message'.", + ); ++ + assertMessageMatches( + message.message, + error.message, + ); + } else if (hasOwnProperty(error, "messageId")) { ++ assert.ok( ++ requireMessage !== "message", ++ "The test run requires 'message' but the error (also) uses 'messageId'.", ++ ); + assert.ok( + ruleHasMetaMessages, + "Error can not use 'messageId' if rule under test doesn't define 'meta.messages'.", +@@ -1239,22 +1265,22 @@ class RuleTester { + const actualLocation = {}; + const expectedLocation = {}; + +- if (hasOwnProperty(error, "line")) { ++ if (hasOwnProperty(error, "line") || requireLocation) { + actualLocation.line = message.line; + expectedLocation.line = error.line; + } + +- if (hasOwnProperty(error, "column")) { ++ if (hasOwnProperty(error, "column") || requireLocation) { + actualLocation.column = message.column; + expectedLocation.column = error.column; + } + +- if (hasOwnProperty(error, "endLine")) { ++ if (hasOwnProperty(error, "endLine") || requireLocation) { + actualLocation.endLine = message.endLine; + expectedLocation.endLine = error.endLine; + } + +- if (hasOwnProperty(error, "endColumn")) { ++ if (hasOwnProperty(error, "endColumn") || requireLocation) { + actualLocation.endColumn = message.endColumn; + expectedLocation.endColumn = error.endColumn; + } +```` + ## Documentation This RFC will be documented in the RuleTester documentation, explaining the new options and how to use them. From d27f1720921b6d5aaa1946e1577ed77c51632f5a Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Mon, 21 Jul 2025 19:52:15 +0200 Subject: [PATCH 06/22] chore: apply suggestions --- .../README.md | 77 ++----------------- 1 file changed, 5 insertions(+), 72 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 04a74717..a597decc 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -60,13 +60,6 @@ ruleTester.run("rule-name", rule, tests, assertionOptions: { * @default false */ requireLocation: boolean; - /** - * Require and expect only the given test scenarios. - * This allows omitting certain scenarios from this run with the current options. - * - * @default ["valid","invalid"] - */ - requiredScenarios: ReadonlyArray<'valid' | 'invalid'>; }); ```` @@ -76,8 +69,8 @@ ruleTester.run("rule-name", rule, tests, assertionOptions: { If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. (See below) This can be done either by providing only a `string` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). -If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. -If `'message'`, errors must extend `string | Array<{ message: string }>`. +If `true`, errors must extend `string | Array<{ message: string | RegExp } | { messageId: string }>`. +If `'message'`, errors must extend `string | Array<{ message: string | RegExp }>`. If `'messageId'`, errors must extend `Array<{ messageId: string }>`. ````ts @@ -151,65 +144,6 @@ ruleTester.run("rule-name", rule, { }); ```` -#### requiredScenarios - -Having this option is optional. -If `requiredScenarios` is set, the `run` will only require and expect the given scenarios. -This can only be used for the `run` method, not the constructor, because there should always be at least one valid and one invalid test case (potentially spread over multiple run calls). -The `requiredScenarios` option can be used to disable having to provide `valid`/`invalid` scenarios. - -````ts -ruleTester.run("rule-name", rule, { - valid: [...], // ✅ - invalid: [...], -}, { - // Default is ["valid", "invalid"] -}); -ruleTester.run("rule-name", rule, { - valid: [...], // ❌ -}, { - // Default is ["valid", "invalid"] -}); -ruleTester.run("rule-name", rule, { - valid: [...], // ✅ - invalid: [...], -}, { - requiredScenarios: ["valid", "invalid"] -}); -ruleTester.run("rule-name", rule, { - valid: [...], // ✅ -}, { - requiredScenarios: ["valid"] -}); -ruleTester.run("rule-name", rule, { - valid: [...], // ❌ - invalid: [...], -}, { - requiredScenarios: ["invalid"] -}); -```` - -This option is meant to be combined with the other options, to make sure each test set applies the best applicable assertion coverage. - -````ts -ruleTester.run("rule-name", rule, { - valid: [...], - invalid: [...], -}; -ruleTester.run("rule-name", rule, { - invalid: [...], -}, { - requiredScenarios: ["invalid"], - requireMessage: true, -}); -ruleTester.run("rule-name", rule, { - invalid: [...], -}, { - requiredScenarios: ["invalid"], - requireLocation: true, -}); -```` - ## Implementation Hint ````patch @@ -224,21 +158,19 @@ index dbd8c274f..f02d34d95 100644 + * @param {{ + * requireMessage: boolean | "message" | "messageId", + * requireLocation: boolean -+ * requiredScenarios: ["valid" | "invalid"], + * }} [options] Options for the test. * @throws {TypeError|Error} If `rule` is not an object with a `create` method, * or if non-object `test`, or if a required scenario of the given type is missing. * @returns {void} */ - run(ruleName, rule, test) { -- const testerConfig = this.testerConfig, + run(ruleName, rule, test, options = {}) { + const { + requireMessage = false, + requireLocation = false, - requiredScenarios = ["valid", "invalid"], + } = options; -+ const testerConfig = this.testerConfig, + const testerConfig = this.testerConfig, + requiredScenarios = ["valid", "invalid"], scenarioErrors = [], linter = this.linter, ruleId = `rule-to-test/${ruleName}`; @@ -360,6 +292,7 @@ it requires identifying the RuleTester calls in the codebase and might run into 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` 5. Should we add a `strict` option that enables all assertion options by default? +6. ~~Should we expose `requiredScenarios` as option?~~ No, the users can just use empty arrays. ## Help Needed From 69af370897232948b0fdd2f702d1d5bd97d6ead6 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Mon, 21 Jul 2025 20:00:49 +0200 Subject: [PATCH 07/22] chore: apply suggestions --- .../README.md | 205 +++++++++--------- 1 file changed, 104 insertions(+), 101 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index a597decc..aa861cd2 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -148,109 +148,112 @@ ruleTester.run("rule-name", rule, { ````patch diff --git a/lib/rule-tester/rule-tester.js b/lib/rule-tester/rule-tester.js -index dbd8c274f..f02d34d95 100644 +index dbd8c274f..412090df0 100644 --- a/lib/rule-tester/rule-tester.js +++ b/lib/rule-tester/rule-tester.js -@@ -558,13 +558,22 @@ class RuleTester { - * valid: (ValidTestCase | string)[], - * invalid: InvalidTestCase[] - * }} test The collection of tests to run. -+ * @param {{ -+ * requireMessage: boolean | "message" | "messageId", -+ * requireLocation: boolean -+ * }} [options] Options for the test. - * @throws {TypeError|Error} If `rule` is not an object with a `create` method, - * or if non-object `test`, or if a required scenario of the given type is missing. - * @returns {void} - */ -- run(ruleName, rule, test) { -+ run(ruleName, rule, test, options = {}) { -+ const { -+ requireMessage = false, -+ requireLocation = false, -+ } = options; - const testerConfig = this.testerConfig, - requiredScenarios = ["valid", "invalid"], - scenarioErrors = [], - linter = this.linter, - ruleId = `rule-to-test/${ruleName}`; -@@ -1092,6 +1101,10 @@ class RuleTester { - } - - if (typeof item.errors === "number") { -+ assert.ok( -+ !requireMessage && !requireLocation, -+ "Invalid cases must have 'errors' value as an array", -+ ); - if (item.errors === 0) { - assert.fail( - "Invalid cases must have 'error' value greater than 0", -@@ -1137,6 +1150,10 @@ class RuleTester { - - if (typeof error === "string" || error instanceof RegExp) { - // Just an error message. -+ assert.ok( -+ requireMessage !== 'messageId' && !requireLocation, -+ "Invalid cases must have 'errors' value as an array of objects e.g. { message: 'error message' }", -+ ); - assertMessageMatches(message.message, error); - assert.ok( - message.suggestions === void 0, -@@ -1157,6 +1174,10 @@ class RuleTester { - }); - - if (hasOwnProperty(error, "message")) { -+ assert.ok( -+ requireMessage !== "messageId", -+ "The test run requires 'messageId' but the error (also) uses 'message'.", -+ ); - assert.ok( - !hasOwnProperty(error, "messageId"), - "Error should not specify both 'message' and a 'messageId'.", -@@ -1165,11 +1186,16 @@ class RuleTester { - !hasOwnProperty(error, "data"), - "Error should not specify both 'data' and 'message'.", - ); -+ - assertMessageMatches( - message.message, - error.message, - ); - } else if (hasOwnProperty(error, "messageId")) { -+ assert.ok( -+ requireMessage !== "message", -+ "The test run requires 'message' but the error (also) uses 'messageId'.", -+ ); - assert.ok( - ruleHasMetaMessages, - "Error can not use 'messageId' if rule under test doesn't define 'meta.messages'.", -@@ -1239,22 +1265,22 @@ class RuleTester { - const actualLocation = {}; - const expectedLocation = {}; - -- if (hasOwnProperty(error, "line")) { -+ if (hasOwnProperty(error, "line") || requireLocation) { - actualLocation.line = message.line; - expectedLocation.line = error.line; - } - -- if (hasOwnProperty(error, "column")) { -+ if (hasOwnProperty(error, "column") || requireLocation) { - actualLocation.column = message.column; - expectedLocation.column = error.column; - } - -- if (hasOwnProperty(error, "endLine")) { -+ if (hasOwnProperty(error, "endLine") || requireLocation) { - actualLocation.endLine = message.endLine; - expectedLocation.endLine = error.endLine; - } - -- if (hasOwnProperty(error, "endColumn")) { -+ if (hasOwnProperty(error, "endColumn") || requireLocation) { - actualLocation.endColumn = message.endColumn; - expectedLocation.endColumn = error.endColumn; - } +@@ -558,11 +558,19 @@ class RuleTester { + * valid: (ValidTestCase | string)[], + * invalid: InvalidTestCase[] + * }} test The collection of tests to run. ++ * @param {{ ++ * requireMessage: boolean | "message" | "messageId", ++ * requireLocation: boolean, ++ * }} [options] Options for the test. + * @throws {TypeError|Error} If `rule` is not an object with a `create` method, + * or if non-object `test`, or if a required scenario of the given type is missing. + * @returns {void} + */ +- run(ruleName, rule, test) { ++ run(ruleName, rule, test, options = {}) { ++ const { ++ requireMessage = false, ++ requireLocation = false, ++ } = options; + const testerConfig = this.testerConfig, + requiredScenarios = ["valid", "invalid"], + scenarioErrors = [], +@@ -1092,6 +1100,10 @@ class RuleTester { + } + + if (typeof item.errors === "number") { ++ assert.ok( ++ !requireMessage && !requireLocation, ++ "Invalid cases must have 'errors' value as an array", ++ ); + if (item.errors === 0) { + assert.fail( + "Invalid cases must have 'error' value greater than 0", +@@ -1137,6 +1149,10 @@ class RuleTester { + + if (typeof error === "string" || error instanceof RegExp) { + // Just an error message. ++ assert.ok( ++ requireMessage !== 'messageId' && !requireLocation, ++ "Invalid cases must have 'errors' value as an array of objects e.g. { message: 'error message' }", ++ ); + assertMessageMatches(message.message, error); + assert.ok( + message.suggestions === void 0, +@@ -1157,6 +1173,10 @@ class RuleTester { + }); + + if (hasOwnProperty(error, "message")) { ++ assert.ok( ++ requireMessage !== "messageId", ++ "The test run requires 'messageId' but the error (also) uses 'message'.", ++ ); + assert.ok( + !hasOwnProperty(error, "messageId"), + "Error should not specify both 'message' and a 'messageId'.", +@@ -1170,6 +1190,10 @@ class RuleTester { + error.message, + ); + } else if (hasOwnProperty(error, "messageId")) { ++ assert.ok( ++ requireMessage !== "message", ++ "The test run requires 'message' but the error (also) uses 'messageId'.", ++ ); + assert.ok( + ruleHasMetaMessages, + "Error can not use 'messageId' if rule under test doesn't define 'meta.messages'.", +@@ -1242,21 +1266,37 @@ class RuleTester { + if (hasOwnProperty(error, "line")) { + actualLocation.line = message.line; + expectedLocation.line = error.line; ++ } else if (requireLocation) { ++ assert.fail( ++ "Test error must specify a 'line' property.", ++ ); + } + + if (hasOwnProperty(error, "column")) { + actualLocation.column = message.column; + expectedLocation.column = error.column; ++ } else if (requireLocation) { ++ assert.fail( ++ "Test error must specify a 'column' property.", ++ ); + } + + if (hasOwnProperty(error, "endLine")) { + actualLocation.endLine = message.endLine; + expectedLocation.endLine = error.endLine; ++ } else if (requireLocation) { ++ assert.fail( ++ "Test error must specify an 'endLine' property.", ++ ); + } + + if (hasOwnProperty(error, "endColumn")) { + actualLocation.endColumn = message.endColumn; + expectedLocation.endColumn = error.endColumn; ++ } else if (requireLocation) { ++ assert.fail( ++ "Test error must specify an 'endColumn' property.", ++ ); + } + + if (Object.keys(expectedLocation).length > 0) { ```` ## Documentation From cd33bfcf2c5f65bda8aebec58a07d5a56f41cec6 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Mon, 21 Jul 2025 20:04:06 +0200 Subject: [PATCH 08/22] chore: apply suggestions --- .../2025-rule-tester-assertion-options/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index aa861cd2..a5204db7 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -23,8 +23,8 @@ The options could be defined on two levels. On `RuleTester`'s `constructor` effe new RuleTester(testerConfig: {...}, assertionOptions: { /** * Require message assertion for each invalid test case. - * If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. - * If `'message'`, errors must extend `string | Array<{ message: string }>`. + * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. + * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. * * @default false @@ -46,8 +46,8 @@ new RuleTester(testerConfig: {...}, assertionOptions: { ruleTester.run("rule-name", rule, tests, assertionOptions: { /** * Require message assertion for each invalid test case. - * If `true`, errors must extend `string | Array<{ message: string } | { messageId: string }>`. - * If `'message'`, errors must extend `string | Array<{ message: string }>`. + * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. + * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. * * @default false @@ -68,9 +68,9 @@ ruleTester.run("rule-name", rule, tests, assertionOptions: { #### requireMessage If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. (See below) -This can be done either by providing only a `string` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). -If `true`, errors must extend `string | Array<{ message: string | RegExp } | { messageId: string }>`. -If `'message'`, errors must extend `string | Array<{ message: string | RegExp }>`. +This can be done either by providing only a `string`/`RegExp` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). +If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. +If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. If `'messageId'`, errors must extend `Array<{ messageId: string }>`. ````ts From 9a07ded78154ccf6d503f544d3e8e8176b73e0c3 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Mon, 21 Jul 2025 20:07:50 +0200 Subject: [PATCH 09/22] chore: strike out requiredScenarios --- designs/2025-rule-tester-assertion-options/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index a5204db7..d764190e 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -290,7 +290,7 @@ it requires identifying the RuleTester calls in the codebase and might run into ## Open Questions -1. Is there a need for disabling scenarios like `valid` or `invalid`? +~~1. Is there a need for disabling scenarios like `valid` or `invalid`?~~ No, unused scenarios can be omitted using empty arrays. If needed, this option can be added later on. 2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` From ad0293ba7494e9e98b26f9ed79c29a94b44747ad Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Mon, 21 Jul 2025 20:12:06 +0200 Subject: [PATCH 10/22] chore: cleanup --- designs/2025-rule-tester-assertion-options/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index d764190e..fbe768fb 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -295,7 +295,6 @@ it requires identifying the RuleTester calls in the codebase and might run into 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` 5. Should we add a `strict` option that enables all assertion options by default? -6. ~~Should we expose `requiredScenarios` as option?~~ No, the users can just use empty arrays. ## Help Needed From b720c9d69b29b5f64636d55cf3e7352ba48a1d11 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Thu, 7 Aug 2025 22:44:10 +0200 Subject: [PATCH 11/22] chore: add PR link --- designs/2025-rule-tester-assertion-options/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index fbe768fb..78421f97 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -1,6 +1,6 @@ - Repo: https://github.com/eslint/eslint - Start Date: 2025-07-10 -- RFC PR: (leave this empty, to be filled in later) +- RFC PR: https://github.com/eslint/rfcs/pull/137 - Authors: ST-DDT # Rule Tester Assertions Options From eef730613cdcf64a6eeb174bbee2a0bd0423bc39 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Thu, 7 Aug 2025 22:44:54 +0200 Subject: [PATCH 12/22] chore: merge assertionOptions into test parameter --- .../README.md | 105 +++++++++++------- 1 file changed, 64 insertions(+), 41 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 78421f97..08fbbd3f 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -43,23 +43,27 @@ new RuleTester(testerConfig: {...}, assertionOptions: { ### Variant 2 - Test method based options ````ts -ruleTester.run("rule-name", rule, tests, assertionOptions: { - /** - * Require message assertion for each invalid test case. - * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. - * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. - * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. - * - * @default false - */ - requireMessage: boolean | 'message' | 'messageId'; - /** - * Require full location assertions for each invalid test case. - * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. - * - * @default false - */ - requireLocation: boolean; +ruleTester.run("rule-name", rule, { + assertionOptions?: { + /** + * Require message assertion for each invalid test case. + * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. + * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. + * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. + * + * @default false + */ + requireMessage?: boolean | 'message' | 'messageId'; + /** + * Require full location assertions for each invalid test case. + * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. + * + * @default false + */ + requireLocation?: boolean; + }, + valid: [ /* ... */ ], + invalid: [ /* ... */ ], }); ```` @@ -75,6 +79,9 @@ If `'messageId'`, errors must extend `Array<{ messageId: string }>`. ````ts ruleTester.run("rule-name", rule, { + assertionOptions: { + requireMessage: true + }, invalid: [ { code: "const a = 1;", @@ -95,8 +102,6 @@ ruleTester.run("rule-name", rule, { ] } ] -}, { - requireMessage: true }); ```` @@ -106,6 +111,9 @@ If `requireLocation` is set to `true`, the invalid test case's error validation ````ts ruleTester.run("rule-name", rule, { + assertionOptions: { + requireLocation: true + }, invalid: [ { code: "const a = 1;", @@ -139,8 +147,6 @@ ruleTester.run("rule-name", rule, { ] } ] -}, { - requireLocation: true }); ```` @@ -148,31 +154,48 @@ ruleTester.run("rule-name", rule, { ````patch diff --git a/lib/rule-tester/rule-tester.js b/lib/rule-tester/rule-tester.js -index dbd8c274f..412090df0 100644 +index dbd8c274f..c7a076021 100644 --- a/lib/rule-tester/rule-tester.js +++ b/lib/rule-tester/rule-tester.js -@@ -558,11 +558,19 @@ class RuleTester { +@@ -555,6 +555,10 @@ class RuleTester { + * @param {string} ruleName The name of the rule to run. + * @param {RuleDefinition} rule The rule to test. + * @param {{ ++ * assertionOptions?: { ++ * requireMessage?: boolean | "message" | "messageId", ++ * requireLocation?: boolean ++ * }, * valid: (ValidTestCase | string)[], * invalid: InvalidTestCase[] * }} test The collection of tests to run. -+ * @param {{ -+ * requireMessage: boolean | "message" | "messageId", -+ * requireLocation: boolean, -+ * }} [options] Options for the test. - * @throws {TypeError|Error} If `rule` is not an object with a `create` method, - * or if non-object `test`, or if a required scenario of the given type is missing. +@@ -563,6 +567,13 @@ class RuleTester { * @returns {void} */ -- run(ruleName, rule, test) { -+ run(ruleName, rule, test, options = {}) { -+ const { -+ requireMessage = false, -+ requireLocation = false, -+ } = options; + run(ruleName, rule, test) { ++ if (!test || typeof test !== "object") { ++ throw new TypeError( ++ `Test Scenarios for rule ${ruleName} : Could not find test scenario object`, ++ ); ++ } ++ ++ const { requireMessage = false, requireLocation = false } = test.assertionOptions ?? {}; const testerConfig = this.testerConfig, requiredScenarios = ["valid", "invalid"], scenarioErrors = [], -@@ -1092,6 +1100,10 @@ class RuleTester { +@@ -582,12 +593,6 @@ class RuleTester { + ); + } + +- if (!test || typeof test !== "object") { +- throw new TypeError( +- `Test Scenarios for rule ${ruleName} : Could not find test scenario object`, +- ); +- } +- + requiredScenarios.forEach(scenarioType => { + if (!test[scenarioType]) { + scenarioErrors.push( +@@ -1092,6 +1097,10 @@ class RuleTester { } if (typeof item.errors === "number") { @@ -183,18 +206,18 @@ index dbd8c274f..412090df0 100644 if (item.errors === 0) { assert.fail( "Invalid cases must have 'error' value greater than 0", -@@ -1137,6 +1149,10 @@ class RuleTester { +@@ -1137,6 +1146,10 @@ class RuleTester { if (typeof error === "string" || error instanceof RegExp) { // Just an error message. + assert.ok( -+ requireMessage !== 'messageId' && !requireLocation, ++ requireMessage !== "messageId" && !requireLocation, + "Invalid cases must have 'errors' value as an array of objects e.g. { message: 'error message' }", + ); assertMessageMatches(message.message, error); assert.ok( message.suggestions === void 0, -@@ -1157,6 +1173,10 @@ class RuleTester { +@@ -1157,6 +1170,10 @@ class RuleTester { }); if (hasOwnProperty(error, "message")) { @@ -205,7 +228,7 @@ index dbd8c274f..412090df0 100644 assert.ok( !hasOwnProperty(error, "messageId"), "Error should not specify both 'message' and a 'messageId'.", -@@ -1170,6 +1190,10 @@ class RuleTester { +@@ -1170,6 +1187,10 @@ class RuleTester { error.message, ); } else if (hasOwnProperty(error, "messageId")) { @@ -216,7 +239,7 @@ index dbd8c274f..412090df0 100644 assert.ok( ruleHasMetaMessages, "Error can not use 'messageId' if rule under test doesn't define 'meta.messages'.", -@@ -1242,21 +1266,37 @@ class RuleTester { +@@ -1242,21 +1263,37 @@ class RuleTester { if (hasOwnProperty(error, "line")) { actualLocation.line = message.line; expectedLocation.line = error.line; From 429e0a3bcc7273346d21fa1772673f8e6b954342 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Thu, 7 Aug 2025 22:51:38 +0200 Subject: [PATCH 13/22] chore: format file --- .../README.md | 183 +++++++++--------- 1 file changed, 91 insertions(+), 92 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 08fbbd3f..d7a2bf21 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -19,53 +19,53 @@ The options could be defined on two levels. On `RuleTester`'s `constructor` effe ### Variant 1 - Constructor based options -````ts +```ts new RuleTester(testerConfig: {...}, assertionOptions: { + /** + * Require message assertion for each invalid test case. + * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. + * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. + * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. + * + * @default false + */ + requireMessage: boolean | 'message' | 'messageId'; + /** + * Require full location assertions for each invalid test case. + * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. + * + * @default false + */ + requireLocation: boolean; +} = {}); +``` + +### Variant 2 - Test method based options + +```ts +ruleTester.run("rule-name", rule, { + assertionOptions?: { /** * Require message assertion for each invalid test case. * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. - * + * * @default false */ - requireMessage: boolean | 'message' | 'messageId'; + requireMessage?: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. - * + * * @default false */ - requireLocation: boolean; -} = {}); -```` - -### Variant 2 - Test method based options - -````ts -ruleTester.run("rule-name", rule, { - assertionOptions?: { - /** - * Require message assertion for each invalid test case. - * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. - * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. - * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. - * - * @default false - */ - requireMessage?: boolean | 'message' | 'messageId'; - /** - * Require full location assertions for each invalid test case. - * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. - * - * @default false - */ - requireLocation?: boolean; - }, - valid: [ /* ... */ ], - invalid: [ /* ... */ ], + requireLocation?: boolean; + }, + valid: [ /* ... */ ], + invalid: [ /* ... */ ], }); -```` +``` ### Shared Logic @@ -77,82 +77,81 @@ If `true`, errors must extend `string | RegExp | Array<{ message: string | RegEx If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. If `'messageId'`, errors must extend `Array<{ messageId: string }>`. -````ts +```ts ruleTester.run("rule-name", rule, { - assertionOptions: { - requireMessage: true + assertionOptions: { + requireMessage: true, + }, + invalid: [ + { + code: "const a = 1;", + errors: 1, // ❌ Message isn't being checked here }, - invalid: [ - { - code: "const a = 1;", - errors: 1, // ❌ Message isn't being checked here - }, + { + code: "const a = 2;", + errors: [ + "Error message here.", // ✅ we only check the error message + ], + }, + { + code: "const a = 3;", + errors: [ { - code: "const a = 2;", - errors: [ - "Error message here.", // ✅ we only check the error message - ] + message: "Error message here.", // ✅ we check the error message and potentially other properties }, - { - code: "const a = 3;", - errors: [ - { - message: "Error message here.", // ✅ we check the error message and potentially other properties - } - ] - } - ] + ], + }, + ], }); -```` +``` #### requireLocation If `requireLocation` is set to `true`, the invalid test case's error validation must include all location assertions such as `line`, `column`, `endLine`, and `endColumn`. -````ts +```ts ruleTester.run("rule-name", rule, { - assertionOptions: { - requireLocation: true + assertionOptions: { + requireLocation: true, + }, + invalid: [ + { + code: "const a = 1;", + errors: 1, // ❌ Location isn't checked here }, - invalid: [ - { - code: "const a = 1;", - errors: 1, // ❌ Location isn't checked here - }, + { + code: "const a = 2;", + errors: [ + "Error message here.", // ❌ Location isn't checked here + ], + }, + { + code: "const a = 3;", + errors: [ { - code: "const a = 2;", - errors: [ - "Error message here.", // ❌ Location isn't checked here - ] + line: 1, // ❌ Location isn't fully checked here + column: 1, }, + ], + }, + { + code: "const a = 4;", + errors: [ { - code: "const a = 3;", - errors: [ - { - line: 1, // ❌ Location isn't fully checked here - column: 1, - - } - ] + line: 1, // ✅ All location properties have been checked + column: 1, + endLine: 1, + endColumn: 12, }, - { - code: "const a = 4;", - errors: [ - { - line: 1, // ✅ All location properties have been checked - column: 1, - endLine: 1, - endColumn: 12, - } - ] - } - ] + ], + }, + ], }); -```` +``` ## Implementation Hint -````patch +```patch diff --git a/lib/rule-tester/rule-tester.js b/lib/rule-tester/rule-tester.js index dbd8c274f..c7a076021 100644 --- a/lib/rule-tester/rule-tester.js @@ -277,7 +276,7 @@ index dbd8c274f..c7a076021 100644 } if (Object.keys(expectedLocation).length > 0) { -```` +``` ## Documentation @@ -308,12 +307,12 @@ This change should not affect existing ESLint users or plugin developers, as it ## Alternatives As an alternative to this proposal, we could add a eslint rule that applies the same assertions, but uses the central eslint config. -While this would apply the same assertions for all rule testers, it would be a lot more complex to implement and maintain, +While this would apply the same assertions for all rule testers, it would be a lot more complex to implement and maintain, it requires identifying the RuleTester calls in the codebase and might run into issues if the assertions aren't specified inline but via a variable or transformation. ## Open Questions -~~1. Is there a need for disabling scenarios like `valid` or `invalid`?~~ No, unused scenarios can be omitted using empty arrays. If needed, this option can be added later on. +1. ~~Is there a need for disabling scenarios like `valid` or `invalid`?~~ No, unused scenarios can be omitted using empty arrays. If needed, this option can be added later on. 2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` From aa12816db70cbb8d7136a1884dd54c4f408d3c9e Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Thu, 21 Aug 2025 23:08:21 +0200 Subject: [PATCH 14/22] chore: apply review suggestions --- .../README.md | 36 ++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index d7a2bf21..bda6ee5d 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -32,7 +32,8 @@ new RuleTester(testerConfig: {...}, assertionOptions: { requireMessage: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. - * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. + * If `true`, errors must extend `Array<{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }>`. + * `endLine` or `endColumn` may be absent, if the observed error does not contain these properties. * * @default false */ @@ -56,7 +57,8 @@ ruleTester.run("rule-name", rule, { requireMessage?: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. - * If `true`, errors must extend `Array<{ line: number, column: number, endLine: number, endColumn: number }>`. + * If `true`, errors must extend `Array<{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }>`. + * `endLine` or `endColumn` may be absent or undefined, if the observed error does not contain these properties. * * @default false */ @@ -108,6 +110,7 @@ ruleTester.run("rule-name", rule, { #### requireLocation If `requireLocation` is set to `true`, the invalid test case's error validation must include all location assertions such as `line`, `column`, `endLine`, and `endColumn`. +`endLine` or `endColumn` may be absent or `undefined`, if the observed error does not contain these properties. ```ts ruleTester.run("rule-name", rule, { @@ -153,7 +156,7 @@ ruleTester.run("rule-name", rule, { ```patch diff --git a/lib/rule-tester/rule-tester.js b/lib/rule-tester/rule-tester.js -index dbd8c274f..c7a076021 100644 +index dbd8c274f..b39631faa 100644 --- a/lib/rule-tester/rule-tester.js +++ b/lib/rule-tester/rule-tester.js @@ -555,6 +555,10 @@ class RuleTester { @@ -167,7 +170,7 @@ index dbd8c274f..c7a076021 100644 * valid: (ValidTestCase | string)[], * invalid: InvalidTestCase[] * }} test The collection of tests to run. -@@ -563,6 +567,13 @@ class RuleTester { +@@ -563,6 +567,14 @@ class RuleTester { * @returns {void} */ run(ruleName, rule, test) { @@ -177,11 +180,12 @@ index dbd8c274f..c7a076021 100644 + ); + } + -+ const { requireMessage = false, requireLocation = false } = test.assertionOptions ?? {}; ++ const { requireMessage = false, requireLocation = false } = ++ test.assertionOptions ?? {}; const testerConfig = this.testerConfig, requiredScenarios = ["valid", "invalid"], scenarioErrors = [], -@@ -582,12 +593,6 @@ class RuleTester { +@@ -582,12 +594,6 @@ class RuleTester { ); } @@ -194,7 +198,7 @@ index dbd8c274f..c7a076021 100644 requiredScenarios.forEach(scenarioType => { if (!test[scenarioType]) { scenarioErrors.push( -@@ -1092,6 +1097,10 @@ class RuleTester { +@@ -1092,6 +1098,10 @@ class RuleTester { } if (typeof item.errors === "number") { @@ -205,7 +209,7 @@ index dbd8c274f..c7a076021 100644 if (item.errors === 0) { assert.fail( "Invalid cases must have 'error' value greater than 0", -@@ -1137,6 +1146,10 @@ class RuleTester { +@@ -1137,6 +1147,10 @@ class RuleTester { if (typeof error === "string" || error instanceof RegExp) { // Just an error message. @@ -216,7 +220,7 @@ index dbd8c274f..c7a076021 100644 assertMessageMatches(message.message, error); assert.ok( message.suggestions === void 0, -@@ -1157,6 +1170,10 @@ class RuleTester { +@@ -1157,6 +1171,10 @@ class RuleTester { }); if (hasOwnProperty(error, "message")) { @@ -227,7 +231,7 @@ index dbd8c274f..c7a076021 100644 assert.ok( !hasOwnProperty(error, "messageId"), "Error should not specify both 'message' and a 'messageId'.", -@@ -1170,6 +1187,10 @@ class RuleTester { +@@ -1170,6 +1188,10 @@ class RuleTester { error.message, ); } else if (hasOwnProperty(error, "messageId")) { @@ -238,7 +242,7 @@ index dbd8c274f..c7a076021 100644 assert.ok( ruleHasMetaMessages, "Error can not use 'messageId' if rule under test doesn't define 'meta.messages'.", -@@ -1242,21 +1263,37 @@ class RuleTester { +@@ -1242,21 +1264,43 @@ class RuleTester { if (hasOwnProperty(error, "line")) { actualLocation.line = message.line; expectedLocation.line = error.line; @@ -260,7 +264,10 @@ index dbd8c274f..c7a076021 100644 if (hasOwnProperty(error, "endLine")) { actualLocation.endLine = message.endLine; expectedLocation.endLine = error.endLine; -+ } else if (requireLocation) { ++ } else if ( ++ requireLocation && ++ hasOwnProperty(message, "endLine") ++ ) { + assert.fail( + "Test error must specify an 'endLine' property.", + ); @@ -269,7 +276,10 @@ index dbd8c274f..c7a076021 100644 if (hasOwnProperty(error, "endColumn")) { actualLocation.endColumn = message.endColumn; expectedLocation.endColumn = error.endColumn; -+ } else if (requireLocation) { ++ } else if ( ++ requireLocation && ++ hasOwnProperty(message, "endColumn") ++ ) { + assert.fail( + "Test error must specify an 'endColumn' property.", + ); From 55b1ec767b32b5852a2e9ddd80726f2fa5a3eb4c Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sun, 14 Sep 2025 22:27:53 +0200 Subject: [PATCH 15/22] docs: apply suggestion --- designs/2025-rule-tester-assertion-options/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index bda6ee5d..cb5ac9f7 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -326,7 +326,7 @@ it requires identifying the RuleTester calls in the codebase and might run into 2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` -5. Should we add a `strict` option that enables all assertion options by default? +5. ~~Should we add a `strict` option that enables all assertion options by default?~~ No, currently not planned. ## Help Needed From ae0c11d1deb5255db04e7a2b92875e4478c68a34 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sun, 14 Sep 2025 22:33:28 +0200 Subject: [PATCH 16/22] docs: apply suggestion --- designs/2025-rule-tester-assertion-options/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index cb5ac9f7..477f7bcf 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -323,10 +323,12 @@ it requires identifying the RuleTester calls in the codebase and might run into ## Open Questions 1. ~~Is there a need for disabling scenarios like `valid` or `invalid`?~~ No, unused scenarios can be omitted using empty arrays. If needed, this option can be added later on. -2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? +2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? Currently, the method level options (variant 2) is the prefered one. 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` 5. ~~Should we add a `strict` option that enables all assertion options by default?~~ No, currently not planned. +6. ~~Should we inline the options?~~ No, this might cause issues with alphabetical object properties. +7. ~~What should the otpions be called?~~ We name it `assertionOptions` to avoid issues with alphabetical object properties. e.g. `invalid, options, valid` ## Help Needed From ab96b3130b07e9691bc638214ff12794201bf7dc Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sun, 14 Sep 2025 22:42:15 +0200 Subject: [PATCH 17/22] docs: apply suggestion --- designs/2025-rule-tester-assertion-options/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 477f7bcf..6bba7e6a 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -320,6 +320,10 @@ As an alternative to this proposal, we could add a eslint rule that applies the While this would apply the same assertions for all rule testers, it would be a lot more complex to implement and maintain, it requires identifying the RuleTester calls in the codebase and might run into issues if the assertions aren't specified inline but via a variable or transformation. +The following issue makes it easier to identify the exact/estimated error location (outside of/more precise than `RuleTester#run`): + +- https://github.com/eslint/eslint/issues/19936 + ## Open Questions 1. ~~Is there a need for disabling scenarios like `valid` or `invalid`?~~ No, unused scenarios can be omitted using empty arrays. If needed, this option can be added later on. From e378647d5a07c9f6d46aaa9923f2845808bf732c Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sun, 14 Sep 2025 22:46:45 +0200 Subject: [PATCH 18/22] docs: apply suggestion --- designs/2025-rule-tester-assertion-options/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 6bba7e6a..be1618dd 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -17,6 +17,8 @@ The options could be defined on two levels. On `RuleTester`'s `constructor` effe ## Detailed Design +Recommended variant => 2 - Test method based options + ### Variant 1 - Constructor based options ```ts From 5f137b0345ef79a3f5b53553f3497bb69f3c63bf Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Sun, 14 Sep 2025 22:52:45 +0200 Subject: [PATCH 19/22] docs: apply parameter description suggestion --- .../README.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index be1618dd..9aedf201 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -25,16 +25,16 @@ Recommended variant => 2 - Test method based options new RuleTester(testerConfig: {...}, assertionOptions: { /** * Require message assertion for each invalid test case. - * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. - * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. - * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. + * If `true`, each error must extend `string | RegExp | { message: string | RegExp } | { messageId: string }`. + * If `'message'`, each error must extend `string | RegExp | { message: string | RegExp }`. + * If `'messageId'`, each error must extend `{ messageId: string }`. * * @default false */ requireMessage: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. - * If `true`, errors must extend `Array<{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }>`. + * If `true`, each error must extend `{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }`. * `endLine` or `endColumn` may be absent, if the observed error does not contain these properties. * * @default false @@ -50,16 +50,16 @@ ruleTester.run("rule-name", rule, { assertionOptions?: { /** * Require message assertion for each invalid test case. - * If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. - * If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. - * If `'messageId'`, errors must extend `Array<{ messageId: string }>`. + * If `true`, each error must extend `string | RegExp | { message: string | RegExp } | { messageId: string }`. + * If `'message'`, each error must extend `string | RegExp | { message: string | RegExp }`. + * If `'messageId'`, each error must extend `{ messageId: string }`. * * @default false */ requireMessage?: boolean | 'message' | 'messageId'; /** * Require full location assertions for each invalid test case. - * If `true`, errors must extend `Array<{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }>`. + * If `true`, each error must extend `{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }`. * `endLine` or `endColumn` may be absent or undefined, if the observed error does not contain these properties. * * @default false @@ -77,9 +77,9 @@ ruleTester.run("rule-name", rule, { If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. (See below) This can be done either by providing only a `string`/`RegExp` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). -If `true`, errors must extend `string | RegExp | Array<{ message: string | RegExp } | { messageId: string }>`. -If `'message'`, errors must extend `string | RegExp | Array<{ message: string | RegExp }>`. -If `'messageId'`, errors must extend `Array<{ messageId: string }>`. +If `true`, each error must extend `string | RegExp | { message: string | RegExp } | { messageId: string }`. +If `'message'`, each error must extend `string | RegExp | { message: string | RegExp }`. +If `'messageId'`, each error must extend `{ messageId: string }`. ```ts ruleTester.run("rule-name", rule, { From c3a09c4180a23e3f2557b9f6484518501d325bb6 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Fri, 19 Sep 2025 20:37:20 +0200 Subject: [PATCH 20/22] chore: move variant to alternatives --- .../README.md | 64 +++++++++---------- 1 file changed, 31 insertions(+), 33 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 9aedf201..8d64e3f4 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -17,34 +17,6 @@ The options could be defined on two levels. On `RuleTester`'s `constructor` effe ## Detailed Design -Recommended variant => 2 - Test method based options - -### Variant 1 - Constructor based options - -```ts -new RuleTester(testerConfig: {...}, assertionOptions: { - /** - * Require message assertion for each invalid test case. - * If `true`, each error must extend `string | RegExp | { message: string | RegExp } | { messageId: string }`. - * If `'message'`, each error must extend `string | RegExp | { message: string | RegExp }`. - * If `'messageId'`, each error must extend `{ messageId: string }`. - * - * @default false - */ - requireMessage: boolean | 'message' | 'messageId'; - /** - * Require full location assertions for each invalid test case. - * If `true`, each error must extend `{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }`. - * `endLine` or `endColumn` may be absent, if the observed error does not contain these properties. - * - * @default false - */ - requireLocation: boolean; -} = {}); -``` - -### Variant 2 - Test method based options - ```ts ruleTester.run("rule-name", rule, { assertionOptions?: { @@ -71,9 +43,7 @@ ruleTester.run("rule-name", rule, { }); ``` -### Shared Logic - -#### requireMessage +### requireMessage If `requireMessage` is set to `true`, the invalid test case cannot consist of an error count assertion only, but must also include a message assertion. (See below) This can be done either by providing only a `string`/`RegExp` message, or by using the `message`/`messageId` property of the error object in the `TestCaseError` (Same as the current behavior). @@ -109,7 +79,7 @@ ruleTester.run("rule-name", rule, { }); ``` -#### requireLocation +### requireLocation If `requireLocation` is set to `true`, the invalid test case's error validation must include all location assertions such as `line`, `column`, `endLine`, and `endColumn`. `endLine` or `endColumn` may be absent or `undefined`, if the observed error does not contain these properties. @@ -318,10 +288,38 @@ This change should not affect existing ESLint users or plugin developers, as it ## Alternatives +### Constructor based options + +```ts +new RuleTester(testerConfig: {...}, assertionOptions: { + /** + * Require message assertion for each invalid test case. + * If `true`, each error must extend `string | RegExp | { message: string | RegExp } | { messageId: string }`. + * If `'message'`, each error must extend `string | RegExp | { message: string | RegExp }`. + * If `'messageId'`, each error must extend `{ messageId: string }`. + * + * @default false + */ + requireMessage: boolean | 'message' | 'messageId'; + /** + * Require full location assertions for each invalid test case. + * If `true`, each error must extend `{ line: number, column: number, endLine?: number | undefined, endColumn?: number | undefined }`. + * `endLine` or `endColumn` may be absent, if the observed error does not contain these properties. + * + * @default false + */ + requireLocation: boolean; +} = {}); +``` + +### ESLint RuleTester Lint-Rule + As an alternative to this proposal, we could add a eslint rule that applies the same assertions, but uses the central eslint config. While this would apply the same assertions for all rule testers, it would be a lot more complex to implement and maintain, it requires identifying the RuleTester calls in the codebase and might run into issues if the assertions aren't specified inline but via a variable or transformation. +### RuleTester Estimated Error Location + The following issue makes it easier to identify the exact/estimated error location (outside of/more precise than `RuleTester#run`): - https://github.com/eslint/eslint/issues/19936 @@ -329,7 +327,7 @@ The following issue makes it easier to identify the exact/estimated error locati ## Open Questions 1. ~~Is there a need for disabling scenarios like `valid` or `invalid`?~~ No, unused scenarios can be omitted using empty arrays. If needed, this option can be added later on. -2. Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files? Currently, the method level options (variant 2) is the prefered one. +2. ~~Should we use constructor-based options or test method-based options? Do we support both? Or global options so it applies to all test files?~~ Currently, the method level options (variant 2) is the prefered one. 3. ~~Should we enable the `requireMessage` and `requireLocation` options by default? (Breaking change)~~ No 4. ~~Do we add a `requireMessageId` option or should we alter the `requireMessage` option to support both message and messageId assertions?~~ Just `requireMessage: boolean | 'message' | 'messageid'` 5. ~~Should we add a `strict` option that enables all assertion options by default?~~ No, currently not planned. From e3aff7b61acc9cc3f6e28e1f8af0a5c07e0cf3b9 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Fri, 19 Sep 2025 20:41:29 +0200 Subject: [PATCH 21/22] chore: list 3rd alternative --- designs/2025-rule-tester-assertion-options/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 8d64e3f4..718656f8 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -312,6 +312,10 @@ new RuleTester(testerConfig: {...}, assertionOptions: { } = {}); ``` +### Options Setter Method + +Alternatively add an `setAssertionOptions()` method independently of the constructor and test method. + ### ESLint RuleTester Lint-Rule As an alternative to this proposal, we could add a eslint rule that applies the same assertions, but uses the central eslint config. From 50b73234ca6a3793f215cdf43bf24f35ed0e6e43 Mon Sep 17 00:00:00 2001 From: ST-DDT Date: Mon, 22 Sep 2025 09:24:57 +0200 Subject: [PATCH 22/22] Update designs/2025-rule-tester-assertion-options/README.md Co-authored-by: Francesco Trotta --- designs/2025-rule-tester-assertion-options/README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/designs/2025-rule-tester-assertion-options/README.md b/designs/2025-rule-tester-assertion-options/README.md index 718656f8..7c18029d 100644 --- a/designs/2025-rule-tester-assertion-options/README.md +++ b/designs/2025-rule-tester-assertion-options/README.md @@ -276,9 +276,6 @@ While this is true already, it gets slightly more relevant, since now it might r See also: https://github.com/eslint/eslint/issues/19936 -Additionally, since we add the options as a second parameter it might interfere with future additions to the parameters. -This could by eleviated by renaming the parameter from `assertionOptions` to `options` (either from the start or when the need for different type of options arises). - If we enable the `requireMessage` and `requireLocation` options by default, it would be a breaking change for existing tests that do not follow these assertion requirements yet. It is not planned to enable `requireMessage` or `requireLocation` by default.