Summary
kane-cli testmd run has no data-driven mode. --variables / --variables-file binds a single variable set per session, and every run uploads to one test case. There is no way to say "run this _test.md across N rows and record each row as an iteration under a single test case." Covering any combinatorial / pairwise / boundary matrix therefore has no clean path — both available workarounds are bad, and neither produces the Test Manager iterations view (one case, a bound dataset, each row an iteration).
Environment
kane-cli 0.4.10 · Node 24 · Windows 11 · account toddl
Current behavior — both workarounds are bad
A) Stable --name + committed output-<stem>/ cache, re-run per row
Replays the identical recorded flow. It can't vary the row's data (replay freezes the assertion RHS to row 1), and it breaks the moment two rows reach structurally different result pages. Real example from our AUT: an underwriting decision renders an offer amount for Approve/Counter-Offer but not for Decline/RFAI, so a single cached authoring cannot replay across rows.
B) Clear the cache (rm -rf output-*) per row to force fresh authoring
Fresh authoring adapts to each row's page shape — but deleting output-* orphans the meta.json testcase_id, so each row fans out into its own same-named test case. A 14-row matrix becomes 14 duplicate cases cluttering the folder. (This is the coupling reported in #130.)
So today the choice is: replay-of-identical (one case, but can't vary data or handle divergent pages) or fan-out (varies data, but N duplicate cases). There is no one case, N parameterized iterations path.
Requested behavior
A first-class data-driven mode, e.g.:
kane-cli testmd run flow_test.md --dataset rows.json --agent
where rows.json is an array of variable sets:
[
{ "credit_score": {"value": "660"}, "monthly_debt": {"value": "2000"}, "expected_decision": {"value": "Counter-Offer"} },
{ "credit_score": {"value": "540"}, "monthly_debt": {"value": "500"}, "expected_decision": {"value": "Decline"} }
]and each row is committed as an iteration under a single testcase_id — re-authoring per row when the page shape diverges, but preserving the one-case mapping. The Test Manager case would then show N iterations (pass/fail per row) instead of N separate cases.
Relationship to #130
#130 asks for a re-author-in-place flag so that forcing fresh authoring does not mint a new testcase_id. That is the necessary primitive; this issue is the feature built on top of it: bind a dataset to one _test.md and record each row as an iteration. #130 fixes the identity coupling for a single re-run; this adds the dataset/iteration surface for many parameterized runs.
Why it matters
Data-driven / combinatorial coverage is the standard way to test decision engines, pricing tiers, and boundary rules. Reproducible example: a 14-row pairwise set for an underwriting decision (216 combinations reduced to 14 at full pairwise coverage) currently fans out into 14 same-named cases; it should land as one case with 14 iterations. This is the single change that would make combinatorial test design land cleanly in Test Manager via the CLI.
Summary
kane-cli testmd runhas no data-driven mode.--variables/--variables-filebinds a single variable set per session, and every run uploads to one test case. There is no way to say "run this_test.mdacross N rows and record each row as an iteration under a single test case." Covering any combinatorial / pairwise / boundary matrix therefore has no clean path — both available workarounds are bad, and neither produces the Test Manager iterations view (one case, a bound dataset, each row an iteration).Environment
kane-cli
0.4.10· Node24· Windows 11 · accounttoddlCurrent behavior — both workarounds are bad
A) Stable
--name+ committedoutput-<stem>/cache, re-run per rowReplays the identical recorded flow. It can't vary the row's data (replay freezes the assertion RHS to row 1), and it breaks the moment two rows reach structurally different result pages. Real example from our AUT: an underwriting decision renders an offer amount for
Approve/Counter-Offerbut not forDecline/RFAI, so a single cached authoring cannot replay across rows.B) Clear the cache (
rm -rf output-*) per row to force fresh authoringFresh authoring adapts to each row's page shape — but deleting
output-*orphans themeta.jsontestcase_id, so each row fans out into its own same-named test case. A 14-row matrix becomes 14 duplicate cases cluttering the folder. (This is the coupling reported in #130.)So today the choice is: replay-of-identical (one case, but can't vary data or handle divergent pages) or fan-out (varies data, but N duplicate cases). There is no one case, N parameterized iterations path.
Requested behavior
A first-class data-driven mode, e.g.:
where
rows.jsonis an array of variable sets:[ { "credit_score": {"value": "660"}, "monthly_debt": {"value": "2000"}, "expected_decision": {"value": "Counter-Offer"} }, { "credit_score": {"value": "540"}, "monthly_debt": {"value": "500"}, "expected_decision": {"value": "Decline"} } ]and each row is committed as an iteration under a single
testcase_id— re-authoring per row when the page shape diverges, but preserving the one-case mapping. The Test Manager case would then show N iterations (pass/fail per row) instead of N separate cases.Relationship to #130
#130 asks for a re-author-in-place flag so that forcing fresh authoring does not mint a new
testcase_id. That is the necessary primitive; this issue is the feature built on top of it: bind a dataset to one_test.mdand record each row as an iteration. #130 fixes the identity coupling for a single re-run; this adds the dataset/iteration surface for many parameterized runs.Why it matters
Data-driven / combinatorial coverage is the standard way to test decision engines, pricing tiers, and boundary rules. Reproducible example: a 14-row pairwise set for an underwriting decision (216 combinations reduced to 14 at full pairwise coverage) currently fans out into 14 same-named cases; it should land as one case with 14 iterations. This is the single change that would make combinatorial test design land cleanly in Test Manager via the CLI.