Skip to content

Add scenario reports and artifact policies#20

Merged
slideep merged 2 commits into
mainfrom
codex/scenario-reports
May 19, 2026
Merged

Add scenario reports and artifact policies#20
slideep merged 2 commits into
mainfrom
codex/scenario-reports

Conversation

@slideep
Copy link
Copy Markdown
Contributor

@slideep slideep commented May 19, 2026

Summary

  • add JSON and JUnit scenario run reports for run --file and run --path
  • add artifact attachment and failure-capture policy controls
  • preserve deterministic scenario IDs, shard metadata, terminal events, and report output across planning and execution failures
  • cover recursive glob discovery, hash sharding, JUnit artifact output, report failure shapes, and malformed scenario validation

Validation

  • dotnet test Luotsi.Cli.Tests\Luotsi.Cli.Tests.csproj
  • dotnet build -c Release
  • git diff --check

Copilot AI review requested due to automatic review settings May 19, 2026 07:01
Comment thread Luotsi.Cli/Scenarios/ScenarioExecutor.cs Fixed
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class scenario run reporting (JSON + JUnit) and introduces new controls for failure artifact capture and report artifact attachment, while extending scenario path resolution (recursive globs) and sharding (stable hash strategy). This fits into the CLI scenario execution pipeline by wrapping planning/execution with report/event coordinators and enriching scenario identity metadata for correlation.

Changes:

  • Add JSON and JUnit scenario run reports for run --file / run --path, including failure-shape preservation across planning/execution errors.
  • Add artifact policy controls: --capture-on (failure artifacts generation) and --attach-artifacts (report attachment policy), plus deterministic scenario_id propagation in events/results.
  • Add recursive glob discovery (**/*.json) and --shard-strategy index|hash for stable sharding.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
Luotsi.Cli/Scenarios/ScenarioValidator.cs Adjusts validation order to reject empty/whitespace actions before trimming.
Luotsi.Cli/Scenarios/ScenarioReports.cs Introduces report models + writers (JSON + JUnit) and report coordination around scenario planning/execution.
Luotsi.Cli/Scenarios/ScenarioExecutorFactory.cs Threads failure artifact capture policy into ScenarioExecutor creation.
Luotsi.Cli/Scenarios/ScenarioExecutor.cs Adds deterministic scenario_id to events/results and makes failure artifact capture best-effort + policy-controlled.
Luotsi.Cli/Scenarios/ScenarioEvents.cs Adds shard strategy metadata and ensures terminal run events are emitted even on planning failures.
Luotsi.Cli/Scenarios/ScenarioCatalog.cs Adds recursive glob resolution, shard strategy support (index/hash), and stable shard hashing.
Luotsi.Cli/Scenarios/ScenarioBatchExecutorFactory.cs Threads failure artifact capture policy into batch executor creation.
Luotsi.Cli/Scenarios/ScenarioBatchExecutor.cs Propagates catalog metadata (file/id) into batch item results and includes shard strategy in batch result.
Luotsi.Cli/Cli/Routing/ScenarioQueryFactory.cs Parses/normalizes --shard-strategy into ScenarioQuery.
Luotsi.Cli/Cli/Routing/ScenarioCommandDispatcher.cs Wires up report coordinator, capture/attach policies, and planning-failure event/report flows for run.
Luotsi.Cli/Cli/Help.cs Documents new run options and glob-capable --path.
Luotsi.Cli/Cli/Composition/AppHostedCommandCompositionBuilder.cs Registers ScenarioRunReportCoordinatorFactory in composition root.
Luotsi.Cli.Tests/TestSupport.cs Enhances FakeDeviceHost to record/throw failure artifact capture for policy/capture-failure tests.
Luotsi.Cli.Tests/ScenarioExecutorTests.cs Adds coverage for recursive globs, report outputs (JSON/JUnit), planning/execution failure report shapes, artifact policies, and hash sharding.

Comment thread Luotsi.Cli/Scenarios/ScenarioCatalog.cs
Comment thread Luotsi.Cli/Scenarios/ScenarioReports.cs Outdated
@slideep slideep merged commit 2a01c76 into main May 19, 2026
12 checks passed
@slideep slideep deleted the codex/scenario-reports branch May 19, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants