Conversation
…ema support; external suite configuration; better obfuscator patterns; more convenient run-policy
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enhances the tools/e2e-tester tool by adding stronger configuration validation, JSON-schema support for editor assistance, external suite-file loading, improved obfuscation patterns, and a more flexible run_policy mechanism replacing must_pass.
Changes:
- Introduce
run_policy(normal|critical|always) and update runner + console reporting behavior accordingly. - Add config validation (go-playground/validator), external suite-file loading (
suite.filepath), and new/updated examples + docs. - Generate and commit JSON Schemas (
schemas/) plus ago:generateentry and tool dependency for schema generation.
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/e2e-tester/tools/generate.go | Adds go:generate invocation for JSON schema generation. |
| tools/e2e-tester/schemas/suitecontent.schema.json.license | License metadata for generated suite content schema. |
| tools/e2e-tester/schemas/suitecontent.schema.json | Generated schema for external suite content files. |
| tools/e2e-tester/schemas/config.schema.json.license | License metadata for generated config schema. |
| tools/e2e-tester/schemas/config.schema.json | Generated schema for root config file. |
| tools/e2e-tester/pkg/snapshot/manager.go | Fixes log field name typo (expected). |
| tools/e2e-tester/pkg/runner/suite.go | Implements run_policy execution semantics and improves snapshot-not-found handling. |
| tools/e2e-tester/pkg/runner/runner.go | Resolves environment-provided variables and updates “critical failure” logic to use run_policy. |
| tools/e2e-tester/pkg/report/reporter.go | Replaces must_pass reporting with run_policy messaging and final report indicators. |
| tools/e2e-tester/pkg/obfuscator/patterns_test.go | Updates expected obfuscation output for “generated … ago” pattern. |
| tools/e2e-tester/pkg/obfuscator/patterns.go | Adds additional obfuscation patterns (e.g., trd_, uid, traceId). |
| tools/e2e-tester/pkg/environment/manager.go | Adds variable resolution from env (env://…) and updates token prefix handling. |
| tools/e2e-tester/pkg/config/validator_test.go | Adds extensive validation + suite-loading test coverage. |
| tools/e2e-tester/pkg/config/validator.go | Introduces struct-tag validation and custom validators (run_policy, snapshotter, suite, cross-refs). |
| tools/e2e-tester/pkg/config/testdata/valid-external-suite.yaml | Test fixture for valid external suite file. |
| tools/e2e-tester/pkg/config/testdata/invalid-external-suite.yaml | Test fixture for invalid external suite file. |
| tools/e2e-tester/pkg/config/config.go | Adds RunPolicy, external suite loading, and validation tags on config structs. |
| tools/e2e-tester/pkg/command/executor_test.go | Ensures snapshots include the environment name in their ID. |
| tools/e2e-tester/pkg/command/executor.go | Adds support for environment variables in command execution and constants. |
| tools/e2e-tester/main.go | Adds run/verify subcommands, validates config before/after external suite loading. |
| tools/e2e-tester/go.sum | Updates dependency hashes for new validator/schema tooling. |
| tools/e2e-tester/go.mod | Adds validator + schema-gen tool dependencies; updates Go version directive. |
| tools/e2e-tester/examples/basic-suite.yaml | Adds example external suite file with schema header + run_policy usage. |
| tools/e2e-tester/examples/basic-config.yaml | Updates example config to use schema header, env://, and external suite file reference. |
| tools/e2e-tester/README.md | Updates docs for env://, external suites, JSON schema usage, and run_policy. |
| tools/e2e-tester/.gitignore | Adds tool-local ignore rules for binaries, env files, snapshots, IDE files, etc. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+60
to
+64
| { | ||
| Pattern: "uid.*", | ||
| Replace: "uid_OBFUSCATED", | ||
| }, | ||
| { |
| Replace: "uid_OBFUSCATED", | ||
| }, | ||
| { | ||
| Pattern: "traceId.*", |
Comment on lines
+167
to
+172
| "type": "object", | ||
| "required": [ | ||
| "roverctl", | ||
| "environments", | ||
| "suites" | ||
| ], |
Comment on lines
59
to
+63
| // Set up environment variables | ||
| if e.environment.Token != "" { | ||
| cmd.Env = append(cmd.Env, fmt.Sprintf("ROVER_TOKEN=%s", e.environment.Token)) | ||
| cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", roverTokenEnvKey, e.environment.Token)) | ||
| } | ||
|
|
Comment on lines
+133
to
+136
| filepath := s.Filepath | ||
| if !path.IsAbs(filepath) { | ||
| filepath = path.Join(configDir, s.Filepath) | ||
| } |
Comment on lines
+157
to
+162
| hasFilepath := suite.Filepath != "" | ||
| hasCases := len(suite.Cases) > 0 | ||
|
|
||
| if hasFilepath && hasCases { | ||
| sl.ReportError(suite.Filepath, "filepath", "Filepath", | ||
| "filepath_cases_exclusive", "") |
| environments: | ||
| - name: "team-a" # Unique name for the environment | ||
| token: "env:TEAM_A_TOKEN" # Token for authentication, can use env variable with "env:" prefix | ||
| token: "env://TEAM_A_TOKEN" # Token for authentication, can use env variable with "env:" prefix |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.