[evals] Adding in some vally testing for some basic azd scenarios#9282
Open
richardpark-msft wants to merge 20 commits into
Open
[evals] Adding in some vally testing for some basic azd scenarios#9282richardpark-msft wants to merge 20 commits into
richardpark-msft wants to merge 20 commits into
Conversation
added 3 commits
July 22, 2026 16:36
Some evals/tests just didn't make the cut - either they were redundant with our normal unit tests (there were several that were checking help content), or they were evals that seemed to be eval'ing themselves (ie, the answer was encoded in the system prompt). I've reduced this down to two buckets: - eval-azd-troubleshooting.yaml, which is a series of prompts that just see what baseline copilot models can provide without any skills or extra materials. It's there mostly just to keep tabs on things and see if the advice changes in an interesting way, or if we start getting bad ad-hoc advice in chat sessions. - eval-azd-deploy.yaml - this is a suite I plan on expanding - right now I've got the node app test uncommented, and I plan on slowly introducing the others once we get this bootstrappable/trackable in some system (TBD, since there's a lot of moving parts/in-flight efforts here).
…s for some common scenarios - troubleshooting (ie, no azd around, but what does the LLM say?) and deploy, which will actually have some azd knowledge and is being asked - how do I deploy this bare-bones app, and other similar questions.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Replaces the legacy Waza evaluation framework with Vally-based azd scenarios, experiments, reporting, fixtures, and CI integration.
Changes:
- Adds Vally troubleshooting, deployment, and skill-baseline evaluations.
- Adds fixtures and a Markdown report generator.
- Removes legacy Waza tests, graders, and pipelines.
Show a summary per file
| File | Description |
|---|---|
eng/pipelines/eval-waza.yml |
Removes scheduled Waza evaluations. |
eng/pipelines/eval-unit.yml |
Removes legacy ADO unit-test pipeline. |
eng/pipelines/eval-report.yml |
Removes Waza report pipeline. |
eng/pipelines/eval-e2e.yml |
Removes Waza E2E pipeline. |
cli/azd/test/evals/skills/azd/SKILL.md |
Adds evaluation-only azd guidance. |
cli/azd/test/evals/README.md |
Documents Vally suites and commands. |
cli/azd/test/evals/package.json |
Adds suite and reporting scripts. |
cli/azd/test/evals/make-vally-report.go |
Generates Markdown evaluation reports. |
cli/azd/test/evals/fixtures/existing-node-api/package.json |
Defines the Node API fixture. |
cli/azd/test/evals/fixtures/existing-node-api/index.js |
Implements the fixture API. |
cli/azd/test/evals/fixtures/azd-project/staging.env |
Adds staging fixture values. |
cli/azd/test/evals/fixtures/azd-project/production.env |
Adds production fixture values. |
cli/azd/test/evals/fixtures/azd-project/dev.env |
Adds development fixture values. |
cli/azd/test/evals/fixtures/azd-project/config-production.json |
Selects the production fixture environment. |
cli/azd/test/evals/fixtures/azd-project/config-dev.json |
Selects the development fixture environment. |
cli/azd/test/evals/fixtures/azd-project/azure.yaml |
Defines the sample azd project. |
cli/azd/test/evals/eval-azd.experiments.yaml |
Compares skill and model variants. |
cli/azd/test/evals/eval-azd-troubleshooting.yaml |
Adds baseline troubleshooting evaluations. |
cli/azd/test/evals/eval-azd-deploy.yaml |
Adds deployment and lifecycle evaluations. |
cli/azd/test/evals/.gitignore |
Ignores Vally outputs and reports. |
cli/azd/test/eval/tsconfig.json |
Removes legacy TypeScript configuration. |
cli/azd/test/eval/tests/unit/help-text-quality.test.ts |
Removes Waza-era help tests. |
cli/azd/test/eval/tests/unit/flag-validation.test.ts |
Removes legacy flag tests. |
cli/azd/test/eval/tests/unit/command-sequencing.test.ts |
Removes legacy sequencing tests. |
cli/azd/test/eval/tests/unit/command-registry.test.ts |
Removes legacy registry tests. |
cli/azd/test/eval/tests/test-utils.ts |
Removes the old CLI test helper. |
cli/azd/test/eval/tests/human/error-recovery.test.ts |
Removes human error-recovery tests. |
cli/azd/test/eval/tests/human/command-discovery.test.ts |
Removes command-discovery tests. |
cli/azd/test/eval/tests/human/cli-workflow.test.ts |
Removes CLI workflow tests. |
cli/azd/test/eval/tasks/troubleshoot/quota-error.yaml |
Removes the Waza quota scenario. |
cli/azd/test/eval/tasks/troubleshoot/provision-role-conflict.yaml |
Removes the role-conflict scenario. |
cli/azd/test/eval/tasks/troubleshoot/config-error.yaml |
Removes the Waza config scenario. |
cli/azd/test/eval/tasks/troubleshoot/auth-error.yaml |
Removes the Waza authentication scenario. |
cli/azd/test/eval/tasks/negative/raw-azure-cli.yaml |
Removes the raw Azure CLI scenario. |
cli/azd/test/eval/tasks/negative/not-azure.yaml |
Removes the non-Azure scenario. |
cli/azd/test/eval/tasks/negative/general-coding.yaml |
Removes the general coding scenario. |
cli/azd/test/eval/tasks/lifecycle/teardown-only.yaml |
Removes the Waza teardown scenario. |
cli/azd/test/eval/tasks/lifecycle/full-lifecycle.yaml |
Removes the full lifecycle scenario. |
cli/azd/test/eval/tasks/environment/switch-env.yaml |
Removes the environment-switch scenario. |
cli/azd/test/eval/tasks/environment/delete-env.yaml |
Removes the environment-delete scenario. |
cli/azd/test/eval/tasks/environment/create-staging.yaml |
Removes the staging scenario. |
cli/azd/test/eval/tasks/deploy/deploy-python-webapp.yaml |
Removes the Python deployment scenario. |
cli/azd/test/eval/tasks/deploy/deploy-node-api.yaml |
Removes the Node deployment scenario. |
cli/azd/test/eval/tasks/deploy/deploy-existing-project.yaml |
Removes the existing-project scenario. |
cli/azd/test/eval/reports/.gitkeep |
Removes the legacy report placeholder. |
cli/azd/test/eval/README.md |
Removes Waza framework documentation. |
cli/azd/test/eval/package.json |
Removes Waza/Jest dependencies and scripts. |
cli/azd/test/eval/jest.config.ts |
Removes Jest configuration. |
cli/azd/test/eval/graders/test_graders.py |
Removes Python grader tests. |
cli/azd/test/eval/graders/infra_validator.py |
Removes infrastructure grader. |
cli/azd/test/eval/graders/cleanup_validator.py |
Removes cleanup grader. |
cli/azd/test/eval/graders/azure_auth.py |
Removes grader authentication helper. |
cli/azd/test/eval/graders/app_health.py |
Removes application-health grader. |
cli/azd/test/eval/eval.yaml |
Removes the Waza configuration. |
cli/azd/test/eval/.gitignore |
Removes legacy ignore rules. |
cli/azd/.vscode/cspell.yaml |
Removes obsolete Waza spelling overrides. |
.github/workflows/vally-ci.yml |
Splits Vally CI into two jobs. |
.github/workflows/eval-unit.yml |
Removes legacy GitHub unit workflow. |
Review details
Comments suppressed due to low confidence (1)
cli/azd/test/evals/make-vally-report.go:82
- azd-code-reviewer: This unexported field is initialized but never read. Staticcheck U1000 reports unused struct fields, so preflight will fail until both this field and its initializer are removed.
- Files reviewed: 57/59 changed files
- Comments generated: 5
- Review effort level: Medium
Co-authored-by: richardpark-msft <51494936+richardpark-msft@users.noreply.github.com>
added 4 commits
July 23, 2026 14:08
… closer to what it's actually doing.
…re-dev into azd-02-waza-port-evals
…t`, which actually can't create an environment - Use a prompt grader for fuzzy situations (these commands were listed in this order-ish) vs "hey, was the one piece of text mentioned?" which can be an output-matches. - Update the report so it reports _which_ checks failed in a failing test.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:48
- azd-code-reviewer: The PR description says only the Node app eval is enabled initially, but this second stimulus and all following stimuli are active. Because the workflow runs this file with
runs: 3, the deploy job executes 24 trials rather than the described 3. Either keep the planned gradual rollout by disabling the additional stimuli or update the PR description to reflect the expanded CI scope.
- Files reviewed: 57/60 changed files
- Comments generated: 4
- Review effort level: Medium
added 3 commits
July 23, 2026 15:22
…nicode characters on incorrect byte boundaries.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only
deploy-node-apiis enabled and the other deploy scenarios will be introduced later, but this file activates all eight scenarios at three trials each, and the workflow runs the entire file. That materially expands CI cost and flakiness beyond the stated rollout. Either disable the remaining stimuli or update the PR description if enabling all of them is intentional.
- Files reviewed: 57/60 changed files
- Comments generated: 3
- Review effort level: Medium
added 3 commits
July 23, 2026 16:00
…t accidentally flag answers that might mention the 'bad" thing (azd init) in a "well, don't do that" type of answer.
… hopefully do a better job (has a limited rubric, so should be okay)
- Run our vally_report.go after each run, so we get a nice condensed report. It also prints out how it was generated into the report so it's clear it's not a vally thing, just our custom script - Also, made some nice little enhancements - we pick up the skill name when we summarize the tools in our report, etc.... - Added in actual tests for the vally_report.go file - Switch over to more prompt-based grading. A lot of the "hey, what would you do if 'x'" tests can have perfectly valid answers that also include the "don't do this" stuff. Using a prompt is really the only way to go.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only the Node app test is enabled, but this file activates all eight stimuli with three runs each, and the new deploy CI job executes the whole file. Either disable the other seven stimuli as described or update the PR description to reflect the substantially larger CI scope and cost.
- Files reviewed: 62/65 changed files
- Comments generated: 6
- Review effort level: Medium
added 2 commits
July 24, 2026 14:03
…o get to the minimum needed for a passing grade in the meantime.
Most of it was trickery around the threshold value - I wasn't trying to be picky on that value (yet), so it has to be low for now.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only the Node app scenario is enabled and the others will be introduced gradually, but this file activates eight stimuli at three runs each, and
vally-ci.ymlexecutes the whole file. That launches 24 model trials on every matching PR and bypasses the stated rollout. Disable the other stimuli, or update the PR description if enabling all of them is intentional.
- Files reviewed: 62/65 changed files
- Comments generated: 3
- Review effort level: Medium
- use the evals go.mod - we're only installing Go so we can use it with our report tool! - Adjust thresholds and graders again. I need to rethink how we want to do our deterministic graders because the answers are pretty freeform.
richardpark-msft
marked this pull request as ready for review
July 25, 2026 01:56
richardpark-msft
requested review from
JeffreyCA,
RickWinter,
danieljurek,
hemarina,
tg-msft and
vhvb1989
as code owners
July 25, 2026 01:56
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Member
Author
|
(AFAICT, I'm past the initial CCR barrage!) |
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (2)
cli/azd/test/evals/eval-azd-qna.yaml:16
- azd-code-reviewer: This scoring note is stale: each stimulus has three graders, not four, and with equal binary scores a
0.50threshold requires two passes. Document the actual 2-of-3 behavior so future threshold tuning uses the correct denominator.
.github/workflows/vally-ci.yml:46 - azd-code-reviewer: This pinned commit is
actions/checkoutv7.0.1, while the annotation says v7.0.0. Keep the version comment aligned with the SHA so dependency audits and future upgrades identify it correctly.
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.0
- Files reviewed: 62/65 changed files
- Comments generated: 1
- Review effort level: Medium
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v7 | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.0 |
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
jongio
reviewed
Jul 25, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
A couple of minor cleanups on the new vally evals harness, flagged inline. Both are low priority and neither blocks.
vally_report.go:toWorkspaceRelativeis unused. This nested module has its own go.mod, so the repo's golangci-lint doesn't reach it and the vally-eval action only runsgo test, so dead code here won't get flagged automatically.eval-azd-qna.yaml: the scoring comment says each stimulus has 4 graders, but they each have 3 (two output-matches plus one prompt).
| return filepath.ToSlash(rel) | ||
| } | ||
|
|
||
| func toWorkspaceRelative(p string) string { |
Member
There was a problem hiding this comment.
toWorkspaceRelative is unused. Normally the unused linter would catch this, but test/evals is a separate module, so golangci-lint run ./... from cli/azd doesn't descend into it and the vally-eval action only runs go test. Safe to delete.
Comment on lines
+14
to
+16
| # Allow a single grader miss (3 of 4 graders passing) while keeping a high-quality | ||
| # bar for Q&A responses. Every stimulus below has exactly 4 graders so this | ||
| # threshold has the same meaning ("at most one miss") across all of them. |
Member
There was a problem hiding this comment.
This says each stimulus has "exactly 4 graders" and describes "3 of 4 graders passing", but both stimuli here have 3 graders (two output-matches plus one prompt). With 3 equally-weighted binary graders, threshold 0.50 already allows one miss (2/3 = 0.67). Worth updating the comment so the "at most one miss" rationale matches the file.
jongio
approved these changes
Jul 25, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
Went back through the report generator and eval config on the current head. The earlier generator issues are handled: rune-based truncation so multi-byte output isn't split, per-check pass/fail in the markdown instead of the overall trial result, errors.Join with a non-zero exit when a report fails to write, and report links resolved relative to the report directory. Coverage is in via vally_report_test.go and the eval suites are green in CI.
Two things I left inline earlier still stand and neither blocks: toWorkspaceRelative in vally_report.go is unused now that relativeReportLink handles links, and the scoring comment in eval-azd-qna.yaml says four graders per stimulus when each has three. Both are fine to clean up whenever.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Translating over some of the waza evals into vally evals.
Some evals/tests just didn't make the cut - either they were redundant with our normal unit tests (there were several that were checking help content), or they were evals that seemed to be eval'ing themselves (ie, the answer was encoded in the system prompt).
I've reduced this down to two buckets:
here).
I've tuned down the threshold for some of the flakier evals, just to give us a good reason to investigate if this job starts failing.
There's also an experiments file in there, which isn't yet running in CI, that can be useful for some manual testing as we refine things a bit.
Fixes #9199 (by adding in an experiments file)
Fixes #9186 (by porting waza evals)