[evals] Adding in some vally testing for some basic azd scenarios - #9282

Open

richardpark-msft wants to merge 26 commits into

mainfrom

azd-02-waza-port-evals

Commits on Jul 22, 2026

Removing the old waza based evals, we can use vally now.
ripark
committed

Commits on Jul 23, 2026

Translating over some of the waza evals into vally evals.
ripark
committed
Adding in the actual execution of the two jobs for our new vally evals for some common scenarios - troubleshooting (ie, no azd around, but what does the LLM say?) and deploy, which will actually ha…
ripark
committed
Apply remaining changes

Copilot
and
richardpark-msft
authored
Rename troubleshooting to qna (ie, question and answer), since that's closer to what it's actually doing.
ripark
committed
Merge branch 'azd-02-waza-port-evals' of https://github.com/Azure/azure-dev into azd-02-waza-port-evals
ripark
committed
- Tightening up one of the evals so it doesn't let you use `env select`, which actually can't create an environment
ripark
committed
Updating comment
ripark
committed
Interesting, you must use runes to avoid accidentally slicing apart unicode characters on incorrect byte boundaries.
ripark
committed
Whoops, needed to check the length in _runes_ and forgot some more renames.
ripark
committed
Missed a file.
ripark
committed
Small tweak to the "create a staging environment" question so we don't accidentally flag answers that might mention the 'bad" thing (azd init) in a "well, don't do that" type of answer.
ripark
committed
Switch deterministic non-rigorous grader with a prompt grader that'll hopefully do a better job (has a limited rubric, so should be okay)
ripark
committed

Commits on Jul 27, 2026

Commits on Jul 28, 2026