Conversation
…CII compatibility
…mpt model after API key
- Print passing test summary with structural status and semantic % scores - Show percentage scores and PASS/FAIL verdicts in AgentRegressionError report - Suppress transformers 'Loading weights' progress bar during model load Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ntation demo_mock.py: - PatchSet (zero instrumentation) is now the first and primary section - Shows complete lifecycle: record -> pass -> regression report -> approve -> re-pass - Regression step prints full formatted error with percentage scores - Adapters section retains all providers, shown as explicit alternative - Adds pytest fixture code snippet section at the end demo_real.py: - Adds zero_instrumentation_demo() using PatchSet with raw openai.OpenAI client - Scenario regression catches now print str(e) for the full formatted report - Imports PatchSet Also fixes em dash in asserter first-run message (Windows ASCII compat) and suppresses transformers 'Loading weights' progress bar via transformers.utils.logging.disable_progress_bar() during model init. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates AgentSnap’s recommended instrumentation path to favor PatchSet (zero-instrumentation) across docs and demos, and introduces/clarifies semantic-backend configuration via the init wizard and pyproject.toml, alongside improved regression reporting output.
Changes:
- Repositions
PatchSetas the default/recommended capture mechanism in docs and examples; adapters are now presented as an alternative. - Adds/expands setup wizard behavior and configuration to explicitly select a semantic backend (offline embeddings vs LLM judge) and adjusts CLI/init/check flows accordingly.
- Improves error/report formatting and adds/updates tests to cover the new setup/back-end expectations.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| USAGE.md | Reorders and expands guidance to recommend PatchSet first; moves adapters to an “alternative” section. |
| README.md | Updates quickstart to show “no code changes needed” via PatchSet, with adapters as an alternative. |
| pyproject.toml | Introduces backend = "offline" and comments describing semantic backend selection. |
| agentsnap/setup_wizard.py | Writes backend to config and adds interactive connection testing with retry-on-failure for judge setup. |
| agentsnap/exceptions.py | Enhances AgentRegressionError.__str__ formatting (tool match % and PASS/FAIL formatting for semantic checks). |
| agentsnap/core/diff.py | Adds explicit “setup required” gating for embeddings and prevents surprise downloads; adds HF progress-bar suppression. |
| agentsnap/core/asserter.py | Tweaks messaging and prints a per-test PASS summary including similarity percentages. |
| agentsnap/config.py | Loads .env near project root and updates precedence documentation accordingly. |
| agentsnap/cli.py | Updates strings/output and removes redundant init-time connection test (wizard now performs it). |
| examples/demo_real.py | Adds a real-API PatchSet zero-instrumentation demo and simplifies error output printing. |
| examples/demo_mock.py | Refactors mock demo into multiple “journey” sections including PatchSet lifecycle + adapters + langgraph + pytest snippet. |
| tests/unit/test_setup_wizard.py | Adds coverage for “no setup run” backend errors and adjusts offline-backend tests. |
| tests/integration/test_examples.py | Updates expected demo completion output text. |
| tests/integration/test_cli_init.py | Updates wizard input flows and expected outputs for init, including failure/retry behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+21
to
+39
| # Require explicit setup: either wizard wrote backend="offline" to | ||
| # pyproject.toml, or a judge API key is available in the environment. | ||
| api_key = os.environ.get("AGENTSNAP_JUDGE_API_KEY") | ||
| if not api_key: | ||
| try: | ||
| from agentsnap.config import load | ||
| cfg = load() | ||
| backend_configured = bool(cfg.get("backend") or cfg.get("judge_api_key")) | ||
| except Exception: | ||
| backend_configured = False | ||
| else: | ||
| backend_configured = True | ||
|
|
||
| if not backend_configured: | ||
| raise RuntimeError( | ||
| "No semantic backend configured.\n" | ||
| "Run 'agentsnap init' to set up your comparison backend." | ||
| ) | ||
|
|
| run_adapter_demo( | ||
| "anthropic", | ||
| lambda: AnthropicAdapter(MockAnthropicClient(_resp("I'll look that up."), _resp("I'll look that up."))), | ||
| lambda c, t, q: (c.messages.create(model="claude-sonnet-4-6", messages=[{"role": "user", "content": q}], max_tokens=50), t(query=q)) and f"Result: {t(query=q)}", |
Comment on lines
+108
to
+111
| scores = report.semantic_scores or {} | ||
| parts = ["structural: ok"] if not report.structural_diff else [f"structural: mismatch"] | ||
| for step, score in scores.items(): | ||
| parts.append(f"{step}: {int(score * 100)}%") |
Comment on lines
+56
to
60
| # Semantic backend: "offline" (all-MiniLM-L6-v2) or "judge" (LLM via API) | ||
| # Run `agentsnap init` to reconfigure. API key goes in AGENTSNAP_JUDGE_API_KEY env var. | ||
| backend = "offline" | ||
| # LLM judge settings (used when backend = "judge") | ||
| judge_model = "openai/gpt-4o-mini" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.