Add example: self-healing Playwright tests with multi-model fallback#93
Open
Desperado wants to merge 3 commits into
Open
Add example: self-healing Playwright tests with multi-model fallback#93Desperado wants to merge 3 commits into
Desperado wants to merge 3 commits into
Conversation
Adds a new cookbook example that demonstrates: - Generating a Playwright test from a natural-language spec via the Vercel AI SDK, with multi-model fallback (Claude -> GPT -> Gemini) so a single-provider outage doesn't stall the loop. - Running the generated test in the pre-baked `playwright-chromium` template (fresh sandbox per attempt). - On failure: capturing the page HTML via fs.writeFileSync, reading it back out of the sandbox, and feeding it into the next LLM call so the retry gets real DOM to anchor selectors on. Three small modules, ~250 lines total (router.ts / runner.ts / healer.ts). Four runnable examples: basic, self-healing, fallback, parallel suite. Pattern inspired by the self-healing behaviour at qualitymax.io; code is standalone TypeScript. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three updates rolled in from production use at qualitymax.io: 1. Inject test.afterEach capture hook server-side with sentinel-bracketed strip-and-reattach instead of trusting the LLM to write the snapshot. Closes three silent failure modes -- LLM omits the hook on heal, writes to root-owned /app/ (EACCES), or duplicates `import fs` producing a SyntaxError that fails every retry before the test runs. Also fixes a path bug: the prompt told the LLM to write to /app/failure.html but the runner read /home/user/work/failure.html, so the snapshot was never actually loaded. 2. Classify failures (strict_mode_violation / locator_not_found / timeout / assertion_failed / unknown) and only inject a strict-mode hint on the matching bucket. Without targeted guidance, the healer re-emits equivalent fragile locators and burns the heal budget on the same failure. The strict-mode hint points at three concrete fix patterns instead. 3. Sentinel-based progress (QMAX_PHASE:* markers) parsed from streaming onStdout/onStderr so UI consumers see real-time progress instead of an opaque "running...". README updated with a "Battle-tested patterns" section explaining each. typecheck clean.
|
We require contributors to sign our Contributor License Agreement, and we don't have @Desperado on file. You can sign our CLA at https://e2b.dev/docs/cla . Once you've signed, post a comment here that says '@cla-bot check' |
Author
|
FYI — opened a companion PR #95 adding a separate example for the live noVNC browser feed pattern (referenced from this example's new "Battle-tested patterns" section). They're independent — happy to land them in either order. |
Author
|
@cla-bot check |
|
The cla-bot has been summoned, and re-checked this pull request! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new cookbook example demonstrating the self-healing test pattern on top of the
playwright-chromiumtemplate.What it shows
playwright-chromiumsandbox per attempt.page.content()to the filesystem, read it back out, feed it into the next LLM call so the retry can anchor selectors on real DOM.Structure
Three small modules (
router.ts/runner.ts/healer.ts, ~250 lines). Four runnable examples: basic, self-healing, fallback, parallel suite.Notes on
playwright-chromiumThe template ships
playwright+ browser binaries but not@playwright/test. The runner installs that in a user-writable work dir (since/appis root-owned) and pointsPLAYWRIGHT_BROWSERS_PATHat the template's pre-baked Chromium so the test runner doesn't re-download.Verified end-to-end
npm install+npm run typecheckcleannpm run example:healingpasses in 2 attempts (fail → snapshot feedback → pass)Pattern inspired by the self-healing behaviour at qualitymax.io; code is standalone TypeScript.
Also adds a one-line bullet to the root README's "Example use cases" section.