Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54
Open
DEM1TASSE wants to merge 122 commits into
Open
Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54DEM1TASSE wants to merge 122 commits into
DEM1TASSE wants to merge 122 commits into
Conversation
…tool
A built-in submodule turning solved tasks into reusable, executable code skills:
- skills/{library,retrieve,decide,gate,update,llm}: store / retrieve (relevance) /
decide (use·adapt·skip utility) / admission gate (gold|self_verify|none) /
evolve (incremental growth on existing library) — backend-agnostic via configure_llm
over webwright's own Model abstraction (no hardcoded gateway/key/path)
- tools/skill_use.py: solve-time tool (agent invokes like self_reflection/image_qa) ->
retrieve+decide -> JSON recommendation (use/adapt/skip + source path)
- python -m webwright.skills.update --manifest batch.json --library ./lib : batch growth
- tests/skills: 5 unit tests pass against the migrated module (logic == original)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…skill_use CLI - skills/prompt.with_skill_hint: prepend skill-library usage hint to task prompt (non-invasive; webwright merges system_template by replacement, so prompt-level is the clean way) - config/skill_mode.yaml: optional overlay doc + step budget for skill-reuse runs - llm._model(): bare CLI (python -m webwright.tools.skill_use) builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env -> same backend as agent, no hardcoded gateway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: what the module is, the two plug points (skill_use tool + update CLI), components table, gate semantics, backend config, results summary - llm._model(): bare CLI builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: Skill Library section (what it is, reuse via skill_use tool, grow via update CLI, end-to-end validation summary) - tests/skills: 5 unit tests for library/gate/update/evolve/retrieve+decide Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Remove _grow / update() / _UPDATERS dispatch — evolve() is the single entry now; drop the test_update test that exercised the removed grow path. Keep retrieve/llm fallbacks (useful). Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…val) Three bugs hit when update.refine emits a large skill on a slow gateway: - llm() ignored max_tokens -> model default ~4000 truncated the refined skill mid-code - llm() had no timeout override -> model default 120s ReadTimeout'd on the ~16k-token refine (now request_timeout_seconds defaults 600, env SKILL_MODEL_TIMEOUT) - _extract_code returned raw text (with ```python fence) when the closing fence was missing (truncated) -> skill failed to compile; now strips the opening fence anyway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…lve-time reuse, direct skill run) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
@microsoft-github-policy-service agree company="Microsoft" |
…te+manifest -> update -> reuse); fix output_schema examples to gate's {type} form
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bArena numbers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- traces_from_manifest: 'admit' is now REQUIRED per run — a missing gate verdict raises instead of silently defaulting to admitted (was the main pollution risk) - _slug: templates longer than 48 chars get a short content-hash suffix so two templates sharing a long prefix can no longer overwrite each other's skill - skill_use.recommend: the decision's skill_id must be one of the RETRIEVED candidates; anything else (LLM hallucination, even an existing library id) downgrades to skip - tests for all three Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… is truthy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tive paths, missing answer file) Independent cleanroom reproduction (fresh clone + venv, README-only, public GitHub tasks) surfaced usability failures; mechanism itself reproduced end-to-end in 25 min. - with_skill_hint resolves the library path to ABSOLUTE (F2): the hint's command runs in the agent's workspace, where a relative ./library silently resolved to a nonexistent dir -> empty library -> every lookup skipped, no error, answer still right - skill_use.recommend: a missing/empty library now answers skip with an explicit 'warning: library empty at <abspath>' BEFORE Library() can mkdir the bogus path (F3) - README: step 1 now tells the agent to write agent_response.json (stock webwright does not produce it; the gate/manifest flow assumed it) with a copyable ANSWER_SPEC (F1); absolute-path + --library-beats-env notes (F2/F4); custom endpoint tip (F5) - skills/__init__ no longer eagerly imports update -> no more runpy RuntimeWarning on 'python -m webwright.skills.update' (F6); import evolve/Trace from the submodule - tests: hint abspath, empty/missing-library warning (incl. no-mkdir side effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, filled-in inputs - example_library/: the commit-counting skill verbatim as evolve wrote it (runnable standalone via taskspec, no LLM in the loop; functionally verified against a local repo) - README: what a skill looks like (catalog card + the distilled git-log algorithm), measured held-out numbers (33->10 steps; wrong->correct rescues; honest note that reuse costs more than it saves on cheap tasks), three try-it paths - honest coverage-boundary demo: an unseen period shape raises cleanly; on the real held-out run the agent read the source and adapted around it - tasks/batch/taskspec example JSONs matching the how-to-use steps - links from the module README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, safe growth, measured results) + data-flow/interfaces diagram
…no manifest) python -m webwright.skills learn <runs_dir> --library ./library - auto: reads task.json/agent_response.json per run, gates (gold via --golds, else self_verify), groups tasks into templates + extracts params with one LLM call per chunk (default 25; existing templates passed in so chunks refine instead of duplicating), infers output_schema from the answer shape, site from start_url - idempotent: processed runs remembered in library/.learned.json; --dry-run plan mode - README: Quickstart (two commands) + use cases up top; old walkthrough demoted to 'Manual mode'; examples/solve_with_library.sh wrapper (hint + answer instruction) - unit tests for the LLM-free plumbing
…ks, leaks) External-user test of the friendly path surfaced that a trivially-easy config mistake (gateway key + unset OPENAI_ENDPOINT) silently disabled ALL reuse. Fixes: - skill_use: a hard error still degrades to skip (never block solving) but now says LOUDLY it is a LOOKUP FAILURE, not a no-match — error field in the JSON, hint about OPENAI_ENDPOINT/SKILL_MODEL_ENDPOINT, and a stderr line (F1+F2) - README Quickstart: gateway users must export OPENAI_ENDPOINT/OPENAI_MODEL for BOTH steps, stated where step-1 users actually look (F2) - learn: grouping-LLM failure now exits with a one-line actionable message instead of a 40-line traceback (F3); skipped-for-missing-answer runs get a visible summary with the correct pointer (the old message named a command that does not exist) (F4) - learn: strips the answer-output instruction from task text so it cannot leak into templates/skill_ids (F7) - solve_with_library.sh: usage check instead of passing empty args into the CLI (F6)
…ers get them too)
…ssion tests - README: "Only verified solves get in" -> "Validation-gated, exactly as strong as the gate you give it" — states plainly that the default self_verify checks shape only and that the WebArena numbers used the gold gate; learn prints the same warning at run time when no --golds is given - examples/learned_library/: a skill produced by "skills learn" from 3 real GitHub solves — n_solves=3, owner/repo lifted to parameters, two extraction strategies as fallbacks; verified standalone on an unseen repo (numpy/numpy -> v2.5.1, no model); test_learned_example.py locks n_solves>=3 + lifted params + no leak - regression tests for the interface-test findings: F1 (skill_use surfaces hard errors as ERROR, not quiet skip), F3 (learn exits with an actionable message)
There was a problem hiding this comment.
Pull request overview
This PR introduces a new webwright.skills subsystem that turns previously solved tasks into reusable, executable “skills”, enabling solve-time reuse (via a CLI tool) and offline library growth (via learn/update pipelines) while keeping the main agent loop unchanged.
Changes:
- Adds a disk-backed skill library (
Skill/Library) plus retrieve/decide/gate/evolve/learn modules to store, select, admit, and incrementally refine skills. - Adds
webwright.tools.skill_useas a solve-time CLI that recommendsuse|adapt|skipand provides the source path for reuse. - Adds docs/config/examples and new tests to validate deterministic plumbing and example artifacts.
Reviewed changes
Copilot reviewed 29 out of 31 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/skills/test_retrieve_decide.py | Adds deterministic tests for retrieve/decide + skill_use/prompt behavior (currently not pytest-discoverable). |
| tests/skills/test_library.py | Adds tests for disk persistence of Library (currently not pytest-discoverable). |
| tests/skills/test_learned_example.py | Adds a check that the checked-in learned example is aggregated/parameterized (currently not pytest-discoverable). |
| tests/skills/test_learn.py | Adds tests for learn plumbing + regression handling (currently not pytest-discoverable). |
| tests/skills/test_gate.py | Adds tests for gate admission logic (currently not pytest-discoverable). |
| tests/skills/test_evolve.py | Adds tests for evolve behavior and slug collision avoidance (currently not pytest-discoverable). |
| src/webwright/tools/skill_use.py | Introduces solve-time library recommendation tool with guardrails against missing/empty libraries and hallucinated skill IDs. |
| src/webwright/skills/update.py | Implements incremental library evolution and refinement prompt construction + manifest ingestion. |
| src/webwright/skills/retrieve.py | Implements LLM-based retrieval plus a simple deterministic keyword-overlap fallback. |
| src/webwright/skills/decide.py | Implements LLM-based use/adapt/skip decision over retrieved candidates. |
| src/webwright/skills/gate.py | Implements admission gate (gold/self_verify/none/auto) to prevent wrong solves from entering the library. |
| src/webwright/skills/learn.py | Adds “friendly” pipeline to learn skills from run folders with gating, chunked grouping, and an idempotent ledger. |
| src/webwright/skills/library.py | Adds on-disk skill storage (<id>/skill.py + meta.json) and simple list/get/add APIs. |
| src/webwright/skills/llm.py | Adds backend-agnostic LLM helper using Webwright’s Model abstraction. |
| src/webwright/skills/prompt.py | Adds with_skill_hint() helper that prepends a bash command hint to consult the skill library. |
| src/webwright/skills/init.py | Exposes the public webwright.skills API surface for consumers. |
| src/webwright/skills/main.py | Adds `python -m webwright.skills <learn |
| src/webwright/skills/README.md | Adds comprehensive module documentation, usage patterns, and rationale. |
| src/webwright/skills/pipeline_diagram.svg | Adds diagram documenting data flow and interfaces for the skills pipeline. |
| src/webwright/config/skill_mode.yaml | Adds optional config overlay to increase step budget for skill reuse runs. |
| src/webwright/skills/examples/README.md | Adds examples overview and how-to for running skills/tools and batch pipeline. |
| src/webwright/skills/examples/solve_with_library.sh | Adds helper script to prepend hint + answer spec and run Webwright with a library. |
| src/webwright/skills/examples/taskspec.example.json | Adds example taskspec input for running a skill standalone. |
| src/webwright/skills/examples/tasks.example.json | Adds example batch task list input with params/golds. |
| src/webwright/skills/examples/batch.example.json | Adds example manifest for update (admit/params/schema/etc). |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/skill.py | Adds a runnable example skill produced by the pipeline. |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/meta.json | Adds metadata for the example skill. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/skill.py | Adds a checked-in “learned” skill example aggregated from multiple solves. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/meta.json | Adds metadata for the learned skill example. |
| README.md | Adds top-level README section linking to the new skill-library capability and docs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| f"answer={json.dumps(tr.answer, ensure_ascii=False)[:120]})\n```python\n{tr.code}\n```" | ||
| ) | ||
| sys_prompt = _REFINE_SYS + (_REFINE_INCREMENTAL if existing else "") | ||
| code = _extract_code(llm(sys_prompt, "\n\n".join(blocks), max_tokens=16000, timeout=400)) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — removed the kwarg (it was indeed swallowed by llm()'s **_). Timeout is governed by the model's request_timeout_seconds (600s default in _model()).
Comment on lines
+175
to
+179
| if answer is None and (d / "agent_response.json").exists(): | ||
| try: | ||
| answer = json.load(open(d / "agent_response.json")).get("retrieved_data") | ||
| except Exception: | ||
| pass |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+198
to
+199
| manifest = json.load(open(a.manifest, encoding="utf-8")) | ||
| traces = traces_from_manifest(manifest) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+59
to
+64
| def llm_json(system: str, user: str, **kw: Any) -> dict: | ||
| """Call + parse the first {...} JSON object out of the reply.""" | ||
| txt = llm(system, user, **kw) | ||
| match = re.search(r"\{.*\}", txt, re.S) | ||
| if not match: | ||
| return {} |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — llm_json now walks the reply with json.JSONDecoder().raw_decode and returns the first valid JSON object, skipping non-JSON brace snippets; matches the docstring.
Comment on lines
+48
to
+56
| meta = json.loads((d / "meta.json").read_text()) | ||
| code = (d / "skill.py").read_text() if (d / "skill.py").exists() else "" | ||
| return Skill(skill_id=skill_id, code=code, meta=meta) | ||
|
|
||
| def add(self, skill: Skill) -> None: | ||
| d = self._dir(skill.skill_id) | ||
| d.mkdir(parents=True, exist_ok=True) | ||
| (d / "skill.py").write_text(skill.code) | ||
| (d / "meta.json").write_text(json.dumps(skill.meta, ensure_ascii=False, indent=2)) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — explicit UTF-8 on all skill/meta reads and writes in library.py (and the same for learn.py's ledger/run files).
Comment on lines
+34
to
+35
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+86
to
+87
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+26
to
+27
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+69
to
+70
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+53
to
+55
| if __name__ == "__main__": | ||
| run() | ||
| run_regressions() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all) that runs both run() and run_regressions().
The README table (held-out 70% vs 55%, 14.7 vs 17.1 steps; train 26/30 vs
23/30) previously existed only as prose. Now:
- evals/webarena/results/: sanitized per-task records of the exact run behind
the table — task id, answer, gold score, steps, skill verdict; one command
("reproduce.py table --results results") re-derives the table, no setup
- evals/webarena/reproduce.py: self-contained driver that re-runs the whole
experiment (train -> gold-gated update -> held-out with/base -> table)
against your own WebArena deployment via microsoft/webarena-verified;
resumable, parallelizable per template
- run_all.sh + model.eval.yaml (agent-model overrides; gateway pointer)
- tests/skills/test_eval_snapshot.py: CI-locks the records to the published
numbers and enforces snapshot sanitization (no local paths/hosts/keys)
- skills README + examples README now link the records instead of asking for
trust; CI also triggers on evals/webarena/**
- prompt.py: shell-quote task and library in the skill_use hint (shlex.quote) —
$VAR / $(...) / backticks expanded in bash even inside the old double quotes;
regression test added
- llm.py: llm_json now scans for the FIRST valid JSON object (raw_decode) as
documented, instead of a greedy first-{ to last-} regex that could span
unrelated braces
- update.py: drop the misleading llm(..., timeout=400) kwarg (silently
swallowed; the model's request_timeout_seconds already governs); read JSON
files via read_text instead of unclosed open()
- library.py / learn.py: explicit UTF-8 on every skill/meta/ledger read+write
(locale-independent on Windows)
- tests: pytest-discoverable test_all() entry points in all 7 files (CI keeps
running them as scripts too)
adamlu123
reviewed
Jul 9, 2026
Author
There was a problem hiding this comment.
It's the guided tour of the examples/ directory (the module README links here for every "see examples"): two real, checked-in skill libraries — learned_library/ is the Quickstart loop's actual output (3 GitHub solves -> learn -> owner/repo lifted to parameters, runs standalone on unseen repos with no model), example_library/ is verbatim update.evolve output from the WebArena eval — plus the solve wrapper and filled-in copies of every input file the manual pipeline asks you to write. 8ea59be makes this explicit in the file's opening paragraph and adds the learned_library provenance section.
Contributor
There was a problem hiding this comment.
this file looks redundant, can we remove?
Author
There was a problem hiding this comment.
Agreed — removed in 8ea59be. Nothing referenced it: the skill hint is prompt-level (with_skill_hint), and no documented path needed the step_limit bump.
| # 2. turn everything you've solved into skills — no manifest, no fields to learn | ||
| python -m webwright.skills learn outputs/ --library ./library | ||
| ``` | ||
|
|
Contributor
There was a problem hiding this comment.
@DEM1TASSE It is better to add the complete example in the quick start session.
It needs to additionally include how to use the skill library.
Author
There was a problem hiding this comment.
Done in 8ea59be — the Quickstart is now the complete loop on a copy-pasteable example (public GitHub): solve 3 instances -> learn -> an unseen instance reuses the skill (with the expected skill_decision.json shown), plus how to use the library without the agent (querying skill_use directly, and running the learned skill standalone with no model — verified pandas-dev/pandas -> v3.0.4). The same loop's output is checked in at examples/learned_library/.
…larify examples/ - README Quickstart is now the complete loop on a runnable example (public GitHub): solve 3 instances -> learn -> an UNSEEN instance reuses the skill, plus using the library without the agent (skill_use query, running the learned skill standalone, verified: pandas-dev/pandas -> v3.0.4 with a params-only taskspec) - remove config/skill_mode.yaml: nothing referenced it (the skill hint is prompt-level via with_skill_hint; the step_limit bump was never needed by any documented path) - examples/README: state the directory's purpose up front, and document learned_library's provenance (the Quickstart loop's checked-in result) with the unseen-repo runs and the CI test that locks it
--verify off left `verified`/`grade` off the meta entirely. Two problems: a missing field reads as an oversight and breaks the first caller to index it (the current ones only survive on .get()), and it left the question this raises unanswered — isn't that just `reference`? It isn't. `reference` means the replay ran and the skill FAILED it: we know it doesn't reproduce its own training answers. `--verify off` means nobody looked. Labelling the untested one `reference` claims a test we never ran, in the other direction — and the distinction is useful: a reference skill has a .rejected_*.py and a reason to read; an unverified one has nothing, and may well be fine. If you're deciding how far to trust source you're about to reuse, "failed its own replay" and "never checked" are not the same warning. So: three grades, always written — executable / reference / unverified. The test was checked to fail against exactly the collapse this replaces.
The numbers only lived in docs/quickstart.md, so the README asked you to run a skill in 40 s and never said what that was instead of. It's now right under step 1, while the 40 s is still on screen: 25/40/59 steps and 11/26/32 min per route from scratch, against 10 fixed steps, ~40 s and zero model calls. Same measurements as the tutorial's table (rounded here, precise there), taken from the three runs that built the shipped skill. Says the two things worth reading off it rather than leaving them implied: the from-scratch cost is high-variance because the agent re-derives the strategy every time, and the zero in the last column is the structural win — a cron watcher pays for exploration once.
demo ran SEA->DEN, ask and solve ran PDX->AUS, and the training set was a third set of routes — so "the example" was three different tasks wearing one name, and the cost table's columns weren't measuring the same thing. Everything held-out is SEA->DEN now: the standalone run, the retrieval question, and the agent solve all answer the same question, so the numbers can be put beside each other.
The init section taught the spec with "the cheapest {product} on Amazon" — a task
that fails three of the four questions the same document asks you to apply, and
whose $0.00 placeholder answer is the reason the fourth question exists. It now
uses the flight task the rest of the doc uses, pasted verbatim from a real init
run (strict, because a schedule holds still).
shape keeps its lesson but gets a task that doesn't fight the four questions: a
repo's latest release version genuinely drifts, and init picks shape for it on
its own. Both yaml blocks are real output, not written by hand.
Split step 2, which was doing two jobs at once — reusing an existing skill
(ask/solve) and building one from nothing (build). They're now sections 2 and 3,
one direction each, so a reader isn't halfway through wondering which they're in.
The by-hand while-loop folds into a <details>: it's what build already does, and
left open it reads like a prerequisite.
--jobs moves to reference.md with its tuning and rate-limit caveat; the tutorial
keeps one inline comment and a pointer. Same for "vary the parameters" — the
principle stays, the detail goes. Timing unified on ~40 s, measured three times
today: 35, 37, 38.
Also fixed the drift comment init writes: it said "prices/stock/rankings" for
every drifting task, including one about release versions. It now uses the
model's own drift_reason, so the spec explains the task in front of you.
…ference was missing its main commands manual.md: kept, because it still works. Ran its documented flow verbatim — write batch.json, `update --manifest` — and a skill landed. It isn't a stale doc, it's the layer under build/learn, and per-field control and benchmark gold gates still need it. What it lacked was context, so it now opens with one: you probably want build/learn; this is the level below; the examples here are gitlab commit-counting rather than the flights the rest of the docs use, older but mechanically identical. And a real trap it never mentioned: `update` defaults to --verify off while learn defaults to strict, so following manual.md verbatim silently lands `grade: unverified`. That's exactly what my run produced. Now stated, with a pointer to verification and grades. reference.md: its parameter tables covered learn, update and skill_use — and not `init` or `build`, which are the main path. Both added, checked against --help rather than memory (that also turned up --rows, --outputs and --yes missing). learn's table gained --draws, and now says what separates it from --verify-rounds: rounds repair one candidate, draws throw it away and distil another. Two things I checked and did NOT change, contrary to the review: the honest WebArena footnote appears once (reference.md), not three times — nothing to de-duplicate; and README does already carry `unverified`, so the grades aren't out of step.
…ouldn't The banner told readers to prefer build/learn without ever saying what manual is for, which left the obvious question unanswered: build is the higher-level interface, so why does this exist at all? Because the two differ in WHO DECIDES, not in age. build/learn have an LLM infer the template and lift the parameters, and a gate decide whether a solve was correct. update has you state all three. Four cases where that matters, and they are the doc's reason to exist: - A benchmark, where the harness already knows the answers. You don't want our gate guessing — you pipe its evaluator's verdict in as `admit`. This is exactly how this repo's WebArena numbers were produced: eval_webarena.py scores each solve with gold_eval() and writes the result into the manifest as `admit`. Checked, not assumed. - Correctness that isn't string equality. --golds compares with ==; a human review, a judge model or partial credit doesn't fit a golds file but does fit a boolean you set. - Logged-in sites. learn never puts credentials in the Trace, so the replay can't log in; the manifest carries them. That's a real capability gap, and it was documented nowhere. - Not wanting an LLM to guess your template or params. README's doc table said "manifests field by field, gold gates" — true and useless. It now says which situations send you there.
I'd written that the examples "use gitlab rather than flights — older, but the mechanics are identical", which framed the right example as an inconsistency to be tolerated. It's backwards. manual exists for benchmarks with their own evaluator and for sites that need a login, and WebArena gitlab is both at once. Flights is neither — which is precisely why the Quick Start builds it with learn. Swapping the example to flights would have demonstrated manual on the one task that doesn't need manual. While there: credentials was listed as manual's distinguishing capability and then shown as `"credentials": null` in the only example that could have demonstrated it — on a site that needs a login. Filled in, with a line on why it matters (learn can't fill this field, so a logged-in site can't replay) and on what happens to them: _save_examples persists params/start_url/output_schema/ answer and not credentials, so the library stays shareable. Checked the code rather than claiming it.
…les need your host The cost table had the two ends and not the middle: a skill run standalone, and solves from scratch — but never the path most people will actually use, the agent reusing the skill. Measured it on SEA→DEN, the same held-out route the standalone column uses, with and without the library and nothing else changed: from scratch 50 steps 23.5 min 55 calls agent + library 11 steps ~4 min 12 calls standalone 10 steps ~40 s 0 calls All three answered ["WN 4697","Southwest","6:50 AM"], and so did the independent model-free probe — three routes to one answer, not one answer agreeing with itself. The middle column is read honestly rather than sold: 50→11 is a property of this task, not a promise, and on a site the model already drives from memory the gap narrows or inverts. Points at Results for the general version. examples/README claimed "from scratch 25–59 steps" for SEA→DEN — those were the three *training* routes, a different question. Fixed with the real number. manual.md: the WebArena examples don't run as written, and now say so — gitlab.example.com and the credentials are stand-ins for your own Docker instance. (Ours stays out of the repo.)
Moving it out of the tutorial, I kept the tutorial's prose — so a parameter reference had one flag with a section, three code samples and four paragraphs, while every other flag got a table row. Same facts, one row: N at a time, more than you have means all of them, N > 1 logs per solve and ticks every 30 s, the ceiling is the site not the flag (3-5 safe), and only solving parallelises. Retargeted the tutorial's link, which the deletion would otherwise have broken.
Checked both directions mechanically rather than by eye: every --flag argparse defines against every one the reference documents, and every documented default against the parser's actual default. Clean on most of it — no phantom flags, nothing undocumented, defaults all match (learn strict/2/2/reject/25, update off/2/reject, init skill.yaml/3, build jobs 1). Two didn't hold: - update's table was missing --draws, which it has and defaults to 2. - init wrote a build: block without `draws` — I added the flag and forgot the drafted spec. That's the worst place to omit it: the block is where a user learns which knobs exist, and draws is exactly the one to reach for when a skill gets rejected. The second is a class of bug, not an incident, so it's pinned: a test asserts the keys init writes are exactly the keys build reads. A key init writes that build ignores is a lie; a key build reads that init omits is a knob nobody finds. It fails if you remove draws from either side. (Two things I suspected and checked before changing: update's --library really is required=True, and skill_use's table is flag|meaning with no default column. Both were right as written.)
…fusing Its "used by" column mixed commands with concepts — "all LLM calls", "module LLM", "generated skills", "examples/quickstart.sh" — so there was no way to tell what any of it was for. And it still said MODEL_CFG was for "solve/full modes"; full is gone. What it never said is the reason the setup confuses people at all: **there are two models.** The module's — init, learn, skill_use — is configured by these env vars. The agent's, which drives the browser inside build's solves, reads a yaml and ignores them completely. Set only the env vars on a custom gateway and your solves quietly go to api.openai.com while everything else uses your gateway. That's not a guess: openai_model.py never reads OPENAI_ENDPOINT — it takes openai_endpoint from its config, defaulting to api.openai.com. It's the trap the external repro hit, and build now warns about it. So the section leads with a two-row table separating the models, then lists the vars with who actually reads each. Every claim checked against the source: SKILL_MODEL_CLASS defaults to openai and SKILL_MODEL_TIMEOUT to 600 (llm.py:33), SKILL_LIBRARY_ROOT is --library's default (skill_use.py:74), and MODEL_CFG really is read by quickstart.sh alone.
…fields The rewrite still left the actual question unanswered: what is SKILL_MODEL_NAME for, what is openai_endpoint, and how do they differ? It said "env vars" vs "a yaml" as if they were separate worlds, so a reader had no way to see that SKILL_MODEL_NAME / OPENAI_MODEL / the yaml's model_name are all one field. llm.py reads the env and builds the same config dict the yaml spells out by hand, and both models are the same class underneath. So the section now names the two settings that exist at all — which model, what URL — and gives each one a row with both doors, including the defaults you get when you set nothing (gpt-4o, api.openai.com/v1/responses). SKILL_MODEL_* isn't a second setting; it's the same one at higher priority, so you can point distillation somewhere other than whatever else already reads OPENAI_*. Also states the escape hatch it never mentioned: configure_llm() hands the module a model object and every var is ignored, which is how a running agent shares its own backend. Checked: model_name defaults to gpt-4o and openai_endpoint to the responses URL (openai_model.py:106,108); OPENAI_API_KEY is that class's _ENV_VAR (:113), so it's the one var both models really share; llm.py:35-36 is the `SKILL_* or OPENAI_*` fallback; nothing outside llm.py reads any of them.
Takes the rewrite wholesale: the input and output gates are now named as two gates with a table, instead of one paragraph that ran them together; each grade gets a cell saying what it buys; update/skill_use move under a "manual / integration" heading so the day-to-day path is the first thing in All parameters; and Components moves to the end, after the commands, where someone extending the code will look for it. Two real fixes on the way in. The output-gate paragraph said a skill gets "up to --verify-rounds build attempts, then rejected". That contradicted the grade table two paragraphs down (--draws × --verify-rounds) and undersells the budget: update.py:227-231 is two nested loops — --draws independent candidates, each repaired up to --verify-rounds times. Read the old sentence and you'd think --draws did nothing. Now it says both. The pasted copy also carried the pre-87f6be9 env section, which would have reverted the two-doors explanation written for exactly this question. Kept the new one. Cut the "Backend" section: it said configure_llm / SKILL_MODEL_* / no hardcoded key, all of which the env section now says at length, and the llm.py row said it a third time. One mention each, in the env section. Checked while editing: "--verify-rounds and --draws only take effect once you turn --verify on" is right — update.py:200 makes verify=off a straight line through one distillation call, entering neither loop.
…tput's help The skill_use flag table was empty — I truncated the file assembling the last commit (took the line count from the old version, cut the final three rows). It's back, now with defaults like the other tables: --task is required, --library falls back to $SKILL_LIBRARY_ROOT. Writing those rows turned up a lying help string. skill_use's --output says "Write JSON to this path instead of stdout", but line 97 prints unconditionally, so it writes the file AND prints. The docs were right and the code's own --help was wrong. It looks copied from self_reflection.py, which really does have the else branch that makes that sentence true; this one never did. Fixed the help, and the table now says stdout gets it either way. Two models, rewritten again — "drives the browser" vs "does the thinking around the browser" is not a difference anyone can act on. The real one is concrete: one opens a browser and picks the next click ~50 times a solve; the other never opens a browser, reads the finished transcripts, and writes the skill's python. Anchored to build = solve × N + learn, so it's clear where each half runs, and the two bullet lists plus the settings table are now one table. "Where to see each grade pay off" was a six-line paragraph to say two things; now it's two clauses and links straight to Results, without the aside about predating the replay gate. Net 30 lines out, 23 in.
Refactor manual mode documentation for clarity and structure.
Revised the explanation of the input and output gates, including details on their levels and verification processes. Updated the environment variables section for clarity and added backend-agnostic information.
Clarify command modes and parameters for skill factory usage.
…o a stale copy Three fixes, nothing else touched. --verify off named two reasons that are both wrong. "The data has drifted" is what --verify shape is for — the output-gate table says so fourteen lines above, so the page contradicted itself and talked people out of the verification they could have had. "The page moved" is worse: a failing replay is exactly the news there, and off just means never finding out, minting an unverified skill that promises nothing. The real reason, and why update defaults to off, is a replay that *can't* run: a benchmark login needing credentials the library can't store. "The WebArena numbers were produced by a reference-grade library" contradicts the grade table, where reference means the replay ran and failed. Those runs predate the replay gate, so no replay ran — formally unverified, and calling it reference claims more testing than happened. The sentence only ever needed to say a prior pays off, so it now says that and grades nothing; no hedge required. The env section had been reverted to the pre-87f6be9 text (the two rewrites were made after the copy it was edited from), taking the two-doors table with it and bringing the Backend paragraph back. Restored from 09dbdcf. Also filled in what the skill_use table was missing: defaults like every other table, and that --output still prints to stdout (the code's own --help was wrong about this; fixed in 09dbdcf). Left alone: the 27 lines of trailing whitespace — harmless, and clearing them would drag unrelated lines into the diff.
… was wrong I told you "the data has drifted" was a wrong reason for off, because shape handles drift. That's only true while the task still resolves. Drift far enough and the task itself is gone — the date passed, the listing was delisted — and the skill returns nothing at all. shape does not rescue that: an empty answer fails the gate outright (gate(None, method="self_verify").admit is False), so a fine skill lands reference for what the calendar did. The distinction isn't whether a replay can run, it's whether its verdict could be fair. Our own shipped fixtures live in exactly this case: the trajectories are pinned to 2026-08-15, and examples/trajectories/README.md already tells people to keep --verify off because it never opens a browser and so can't go stale. The reference page was contradicting the examples. So: off is for a replay that couldn't be fair — credentials the library can't store, or expired training instances. Drifting *values* still aren't that case (shape replays them and compares loosely), and neither is a moved page, where the failure is real news and reference records it.
…unclutter gpt-4o is real, not a typo: openai_model.py:106 is `model_name: OptStr = "gpt-4o"`, and skill_factory names no model of its own, so that's genuinely what you get when you set neither var. It's an upstream file this PR has no business editing. But quoting it in the table next to Results-at-gpt-5.4 reads as a recommendation that contradicts the experiments. So the table now says "the class's fallback" for both the model and the URL, and one line under it names them and says what they are: inherited, not suggestions, the Results ran on a much newer model, so name the model you want. --jobs was five sentences in one cell nobody can scan. Cell keeps what a lookup needs (solve N at once, solving only, more than you have means all); the throttling advice moves to a line under the table, where "3 to 5" also retires the file's last en-dash. SKILL_MODEL_ENDPOINT and SKILL_MODEL_NAME were crammed into one <br> row while their OPENAI_* twins had a row each. One var per row now, matching every other table. Confirmed, unchanged: skill_use --output writes an extra copy — print(payload) at skill_use.py:97 is unconditional, so stdout gets the verdict either way. The table is right and this is the behaviour the older "also write" wording meant. gate.py's methods (gold | self_verify | none) still match the input-gate levels; `auto` is a fourth, the signature default, which just dispatches to gold when a gold is present and self_verify otherwise — neither section lists it, and they stay consistent.
…fault The moved-page sentence said a failed replay "is real news and reference records it", which reads as automatic. It isn't: learn defaults to --on-fail reject, as its own table two sections down says, so by default a moved page leaves the runs retryable and nothing lands. Now the sentence names the flag and says what the default does instead. Dropped the em-dashes from the paragraphs I added (verification, two-models); the title's is yours, left alone. skill_use is introduced as belonging to neither mode, then appears in the two-models table as a caller of the module's model, which looks like a contradiction until you know it makes a model call at all. It does: skill_use.py:48 calls decide(), and decide.py:38 is an llm_json round trip. Retrieval itself is local. Said so where the command is introduced, which also answers "how much does asking cost".
… gpt-4o Reviewing the docs kept snagging on the same thing: the reference said the fallback model is gpt-4o while Results says gpt-5.4. The docs were right — openai_model.py:106 really does default model_name to "gpt-4o", and skill_factory names no model of its own, so setting neither SKILL_MODEL_NAME nor OPENAI_MODEL genuinely distils on gpt-4o. That default belongs to upstream and isn't this PR's to change; the fix is to stop it happening silently. _model() now prints one line to stderr when no name was set, naming the model it fell back to. Every skill in a library is written by this model, so which one it is shouldn't be something you discover later from a bad skill. It's the same shape as build's existing half-configured-gateway warning: warn, don't fail, so nobody's working setup breaks. stderr specifically, not stdout: skill_use prints JSON on stdout for an agent to parse, and a warning there would corrupt it. There's a test pinning that. Adds tests/skill_factory/test_llm_env.py — the precedence llm.py owns had no coverage at all: SKILL_MODEL_* over OPENAI_*, configure_llm over both, an unset endpoint not clobbering the class default with an empty string, and the warning (fires, names the fallback, stderr only, once rather than per chunk per draw). Each of the six was checked to fail against the bug it describes by mutating llm.py and rerunning it. Docs: the env section now says the warning exists, and says why the name matters rather than just that it's inherited.
…two vars "Set neither and you get..." made the reader stop and work out which two things "neither" meant, right after a sentence naming SKILL_MODEL_NAME, OPENAI_MODEL, and a yaml key. One word fixes the antecedent. Confirmed while here, since the sentence is a claim about behaviour: the warning fires only when both vars are missing. llm.py:35 ors them into one `name` and :42 warns on `not name`, so anyone who set only OPENAI_MODEL — the setup the paragraph above tells them is fine — stays silent. test_llm_env.py's test_naming_a_model_says_nothing pins exactly that case.
The redundancy had a root cause: "a schedule holds still so strict is fair, a fare drifts so it isn't" had no owner, so every section that touched it argued it again from scratch — section 1's closing, section 3's "Verification honestly", section 4's "When the answer moves on its own", and choosing-a-task's second question. Four passes at one idea. Choosing-a-task owns it now; everyone else points there. - Deleted "When the answer moves on its own" whole. It was question 2 with a different example. Its one unique fact — shape catches a broken skill but never a wrong one, which is what --golds is for — moved into question 2. - "Verification, honestly" keeps only what nothing else says: the standalone answer is byte-identical to an independent model-free probe. Two code paths, one answer. The strict-is-fair-here argument around it went. - Section 1's closing paragraph is one sentence and a link. - The timeout warning at the top now lives in section 4's --jobs paragraph, next to the flag it's about, instead of greeting you before you've run anything. Also unpacked the sentences that were carrying three clauses each (the step-1 screenshot line, how-to-read-the-middle-column, vary-the-parameters). Untouched on purpose: the cost table and how to read it (that's quickstart's own measurement, not the README's WebArena numbers), all four choosing-a-task questions, and the rejection table. 296 lines to 264, and what's left is what only this page says.
Updated quickstart instructions for running the skill and agent, clarified output and execution steps.
…" wasn't Your annotations, applied on top of your section 1-2 rewrite: - the cost table's task line names the date it was measured on, 2026-08-15 — pinned, unlike section 1's rolling default, which is why it's worth stating — and keeps "a route the skill never trained on", since held-out is the point of the table; - "This loop 已经提交和展示到" -> checked in at examples/learned_library/, with the provenance pointer; - section 4 becomes the two doors you listed, learn first: 4.1 you have trajectories, 4.2 you have a task but no runs. It used to open with init and build and bury learn at the end, which is backwards — learn is the cheap day-to-day path. Same order and same two doors as the README's 3.1/3.2. One real defect found while I was in there. The spec block says "this is its real output, verbatim" and it wasn't: no draws: 2 at all (I added --draws and never updated this page), and two of init's comments had drifted. Regenerated it by calling init's own _yaml_skeleton, so it's byte-identical now, long comment line and all. quickstart.sh: ask and solve now print the date they're asking about. demo already did. solve's answer depends on that date and it wasn't saying which.
"Step 3" meant two different things depending on which page you were on. The README's three steps are run / agent / build your own; quickstart had four, because "Watch the library get built from nothing" sat between 2 and 4 holding a step number. So quickstart's 4.1/4.2 — which I'd just aligned with the README's 3.1/3.2 — were reachable from a section number that didn't exist over there. The tell was in the title: sections 1, 2 and 4 are things you do, and section 3 was something you watch. It isn't a step. It's the worked example of the step after it, so it moves inside "do it for your own task" as an unnumbered "the same loop, on our task", keeping all of it — the dry-run, the by-hand fold-out, the cost table. Nothing is cut; it just stops claiming to be a step you take. Numbered sections now map 1:1 onto the README's, down to 3.1/3.2, and the header says so and links there. Checked: nothing anywhere linked to the two anchors this retires.
The quickstart was a second telling, not a second half. Its sections 1 and 2 had drifted into near-copies of the README's — same commands, same ten-steps line, same cost table, same probe claim — and once I renumbered it to match the README last commit, the overlap was the whole top of the page. Two files, one story, already disagreeing in places. What it had that the README didn't is now in the README, folded so the section stays scannable: - **the loop by hand** — the solve/learn/reuse commands build wraps, minus its copy of the gateway setup, which the README already documents right below; - **getting a skill that lands** — vary the parameters rather than the count, and which rejection means "wrong verify mode" versus "actually broken", with the .rejected_<id>.py post-mortem path. "What to expect" said rejection is cheap and not your fault; it never said how to read one. Three things the quickstart answered and the README didn't, now answered inline: `./quickstart.sh` with no arguments IS demo mode, on SEA→DEN thirty days out — so the date rolls, which is why it prints the one it picked and why your flight won't be the one in the table. The table's three runs are pinned to 2026-08-15; that's a measurement, and it now says so, because otherwise the first thing a reader does is run the command and get a different flight than the row above. The run directory keeps the trajectory, and QUICKSTART_WORKDIR keeps the run directory. `build` now shows the spec we ship (examples/flights.skill.yaml), so you can dry -run the real thing before writing one. `learn` already had its shipped trajectories in 3.1. Dropped with the file, deliberately: choosing-a-task, and the cost table's training-run spread. Every inbound link updated — the README's two, and reference.md's, which now points at the Quick Start. Nothing in the repo mentions quickstart.md.
Updated README.md to clarify command usage and execution steps.
…tput It showed verify, draws and on_fail. init writes five: verify_rounds and chunk were missing, so a reader would conclude they were CLI-only flags — you can put either in the spec, and build reads both. That was survivable while the quickstart carried a verbatim copy; deleting it made this the only spec anyone sees. It had drifted twice, both times the same way: I added --draws, then --verify-rounds, updated init.py, and never came back here. Nothing pinned the docs to the code. There's a test for init-writes == build-reads, and none for what the docs show. So test_the_readme_spec_shows_every_key_init_writes parses the README's yaml block and compares its build keys against _yaml_skeleton's. Checked both directions by mutation: drop verify_rounds from the README, it fails; make init write a key the README lacks, it fails. While there, the two keys people confuse now explain themselves in the block: draws bins the candidate and distils a fresh one, verify_rounds feeds one candidate its own failures. And the header says spec keys are CLI flags too, and the flag wins.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Web Skill Factory (
webwright.skill_factory) — a self-evolving skill factory (MVP): turn solved tasks into reusable,executable code skills, retrieve + judge them at solve time, gate what enters the library, and grow
the library incrementally. A self-evolving loop on top of Webwright's code-as-action solves:
This is the reuse + accumulation layer on top of Webwright's code-as-action solves: it consumes
the
final_script.pyevery solve already produces (plain or crafted mode — both work), accumulatesskills across tasks, judges when a prior skill applies, and improves skills as more solves arrive —
with a gate so wrong solves don't pollute the library. It complements
crafted_cli: wherecrafted_cliparameterizes a single task's script by anticipating what might vary,update.refineparameterizes across multiple verified solves — the differences actually observed between
instances become the parameters.
Modular composition (~810 lines of core code)
Ten small, single-responsibility modules — each with a stable interface and a swappable
implementation:
skill_factory/library.pySkill+Library, skills on disk (skill.py+meta.json)skill_factory/retrieve.pyskill_factory/decide.pyskill_factory/gate.pyskill_factory/update.pyrefineparameterizes + decomposes into primitivesskill_factory/llm.pyModel(no endpoint/key hardcoded)skill_factory/prompt.pywith_skill_hint)skill_factory/learn.pylearn <runs_dir>: auto-group runs into templates, gate, evolve; no manifest to writeskill_factory/__main__.pypython -m webwright.skill_factory <learn|update>dispatchertools/skill_use.pyHow it plugs in (no change to the agent loop or default config)
skill_usetool, invoked from bash likeself_reflection/image_qa:{verdict: use|adapt|skip, skill_id, source_path, how_to_reuse}.updateCLI distills a batch of gate-passed solves into aparameterized, primitive-decomposed skill:
learngroups a folder of finished runs intotemplates (one LLM call per chunk), gates them, and evolves the library — idempotent,
--dry-run:examples/learned_library/checks in the skill this produced from 3 real Google Flightssolves — five parameters lifted (origin/destination city+code, date), verified on an unseen
route three independent ways (from scratch / reuse / standalone, same answer) — with a CI
test locking it.
Validation
WebArena: 10 templates × 3 domains — reuse lifts accuracy +15pp and saves steps on held-out tasks
10 retrieve-type task templates across shopping_admin / gitlab / map. Per template: 3 train
tasks build the library (solved from scratch; only gold-verified solves are admitted), 2 held-out
tasks (unseen instances of the template — different parameter values) measure reuse. Every task is
solved both WITH the library and from scratch (BASE) — 100 solves total.
Per-task records and a reproduction driver are kept in the companion research repo and can be
shipped here on request.
Highlights:
library; net reuse-wins 7 vs 1 regression across the 20 held-out tasks.
33 steps (scratch) to 10 (reuse); a map routing task from 29 to 16.
here. (The gate is exactly as strong as its verifier — the default
self_verifyis a shapecheck only; see the README's "validation-gated" section.)
update.refinelifts per-instance differences into parametersand bakes the aggregation logic (top-n ranking, commit counting, route-time extraction) into
primitives, so unseen instances of the template solve by a direct
useof the skill.skill from the shared library (grown to 10 skills over the run), including telling apart two
near-duplicate gitlab commit-counting skills (by-date vs by-period).
evolvebatches produce 4 independent skills — new templates get added, existing skills arerefined in place (working functions kept), skills with no new traces stay byte-identical, and
zero cross-contamination between skills; held-out reuse against the mixed-built library matches
the per-template-built one.
Real website (public GitHub, read-only): the full loop end-to-end
Solve two repos from scratch ->
updatebuilds a parameterized skill -> a held-out repo is solved byreusing it (the agent calls
skill_use, verdictuse, answer correct). Reuse pays off most onmulti-step tasks where saved exploration outweighs the lookup overhead (see the WebArena numbers);
on short single-page lookups it is roughly break-even.
7 unit-test files under
tests/skill_factory/(library / gate / evolve / retrieve+decide / learn /learned-example lock / eval-snapshot lock) run in CI on every push touching the module
(
.github/workflows/skills-tests.yml).Status: a deliberately simplistic MVP
Most steps are a single LLM call (retrieve = one catalog prompt, decide = one prompt, refine =
one batched prompt) — chosen for clarity, not yet for scale/accuracy. The point is the modular
shape: each stage has a stable interface, so swapping in something stronger (embedding retrieval,
a learned ranker, WebJudge / cross-source consistency for the real-website gate) is a localized
change that does not touch the others or the agent loop.
Scope
Purely additive (zero deletions), confined to
src/webwright/skill_factory/(module + examples,including a checked-in learned skill),
src/webwright/tools/skill_use.py,tests/skill_factory/and one CI workflow. The actual implementation is ~670 lines of logic(non-blank, non-comment, across the skills module + the
skill_usetool); the rest is tests,examples, eval records, and docs. No edits to the agent loop, models, or existing configs.
Module README:
src/webwright/skill_factory/README.md.