diff --git a/README.md b/README.md index 82676f8..e2a1b5d 100644 --- a/README.md +++ b/README.md @@ -14,9 +14,9 @@ process. Every skill folder is content-addressed, every proposed change is quara reads the evidence and approves it, and promotion is atomic, snapshots what it replaced, and is recorded. An **MCP server** then serves the approved revision of the right skill for each task, which is what lets a cheap or local model reuse methods that would otherwise need a frontier model. -**SkillOpt integration** closes the loop: Ingot can learn from real agent traces, train skill -instructions with SkillOpt's reflective optimizer, compare the result on held-out tasks, and put a -measured proposal in front of a human reviewer. +**[SkillOpt integration](https://github.com/microsoft/SkillOpt)** closes the loop: Ingot can learn +from real agent traces, train skill instructions with SkillOpt's reflective optimizer, compare the +result on held-out tasks, and put a measured proposal in front of a human reviewer. What the system guarantees: @@ -34,8 +34,9 @@ What the system guarantees: Built for individual users first, ready to share: - **Batteries included.** `docker compose up` starts the router, the change-control UI, and a - self-hosted Langfuse for traces and experiments. An included Compose override connects Cloud or - another self-hosted Langfuse without starting the bundled stack. + self-hosted [Langfuse](https://github.com/langfuse/langfuse) for traces and experiments. An + included Compose override connects Cloud or another self-hosted Langfuse without starting the + bundled stack. - **Local.** Point it at Ollama or vLLM and it runs with no API key; nothing leaves your machine. - **Secure.** Hosted calls use OpenRouter with zero-data-retention routing enforced on every request, everything binds localhost, and Compose password-gates the shared UI by default. @@ -74,6 +75,25 @@ settings live in [Configuration](docs/configuration.md). Then walk through the full loop, from mining a stale Tailwind skill through promotion, in the [**Tutorial**](docs/tutorial.md). +### Optimize and review in the UI + +Click a skill's name or description to open its read-only detail view. The version selector lets +you explore the active revision, a pending challenger, and rollback snapshots. The file selector +shows `SKILL.md` plus any bundled resources for the selected version. Browsing does not change the +revision being served. Each skill row shows the total number of active, pending, and snapshotted +versions available in that explorer. + +Find a skill with an eval set and click **Optimize with SkillOpt**. The UI immediately explains +that optimization can take a few minutes, disables the button, and opens the live generation log +directly beneath that skill. The log updates automatically, so progress stays attached to the +change you started instead of appearing in a page-level activity panel. + +![SkillOpt optimization progress shown directly beneath the tailwind skill](docs/ui-home.webp) + +When the run finishes, its challenger remains quarantined. Review the risk summary, unified or +side-by-side diff, held-out task table, model names, and token changes before approving or +rejecting it. Clicking **Approve & promote** in the comparison panel makes the final decision. + For a hardened open-source Langfuse deployment with agents on other LAN machines, follow [Production setup](PRODUCTION_SETUP.md). diff --git a/docs/how-it-works.md b/docs/how-it-works.md index b7aec74..8323459 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -41,4 +41,6 @@ - **`ui/`**: FastAPI change-control UI (one HTML page, no build step): evidence, risk summary, unified and side-by-side diffs, approve or reasoned-reject decisions, revision exploration, history, rollback, and searchable skill inventory. It is the only normal application path that - activates a pending rewrite. + activates a pending rewrite. Clicking a skill opens a read-only explorer for its active revision, + pending challenger, and rollback snapshots, including `SKILL.md` and bundled files. Selecting a + version in this explorer never changes the served revision. diff --git a/docs/tutorial.md b/docs/tutorial.md index 3a1c960..b1b491d 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -16,11 +16,15 @@ docker compose up --build This brings up the MCP server (`localhost:8000`), the change-control UI (`localhost:8080`), and a self-hosted Langfuse for traces and experiments (`localhost:3100`), then runs the agent once on a -demo task. (First boot pulls the Langfuse stack, so give it a minute.) The UI lists every skill the router serves, each with its -content-hash revision and a load count (how often it has actually been served), and surfaces -anything awaiting review: +demo task. (First boot pulls the Langfuse stack, so give it a minute.) The UI lists every skill the +router serves, each with its content-hash revision and a load count (how often it has actually been +served), and surfaces anything awaiting review. -![Change-control UI, the skills library, with revisions and load counts](ui-home.webp) +Click a skill's name or description to open its read-only detail view. Use **Version** to switch +among the active revision, a pending challenger, and available rollback snapshots. Use **File** to +inspect `SKILL.md` or any bundled resource in that version. This is an explorer only: selecting a +version there does not promote, restore, or otherwise change what the router serves. The version +count in each skill row includes the active revision, pending challenger, and rollback snapshots. No skills are committed to this repo. `fetch_skills.sh` clones each source, copies its skills in, and deletes the clone, so everything stays under its own upstream license. Without an API key @@ -160,7 +164,12 @@ docker compose run --rm optimize tailwind SkillOpt reflects on failing minibatches, proposes bounded edits, keeps only train-set improvements, and A/Bs its best revision against the champion through the full agent on held-out -tasks (about two minutes, a few cents). The UI's **Optimize with SkillOpt** button starts the same run. +tasks (about two minutes, a few cents). The UI's **Optimize with SkillOpt** button starts the same +run. After the click, the UI warns that optimization can take a few minutes and shows the live log +directly beneath the selected skill. The button stays disabled until the run finishes. + +![SkillOpt optimization progress shown directly beneath the tailwind skill](ui-home.webp) + Our run: ``` @@ -234,12 +243,12 @@ SkillOpt removed and the green lines are the v4 body it wrote: ![Review card, the tailwind v3→v4 challenger, promotable with warnings](ui-review.webp) -**Approve** doesn't promote in one click: it opens a comparison panel with the real serving and -judge model names, input tokens above output tokens, and a numbered before/after table for every -held-out task. A final **Approve & promote** reveals a separate **Confirm**, so a promotion is -deliberate. +**Approve & promote** on the review card opens a comparison panel with the real serving and judge +model names, input tokens above output tokens, and a numbered before/after table for every held-out +task. **Approve & promote** in that panel is the final decision. There is no third confirmation +button. -![Comparison panel, model, tokens, and judge scores before Confirm](ui-compare.webp) +![Comparison panel with models, numbered task scores, and input-first token usage](ui-compare.webp) **Approve & promote** verifies the evidence still matches the on-disk champion, snapshots the prior revision into `runs/revisions/tailwind//`, swaps the challenger into `skills/tailwind/` by diff --git a/docs/ui-compare.webp b/docs/ui-compare.webp index 9cd1c68..2f29c59 100644 Binary files a/docs/ui-compare.webp and b/docs/ui-compare.webp differ diff --git a/docs/ui-home.webp b/docs/ui-home.webp index 2073206..7905099 100644 Binary files a/docs/ui-home.webp and b/docs/ui-home.webp differ diff --git a/docs/ui-review.webp b/docs/ui-review.webp index 86d1d75..af0adea 100644 Binary files a/docs/ui-review.webp and b/docs/ui-review.webp differ diff --git a/optimize/rollout.py b/optimize/rollout.py index 92be2bf..8f3f949 100644 --- a/optimize/rollout.py +++ b/optimize/rollout.py @@ -10,7 +10,7 @@ from langchain_openai import ChatOpenAI -from . import agent_model, skillopt_model +from . import agent_model, api_key, skillopt_model MODEL = agent_model() # The teacher LM (the skill *author*): a stronger model than the serving agent, per the @@ -119,6 +119,7 @@ def reflection_lm(prompt) -> str: messages = prompt if isinstance(prompt, list) else [{"role": "user", "content": prompt}] if is_openrouter(base): response = litellm.completion(model=f"openrouter/{SKILLOPT_MODEL}", messages=messages, + api_base=base, api_key=api_key(), extra_body=openrouter_extra_body()) else: kwargs = client_kwargs(base) diff --git a/tests/test_optimize.py b/tests/test_optimize.py index 5064afc..4971d97 100644 --- a/tests/test_optimize.py +++ b/tests/test_optimize.py @@ -164,6 +164,30 @@ def test_length_penalty_is_zero_under_target_and_grows_above(): assert length_penalty("x" * (BODY_TARGET_CHARS * 100)) == LENGTH_PENALTY # capped, never unbounded +def test_reflection_lm_sends_generic_api_key_to_openrouter(monkeypatch): + import sys + from types import SimpleNamespace + from optimize import rollout as rollout_mod + + captured = {} + + def completion(**kwargs): + captured.update(kwargs) + message = SimpleNamespace(content="candidate") + return SimpleNamespace(choices=[SimpleNamespace(message=message)]) + + fake_litellm = SimpleNamespace(completion=completion, success_callback=None) + monkeypatch.setitem(sys.modules, "litellm", fake_litellm) + monkeypatch.setattr(rollout_mod, "teacher_base_url", + lambda: "https://openrouter.ai/api/v1") + monkeypatch.setattr(rollout_mod, "api_key", lambda: "sk-or-generic") + + assert rollout_mod.make_reflection_lm()("improve this skill") == "candidate" + assert captured["api_key"] == "sk-or-generic" + assert captured["api_base"] == "https://openrouter.ai/api/v1" + assert captured["model"] == f"openrouter/{rollout_mod.SKILLOPT_MODEL}" + + def test_failed_dimensions_flags_only_non_pass(): dims = {"correctness": "pass", "completeness": "missing OCR", "instruction_following": "ok", "efficiency": "padded output"} diff --git a/tests/test_ui.py b/tests/test_ui.py index 42b7aee..3652975 100644 --- a/tests/test_ui.py +++ b/tests/test_ui.py @@ -295,12 +295,14 @@ def test_pending_exposes_model_and_judge_for_the_comparison_panel(client): def test_comparison_panel_controls_are_in_the_page(client): - """The two-step approve panel (Approve -> comparison -> final Approve -> Confirm) must ship in - the served page, with the confirm button nested inside the modal overlay.""" - layout = _Layout(client.get("/").text) - for element_id in ("cmp-overlay", "cmp-body", "cmp-approve", "cmp-confirm", "cmp-cancel"): + """The review action opens evidence, then the modal's primary button makes the decision.""" + html = client.get("/").text + layout = _Layout(html) + for element_id in ("cmp-overlay", "cmp-body", "cmp-approve", "cmp-cancel"): assert element_id in layout.ancestors, f"comparison panel missing #{element_id}" - assert "cmp-overlay" in layout.ancestors["cmp-confirm"] + assert "cmp-overlay" in layout.ancestors["cmp-approve"] + assert "cmp-confirm" not in layout.ancestors + assert '$("#cmp-approve").onclick = () => act("#cmp-msg"' in html def test_review_panel_ships_risk_side_diff_and_confirmed_rejection(client): @@ -328,6 +330,8 @@ def test_skill_list_ships_search_filters_version_explorer_and_live_updates(clien assert "signature !== boardSignature" in html assert "skillInventory.filter" in html assert "/api/skills/${encodeURIComponent(skill)}/versions" in html + assert "versionCount" in html and "active, pending, and snapshotted versions" in html + assert "renderSkills(skills, runs || runInventory, hist)" in html def test_comparison_panel_orders_tokens_and_tables_numbered_task_scores(client): @@ -444,11 +448,21 @@ def test_index_leads_with_review_before_candidate_generation(client): html = client.get("/").text assert html.index('id="review-section"') < html.index('id="history-section"') assert html.index('id="history-section"') < html.index('id="skills"') - assert html.index('id="skills"') < html.index('id="run-section"') + assert 'id="run-section"' not in html assert "Evidence-gated change control" in html assert "change control" in html and "skill optimizer" not in html +def test_index_nests_each_skillopt_log_under_its_skill(client): + html = client.get("/").text + assert "runInventory[s.name]" in html + assert 'class="skill-run-log"' in html + assert 'role="log"' in html and 'aria-live="polite"' in html + assert "renderSkills(skills, runs || runInventory, hist)" in html + assert "Optimization can take a few minutes" in html + assert 'id="log"' not in html + + def test_carn_viewer_is_gone(client): """The optional CARN integration was removed outright: no page, no routes, no config flag. @@ -851,7 +865,8 @@ def test_index_follows_the_queue_when_the_reviewed_card_is_gone(client): assert "if (!quarantined.length) { showNoPending(); return; }" in html # a card opened by hand still clears the previous result assert "if (!keepMessage) say(\"#pending-msg\", \"\", false);" in html - assert 'onclick="showPending(\'${esc(s.name)}\')"' in html + assert 'onclick="showPending(\'${esc(s.name)}\', {scroll: true})"' in html + assert 'if (scroll) {' in html and '$("#review-section").scrollIntoView' in html def test_index_renders_the_board_when_history_is_unavailable(client): diff --git a/ui/static/index.html b/ui/static/index.html index ad2ba84..4a69970 100644 --- a/ui/static/index.html +++ b/ui/static/index.html @@ -169,7 +169,7 @@ .btn.danger { border-color: var(--fail); background: var(--fail); color: var(--paper); } .btn.danger:hover:not(:disabled) { filter: brightness(.92); } - /* Promotion comparison modal (two-step approve) */ + /* Promotion comparison modal */ .cmp-overlay { position: fixed; inset: 0; background: rgba(15,20,32,.6); z-index: 50; display: flex; align-items: center; justify-content: center; padding: 1.5rem; } .cmp-overlay[hidden] { display: none; } @@ -257,10 +257,17 @@ .audit .aact { font-weight: 700; color: var(--rust); text-transform: uppercase; } .audit .aact.reject { color: var(--fail); } /* a discarded challenger reads distinct from a promotion */ - /* ---- candidate-generation log ---- */ - .log { font-family: var(--mono); font-size: .74rem; line-height: 1.6; margin-top: 1.1rem; + /* SkillOpt run log, nested under the skill being optimized */ + .skill-run { flex: 1 0 100%; min-width: 0; margin-top: .25rem; } + .skill-run-head { display: flex; justify-content: space-between; gap: .75rem; margin-bottom: .35rem; + font: 600 .66rem/1.4 var(--mono); color: var(--rust); text-transform: uppercase; + letter-spacing: .06em; } + .skill-run-head .count { color: var(--ink-3); font-weight: 400; } + .skill-run-note { margin: 0 0 .45rem; font: .72rem/1.5 var(--mono); color: var(--ink-2); } + .skill-run-log { font-family: var(--mono); font-size: .74rem; line-height: 1.6; margin: 0; background: var(--paper-2); border: 1px solid var(--line); border-radius: 14px; box-shadow: var(--shadow-sm); - padding: .85rem 1rem; overflow: auto; max-height: 22rem; white-space: pre-wrap; color: var(--ink-2); } + padding: .85rem 1rem; overflow: auto; max-height: 18rem; white-space: pre-wrap; color: var(--ink-2); } + .skill-run.error .skill-run-head { color: var(--fail); } /* ---- review: champion vs challenger ---- */ .duel { display: grid; grid-template-columns: 1fr auto 1fr; gap: 1rem; align-items: stretch; margin-top: 1.3rem; } @@ -396,7 +403,6 @@ Review History Skills - Activity Langfuse …