Improve SkillOpt UI progress and review flow

README.md

@@ -14,9 +14,9 @@ process. Every skill folder is content-addressed, every proposed change is quara
 reads the evidence and approves it, and promotion is atomic, snapshots what it replaced, and is
 recorded. An **MCP server** then serves the approved revision of the right skill for each task,
 which is what lets a cheap or local model reuse methods that would otherwise need a frontier model.
-**SkillOpt integration** closes the loop: Ingot can learn from real agent traces, train skill
-instructions with SkillOpt's reflective optimizer, compare the result on held-out tasks, and put a
-measured proposal in front of a human reviewer.
+**[SkillOpt integration](https://github.com/microsoft/SkillOpt)** closes the loop: Ingot can learn
+from real agent traces, train skill instructions with SkillOpt's reflective optimizer, compare the
+result on held-out tasks, and put a measured proposal in front of a human reviewer.
 
 What the system guarantees:
 
@@ -34,8 +34,9 @@ What the system guarantees:
 Built for individual users first, ready to share:
 
 - **Batteries included.** `docker compose up` starts the router, the change-control UI, and a
-  self-hosted Langfuse for traces and experiments. An included Compose override connects Cloud or
-  another self-hosted Langfuse without starting the bundled stack.
+  self-hosted [Langfuse](https://github.com/langfuse/langfuse) for traces and experiments. An
+  included Compose override connects Cloud or another self-hosted Langfuse without starting the
+  bundled stack.
 - **Local.** Point it at Ollama or vLLM and it runs with no API key; nothing leaves your machine.
 - **Secure.** Hosted calls use OpenRouter with zero-data-retention routing enforced on every
   request, everything binds localhost, and Compose password-gates the shared UI by default.
@@ -74,6 +75,25 @@ settings live in [Configuration](docs/configuration.md).
 Then walk through the full loop, from mining a stale Tailwind skill through promotion, in the
 [**Tutorial**](docs/tutorial.md).
 
+### Optimize and review in the UI
+
+Click a skill's name or description to open its read-only detail view. The version selector lets
+you explore the active revision, a pending challenger, and rollback snapshots. The file selector
+shows `SKILL.md` plus any bundled resources for the selected version. Browsing does not change the
+revision being served. Each skill row shows the total number of active, pending, and snapshotted
+versions available in that explorer.
+
+Find a skill with an eval set and click **Optimize with SkillOpt**. The UI immediately explains
+that optimization can take a few minutes, disables the button, and opens the live generation log
+directly beneath that skill. The log updates automatically, so progress stays attached to the
+change you started instead of appearing in a page-level activity panel.
+
+![SkillOpt optimization progress shown directly beneath the tailwind skill](docs/ui-home.webp)
+
+When the run finishes, its challenger remains quarantined. Review the risk summary, unified or
+side-by-side diff, held-out task table, model names, and token changes before approving or
+rejecting it. Clicking **Approve & promote** in the comparison panel makes the final decision.
+
 For a hardened open-source Langfuse deployment with agents on other LAN machines, follow
 [Production setup](PRODUCTION_SETUP.md).
 

docs/how-it-works.md

@@ -41,4 +41,6 @@
 - **`ui/`**: FastAPI change-control UI (one HTML page, no build step): evidence, risk summary,
   unified and side-by-side diffs, approve or reasoned-reject decisions, revision exploration,
   history, rollback, and searchable skill inventory. It is the only normal application path that
-  activates a pending rewrite.
+  activates a pending rewrite. Clicking a skill opens a read-only explorer for its active revision,
+  pending challenger, and rollback snapshots, including `SKILL.md` and bundled files. Selecting a
+  version in this explorer never changes the served revision.

docs/tutorial.md

@@ -16,11 +16,15 @@ docker compose up --build
 
 This brings up the MCP server (`localhost:8000`), the change-control UI (`localhost:8080`), and a
 self-hosted Langfuse for traces and experiments (`localhost:3100`), then runs the agent once on a
-demo task. (First boot pulls the Langfuse stack, so give it a minute.) The UI lists every skill the router serves, each with its
-content-hash revision and a load count (how often it has actually been served), and surfaces
-anything awaiting review:
+demo task. (First boot pulls the Langfuse stack, so give it a minute.) The UI lists every skill the
+router serves, each with its content-hash revision and a load count (how often it has actually been
+served), and surfaces anything awaiting review.
 
-![Change-control UI, the skills library, with revisions and load counts](ui-home.webp)
+Click a skill's name or description to open its read-only detail view. Use **Version** to switch
+among the active revision, a pending challenger, and available rollback snapshots. Use **File** to
+inspect `SKILL.md` or any bundled resource in that version. This is an explorer only: selecting a
+version there does not promote, restore, or otherwise change what the router serves. The version
+count in each skill row includes the active revision, pending challenger, and rollback snapshots.
 
 No skills are committed to this repo. `fetch_skills.sh` clones each source, copies its skills in,
 and deletes the clone, so everything stays under its own upstream license. Without an API key
@@ -160,7 +164,12 @@ docker compose run --rm optimize tailwind
 
 SkillOpt reflects on failing minibatches, proposes bounded edits, keeps only train-set
 improvements, and A/Bs its best revision against the champion through the full agent on held-out
-tasks (about two minutes, a few cents). The UI's **Optimize with SkillOpt** button starts the same run.
+tasks (about two minutes, a few cents). The UI's **Optimize with SkillOpt** button starts the same
+run. After the click, the UI warns that optimization can take a few minutes and shows the live log
+directly beneath the selected skill. The button stays disabled until the run finishes.
+
+![SkillOpt optimization progress shown directly beneath the tailwind skill](ui-home.webp)
+
 Our run:
 
 ```
@@ -234,12 +243,12 @@ SkillOpt removed and the green lines are the v4 body it wrote:
 
 ![Review card, the tailwind v3→v4 challenger, promotable with warnings](ui-review.webp)
 
-**Approve** doesn't promote in one click: it opens a comparison panel with the real serving and
-judge model names, input tokens above output tokens, and a numbered before/after table for every
-held-out task. A final **Approve & promote** reveals a separate **Confirm**, so a promotion is
-deliberate.
+**Approve & promote** on the review card opens a comparison panel with the real serving and judge
+model names, input tokens above output tokens, and a numbered before/after table for every held-out
+task. **Approve & promote** in that panel is the final decision. There is no third confirmation
+button.
 
-![Comparison panel, model, tokens, and judge scores before Confirm](ui-compare.webp)
+![Comparison panel with models, numbered task scores, and input-first token usage](ui-compare.webp)
 
 **Approve & promote** verifies the evidence still matches the on-disk champion, snapshots the prior
 revision into `runs/revisions/tailwind/<revision>/`, swaps the challenger into `skills/tailwind/` by

optimize/rollout.py

@@ -10,7 +10,7 @@
 
 from langchain_openai import ChatOpenAI
 
-from . import agent_model, skillopt_model
+from . import agent_model, api_key, skillopt_model
 
 MODEL = agent_model()
 # The teacher LM (the skill *author*): a stronger model than the serving agent, per the
@@ -119,6 +119,7 @@ def reflection_lm(prompt) -> str:
         messages = prompt if isinstance(prompt, list) else [{"role": "user", "content": prompt}]
         if is_openrouter(base):
             response = litellm.completion(model=f"openrouter/{SKILLOPT_MODEL}", messages=messages,
+                                          api_base=base, api_key=api_key(),
                                           extra_body=openrouter_extra_body())
         else:
             kwargs = client_kwargs(base)

tests/test_optimize.py

@@ -164,6 +164,30 @@ def test_length_penalty_is_zero_under_target_and_grows_above():
     assert length_penalty("x" * (BODY_TARGET_CHARS * 100)) == LENGTH_PENALTY  # capped, never unbounded
 
 
+def test_reflection_lm_sends_generic_api_key_to_openrouter(monkeypatch):
+    import sys
+    from types import SimpleNamespace
+    from optimize import rollout as rollout_mod
+
+    captured = {}
+
+    def completion(**kwargs):
+        captured.update(kwargs)
+        message = SimpleNamespace(content="candidate")
+        return SimpleNamespace(choices=[SimpleNamespace(message=message)])
+
+    fake_litellm = SimpleNamespace(completion=completion, success_callback=None)
+    monkeypatch.setitem(sys.modules, "litellm", fake_litellm)
+    monkeypatch.setattr(rollout_mod, "teacher_base_url",
+                        lambda: "https://openrouter.ai/api/v1")
+    monkeypatch.setattr(rollout_mod, "api_key", lambda: "sk-or-generic")
+
+    assert rollout_mod.make_reflection_lm()("improve this skill") == "candidate"
+    assert captured["api_key"] == "sk-or-generic"
+    assert captured["api_base"] == "https://openrouter.ai/api/v1"
+    assert captured["model"] == f"openrouter/{rollout_mod.SKILLOPT_MODEL}"
+
+
 def test_failed_dimensions_flags_only_non_pass():
     dims = {"correctness": "pass", "completeness": "missing OCR", "instruction_following": "ok",
             "efficiency": "padded output"}

tests/test_ui.py

@@ -295,12 +295,14 @@ def test_pending_exposes_model_and_judge_for_the_comparison_panel(client):
 
 
 def test_comparison_panel_controls_are_in_the_page(client):
-    """The two-step approve panel (Approve -> comparison -> final Approve -> Confirm) must ship in
-    the served page, with the confirm button nested inside the modal overlay."""
-    layout = _Layout(client.get("/").text)
-    for element_id in ("cmp-overlay", "cmp-body", "cmp-approve", "cmp-confirm", "cmp-cancel"):
+    """The review action opens evidence, then the modal's primary button makes the decision."""
+    html = client.get("/").text
+    layout = _Layout(html)
+    for element_id in ("cmp-overlay", "cmp-body", "cmp-approve", "cmp-cancel"):
         assert element_id in layout.ancestors, f"comparison panel missing #{element_id}"
-    assert "cmp-overlay" in layout.ancestors["cmp-confirm"]
+    assert "cmp-overlay" in layout.ancestors["cmp-approve"]
+    assert "cmp-confirm" not in layout.ancestors
+    assert '$("#cmp-approve").onclick = () => act("#cmp-msg"' in html
 
 
 def test_review_panel_ships_risk_side_diff_and_confirmed_rejection(client):
@@ -328,6 +330,8 @@ def test_skill_list_ships_search_filters_version_explorer_and_live_updates(clien
     assert "signature !== boardSignature" in html
     assert "skillInventory.filter" in html
     assert "/api/skills/${encodeURIComponent(skill)}/versions" in html
+    assert "versionCount" in html and "active, pending, and snapshotted versions" in html
+    assert "renderSkills(skills, runs || runInventory, hist)" in html
 
 
 def test_comparison_panel_orders_tokens_and_tables_numbered_task_scores(client):
@@ -444,11 +448,21 @@ def test_index_leads_with_review_before_candidate_generation(client):
     html = client.get("/").text
     assert html.index('id="review-section"') < html.index('id="history-section"')
     assert html.index('id="history-section"') < html.index('id="skills"')
-    assert html.index('id="skills"') < html.index('id="run-section"')
+    assert 'id="run-section"' not in html
     assert "Evidence-gated change control" in html
     assert "change control" in html and "skill optimizer" not in html
 
 
+def test_index_nests_each_skillopt_log_under_its_skill(client):
+    html = client.get("/").text
+    assert "runInventory[s.name]" in html
+    assert 'class="skill-run-log"' in html
+    assert 'role="log"' in html and 'aria-live="polite"' in html
+    assert "renderSkills(skills, runs || runInventory, hist)" in html
+    assert "Optimization can take a few minutes" in html
+    assert 'id="log"' not in html
+
+
 def test_carn_viewer_is_gone(client):
     """The optional CARN integration was removed outright: no page, no routes, no config flag.
 
@@ -851,7 +865,8 @@ def test_index_follows_the_queue_when_the_reviewed_card_is_gone(client):
     assert "if (!quarantined.length) { showNoPending(); return; }" in html
     # a card opened by hand still clears the previous result
     assert "if (!keepMessage) say(\"#pending-msg\", \"\", false);" in html
-    assert 'onclick="showPending(\'${esc(s.name)}\')"' in html
+    assert 'onclick="showPending(\'${esc(s.name)}\', {scroll: true})"' in html
+    assert 'if (scroll) {' in html and '$("#review-section").scrollIntoView' in html
 
 
 def test_index_renders_the_board_when_history_is_unavailable(client):