From 260c3d9f5cc986ee5c537d595b23a579912e2bdd Mon Sep 17 00:00:00 2001 From: Paul Logan <101486603+laulpogan@users.noreply.github.com> Date: Tue, 21 Jul 2026 23:11:38 -0700 Subject: [PATCH] Correct README trust boundaries Replace stale visual proof and overbroad guarantees with current source-backed behavior, recorded terminal evidence, and a narrower first-run path. --- README.md | 132 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 92 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 82676f8..f71fa1b 100644 --- a/README.md +++ b/README.md @@ -4,62 +4,107 @@ [![CI](https://github.com/SlanchaAI/ingot/actions/workflows/ci.yml/badge.svg)](https://github.com/SlanchaAI/ingot/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/github/license/SlanchaAI/ingot)](LICENSE) [![Python 3.12](https://img.shields.io/badge/Python-3.12-3776AB?logo=python&logoColor=white)](Dockerfile) [![Docker](https://img.shields.io/badge/Docker-Compose-2496ED?logo=docker&logoColor=white)](docker-compose.yml) -

- Ingot, the mascot, handing skills out to AI agents -

- -An agent's [skills](https://github.com/anthropics/skills) are instructions it will follow. **Ingot** -is a local-first library that treats them as what they are: versioned state that needs a review -process. Every skill folder is content-addressed, every proposed change is quarantined until a human -reads the evidence and approves it, and promotion is atomic, snapshots what it replaced, and is -recorded. An **MCP server** then serves the approved revision of the right skill for each task, -which is what lets a cheap or local model reuse methods that would otherwise need a frontier model. -**SkillOpt integration** closes the loop: Ingot can learn from real agent traces, train skill -instructions with SkillOpt's reflective optimizer, compare the result on held-out tasks, and put a -measured proposal in front of a human reviewer. - -What the system guarantees: +An agent's [skills](https://github.com/anthropics/skills) are instructions it follows. **Ingot** is a +local-first library and MCP server for individual developers who serve versioned skills, evaluate +optimizer-generated challengers, and require human promotion before those challengers replace live +instructions. + +**SkillOpt integration** learns from real agent traces, trains bounded instruction edits, compares +them on held-out tasks, and produces an evidence-backed proposal. SkillOpt can propose a change but +cannot activate one. + +[Quickstart](#quickstart) · [Tutorial](docs/tutorial.md) · [How it works](docs/how-it-works.md) · +[Configuration](docs/configuration.md) · [Architecture](ARCHITECTURE.md) · +[Production setup](PRODUCTION_SETUP.md) · [Contributing](CONTRIBUTING.md) · +[Security](SECURITY.md) · [MIT license](LICENSE) + +## What Ingot controls - **A revision names an exact skill.** Every file in a skill folder is hashed, so the revision on a trace, in a piece of evidence, and on disk are comparable. -- **Changes are quarantined.** Generated rewrites land in `runs/pending/` and cannot route - traffic until a human approves them. +- **Optimizer-generated changes are quarantined.** Challengers land in `runs/pending/` and cannot + route traffic until a human promotes them. - **Approval needs evidence.** A rewrite carries held-out champion-vs-challenger scores, per-case deltas, token cost, and a gate verdict; promotion re-checks the evidence still matches disk. -- **Promotion is atomic and reversible.** The displaced revision is snapshotted and the directory - swapped by rename; restore any snapshot from the UI or CLI. -- **Decisions are audited.** Approvals, rejections, and rollbacks append metadata-only records (action, skill, - revision, actor, timestamp, and an optional rejection reason), never skill text or credentials. +- **Promotion is staged and recoverable.** Ingot snapshots the active revision, stages the + replacement, installs it with same-filesystem renames, and restores the prior directory if + installation fails. Restore any snapshot from the UI or CLI. +- **Decisions produce a local audit trail.** Approvals, rejections, and rollbacks attempt to append + metadata-only records after the transition. Audit-write failures are logged and do not roll back + the decision; the local trail is not tamper-proof. + +MCP serves the current contents of `skills/`. Direct edits, fetched skills, and copied or restored +folders bypass the proposal workflow and become active. Review them as trusted code before use. + +## A recorded gated change + +The current [tutorial](docs/tutorial.md) starts with stale Tailwind v3 instructions, trains a +challenger, and evaluates it on six held-out tasks. The recorded run improved the mean judge score +and reduced output tokens, but it also flagged a residual failure and an 80% body deletion: + +```text +[ab] champion: mean judge score 0.133 [0.0, 0.2, 0.2, 0.0, 0.2, 0.2] +[ab] challenger: mean judge score 0.667 [0.8, 1.0, 0.9, 0.9, 0.4, 0.0] +[ab] output tokens/task: 1278 -> 808 (-470) +[ab] ⚠ acceptance criteria (minority, flagged for review): 'no_v3_tailwind_directives': 1/6 holdout answer(s) matched +[ab] ⚠ challenger drops 80% of the champion body, gated on only 6 held-out task(s), review the deletions carefully +[ab] pending approval written to /app/runs/pending/tailwind.json - review + promote at http://localhost:8080 +``` + +The run stopped at a pending proposal; it did not change the live skill. The tutorial continues +through evidence review, deliberate promotion, and rollback. -Built for individual users first, ready to share: +## Deployment model - **Batteries included.** `docker compose up` starts the router, the change-control UI, and a self-hosted Langfuse for traces and experiments. An included Compose override connects Cloud or another self-hosted Langfuse without starting the bundled stack. -- **Local.** Point it at Ollama or vLLM and it runs with no API key; nothing leaves your machine. -- **Secure.** Hosted calls use OpenRouter with zero-data-retention routing enforced on every - request, everything binds localhost, and Compose password-gates the shared UI by default. +- **Local or hosted inference.** Point it at Ollama or vLLM for local model traffic. Hosted calls + use OpenRouter with zero-data-retention routing enforced on every request. +- **Local development by default.** Compose binds the public ports to localhost and password-gates + the UI. The MCP endpoint has no built-in authentication, and Ingot is not a hardened multi-tenant + service. Follow the production guide before sharing it beyond one trusted machine. - **Easy.** A skill is a folder with a `SKILL.md`. Drop one in and it is live on the next request. -SkillOpt is a central part of Ingot's value: it turns weak real-world runs into bounded instruction -edits, validates them against held-out tasks, and produces evidence-backed proposals. Running an -optimization is always an operator choice, and SkillOpt can propose but never activate a change. +## Quickstart -[Quickstart](#quickstart) · [Tutorial](docs/tutorial.md) · [How it works](docs/how-it-works.md) · -[Configuration](docs/configuration.md) · [Architecture](ARCHITECTURE.md) · -[Production setup](PRODUCTION_SETUP.md) · -[Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) · [MIT license](LICENSE) +Prerequisites: -## Quickstart +- Git, a running Docker daemon, and Docker Compose 2.24.4 or newer. +- Free localhost ports `8000`, `8080`, and `3100`. +- An OpenRouter API key, or a reachable OpenAI-compatible Ollama or vLLM endpoint. + +`scripts/fetch_skills.sh` copies unpinned third-party skills into the live `skills/` directory. For +this PDF demo, fetch only Anthropic's document skills. Review their instructions and per-skill +licenses in [Skill sources](docs/skill-sources.md) before running the fetch; add other sources after +the first run. ```bash git clone https://github.com/SlanchaAI/ingot.git && cd ingot -cp .env.example .env # add an OpenRouter key, or point BASE_URL at Ollama (no key) -scripts/fetch_skills.sh all # fetch ~70 real skills into ./skills (see docs/skill-sources.md) -docker compose up # router (:8000), UI (:8080), Langfuse (:3100), and one agent run +cp .env.example .env # set API_KEY, or point BASE_URL at Ollama or vLLM +scripts/fetch_skills.sh anthropics # fetch the document skills used by this demo +docker compose up -d --build # router (:8000), UI (:8080), Langfuse (:3100) +docker compose ps docker compose run --rm agent "How do I merge several PDFs into one and add page numbers?" ``` +A successful run names the route and the exact skill revision loaded before the answer. This is an +excerpt from the recorded tutorial run; scores, hashes, token counts, and model output vary: + +```text +COMPATIBLE ROUTE (MCP route_and_load): + 0.74 pdf: Use this skill whenever the user wants to do anything with PDF files... + +LOADED SKILLS (MCP route_and_load): ['pdf@83a75cf1f9b5…'] +TOKENS: 23233 in / 698 out + +RESULT: +... (working pypdf + reportlab code, following the loaded skill) +``` + +The `pdf@…` line is the first-value signal: Ingot routed the task and supplied a content-addressed +skill revision. The full trace appears in Langfuse at `localhost:3100`. + The change-control UI at `localhost:8080` asks for a login; the compose default is **`admin` / `ingot`**. Change `AUTH_PASSWORD` in `.env` before sharing it on your LAN. To run without a login, set `AUTH_MODE=open` explicitly. See [Privacy & security](docs/security.md#network-exposure). @@ -144,10 +189,10 @@ curl http://localhost:3100/api/public/health # should return a healthy Langfuse Ingot does three things around your skill library: -- **Serve.** An MCP server routes each task to the approved revision of the right skill (embedding - routing on CPU, no GPU) so a weak or local model can reuse a strong method. -- **Govern.** Every change is quarantined, carries held-out evidence, and needs a human approval; - promotion is atomic, snapshotted, reversible, and audited. +- **Serve.** An MCP server routes each task to the current live revision of the right skill (embedding + routing on CPU, no GPU) and supplies the current content-addressed instructions. +- **Govern.** Optimizer-generated challengers are quarantined, carry held-out evidence, and need + human promotion. Promotion is snapshotted and recoverable. - **Improve.** SkillOpt integration mines real traces for failing skills, trains bounded instruction edits with its reflective optimizer, and A/Bs the result on held-out tasks, leaving a reviewable proposal that only a human can activate. @@ -172,3 +217,10 @@ The component map is in [docs/how-it-works.md](docs/how-it-works.md); deeper des ## License MIT, see [LICENSE](LICENSE). + +## Project status and support + +Ingot supports local development for one trusted operator. Only the latest `master` revision +receives security fixes. Report bugs through [GitHub Issues](https://github.com/SlanchaAI/ingot/issues), +ask usage questions on [Discord](https://discord.gg/TtYVvwHJRQ), and report vulnerabilities through +the private process in [SECURITY.md](SECURITY.md).