Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 94 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,63 +4,107 @@

[![CI](https://github.com/SlanchaAI/ingot/actions/workflows/ci.yml/badge.svg)](https://github.com/SlanchaAI/ingot/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/github/license/SlanchaAI/ingot)](LICENSE) [![Python 3.12](https://img.shields.io/badge/Python-3.12-3776AB?logo=python&logoColor=white)](Dockerfile) [![Docker](https://img.shields.io/badge/Docker-Compose-2496ED?logo=docker&logoColor=white)](docker-compose.yml)

<p align="center">
<img src="docs/ingot.jpg" alt="Ingot, the mascot, handing skills out to AI agents" width="640">
</p>

An agent's [skills](https://github.com/anthropics/skills) are instructions it will follow. **Ingot**
is a local-first library that treats them as what they are: versioned state that needs a review
process. Every skill folder is content-addressed, every proposed change is quarantined until a human
reads the evidence and approves it, and promotion is atomic, snapshots what it replaced, and is
recorded. An **MCP server** then serves the approved revision of the right skill for each task,
which is what lets a cheap or local model reuse methods that would otherwise need a frontier model.
**[SkillOpt integration](https://github.com/microsoft/SkillOpt)** closes the loop: Ingot can learn
from real agent traces, train skill instructions with SkillOpt's reflective optimizer, compare the
result on held-out tasks, and put a measured proposal in front of a human reviewer.

What the system guarantees:
An agent's [skills](https://github.com/anthropics/skills) are instructions it follows. **Ingot** is a
local-first library and MCP server for individual developers who serve versioned skills, evaluate
optimizer-generated challengers, and require human promotion before those challengers replace live
instructions.

**SkillOpt integration** learns from real agent traces, trains bounded instruction edits, compares
them on held-out tasks, and produces an evidence-backed proposal. SkillOpt can propose a change but
cannot activate one.

[Quickstart](#quickstart) · [Tutorial](docs/tutorial.md) · [How it works](docs/how-it-works.md) ·
[Configuration](docs/configuration.md) · [Architecture](ARCHITECTURE.md) ·
[Production setup](PRODUCTION_SETUP.md) · [Contributing](CONTRIBUTING.md) ·
[Security](SECURITY.md) · [MIT license](LICENSE)

## What Ingot controls

- **A revision names an exact skill.** Every file in a skill folder is hashed, so the revision on a
trace, in a piece of evidence, and on disk are comparable.
- **Changes are quarantined.** Generated rewrites land in `runs/pending/` and cannot route
traffic until a human approves them.
- **Optimizer-generated changes are quarantined.** Challengers land in `runs/pending/` and cannot
route traffic until a human promotes them.
- **Approval needs evidence.** A rewrite carries held-out champion-vs-challenger scores, per-case
deltas, token cost, and a gate verdict; promotion re-checks the evidence still matches disk.
- **Promotion is atomic and reversible.** The displaced revision is snapshotted and the directory
swapped by rename; restore any snapshot from the UI or CLI.
- **Decisions are audited.** Approvals, rejections, and rollbacks append metadata-only records (action, skill,
revision, actor, timestamp, and an optional rejection reason), never skill text or credentials.
- **Promotion is staged and recoverable.** Ingot snapshots the active revision, stages the
replacement, installs it with same-filesystem renames, and restores the prior directory if
installation fails. Restore any snapshot from the UI or CLI.
- **Decisions produce a local audit trail.** Approvals, rejections, and rollbacks attempt to append
metadata-only records after the transition. Audit-write failures are logged and do not roll back
the decision; the local trail is not tamper-proof.

MCP serves the current contents of `skills/`. Direct edits, fetched skills, and copied or restored
folders bypass the proposal workflow and become active. Review them as trusted code before use.

## A recorded gated change

The current [tutorial](docs/tutorial.md) starts with stale Tailwind v3 instructions, trains a
challenger, and evaluates it on six held-out tasks. The recorded run improved the mean judge score
and reduced output tokens, but it also flagged a residual failure and an 80% body deletion:

```text
[ab] champion: mean judge score 0.133 [0.0, 0.2, 0.2, 0.0, 0.2, 0.2]
[ab] challenger: mean judge score 0.667 [0.8, 1.0, 0.9, 0.9, 0.4, 0.0]
[ab] output tokens/task: 1278 -> 808 (-470)
[ab] ⚠ acceptance criteria (minority, flagged for review): 'no_v3_tailwind_directives': 1/6 holdout answer(s) matched
[ab] ⚠ challenger drops 80% of the champion body, gated on only 6 held-out task(s), review the deletions carefully
[ab] pending approval written to /app/runs/pending/tailwind.json - review + promote at http://localhost:8080
```

The run stopped at a pending proposal; it did not change the live skill. The tutorial continues
through evidence review, deliberate promotion, and rollback.

Built for individual users first, ready to share:
## Deployment model

- **Batteries included.** `docker compose up` starts the router, the change-control UI, and a
self-hosted [Langfuse](https://github.com/langfuse/langfuse) for traces and experiments. An
included Compose override connects Cloud or another self-hosted Langfuse without starting the
bundled stack.
- **Local.** Point it at Ollama or vLLM and it runs with no API key; nothing leaves your machine.
- **Secure.** Hosted calls use OpenRouter with zero-data-retention routing enforced on every
request, everything binds localhost, and Compose password-gates the shared UI by default.
self-hosted Langfuse for traces and experiments. An included Compose override connects Cloud or
another self-hosted Langfuse without starting the bundled stack.
- **Local or hosted inference.** Point it at Ollama or vLLM for local model traffic. Hosted calls
use OpenRouter with zero-data-retention routing enforced on every request.
- **Local development by default.** Compose binds the public ports to localhost and password-gates
the UI. The MCP endpoint has no built-in authentication, and Ingot is not a hardened multi-tenant
service. Follow the production guide before sharing it beyond one trusted machine.
- **Easy.** A skill is a folder with a `SKILL.md`. Drop one in and it is live on the next request.

SkillOpt is a central part of Ingot's value: it turns weak real-world runs into bounded instruction
edits, validates them against held-out tasks, and produces evidence-backed proposals. Running an
optimization is always an operator choice, and SkillOpt can propose but never activate a change.
## Quickstart

[Quickstart](#quickstart) · [Tutorial](docs/tutorial.md) · [How it works](docs/how-it-works.md) ·
[Configuration](docs/configuration.md) · [Architecture](ARCHITECTURE.md) ·
[Production setup](PRODUCTION_SETUP.md) ·
[Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) · [MIT license](LICENSE)
Prerequisites:

## Quickstart
- Git, a running Docker daemon, and Docker Compose 2.24.4 or newer.
- Free localhost ports `8000`, `8080`, and `3100`.
- An OpenRouter API key, or a reachable OpenAI-compatible Ollama or vLLM endpoint.

`scripts/fetch_skills.sh` copies unpinned third-party skills into the live `skills/` directory. For
this PDF demo, fetch only Anthropic's document skills. Review their instructions and per-skill
licenses in [Skill sources](docs/skill-sources.md) before running the fetch; add other sources after
the first run.

```bash
git clone https://github.com/SlanchaAI/ingot.git && cd ingot
cp .env.example .env # add an OpenRouter key, or point BASE_URL at Ollama (no key)
scripts/fetch_skills.sh all # fetch ~70 real skills into ./skills (see docs/skill-sources.md)
docker compose up # router (:8000), UI (:8080), Langfuse (:3100), and one agent run
cp .env.example .env # set API_KEY, or point BASE_URL at Ollama or vLLM
scripts/fetch_skills.sh anthropics # fetch the document skills used by this demo
docker compose up -d --build # router (:8000), UI (:8080), Langfuse (:3100)
docker compose ps
docker compose run --rm agent "How do I merge several PDFs into one and add page numbers?"
```

A successful run names the route and the exact skill revision loaded before the answer. This is an
excerpt from the recorded tutorial run; scores, hashes, token counts, and model output vary:

```text
COMPATIBLE ROUTE (MCP route_and_load):
0.74 pdf: Use this skill whenever the user wants to do anything with PDF files...

LOADED SKILLS (MCP route_and_load): ['pdf@83a75cf1f9b5…']
TOKENS: 23233 in / 698 out

RESULT:
... (working pypdf + reportlab code, following the loaded skill)
```

The `pdf@…` line is the first-value signal: Ingot routed the task and supplied a content-addressed
skill revision. The full trace appears in Langfuse at `localhost:3100`.

The change-control UI at `localhost:8080` asks for a login; the compose default is **`admin` /
`ingot`**. Change `AUTH_PASSWORD` in `.env` before sharing it on your LAN. To run without a login,
set `AUTH_MODE=open` explicitly. See [Privacy & security](docs/security.md#network-exposure).
Expand Down Expand Up @@ -164,10 +208,10 @@ curl http://localhost:3100/api/public/health # should return a healthy Langfuse

Ingot does three things around your skill library:

- **Serve.** An MCP server routes each task to the approved revision of the right skill (embedding
routing on CPU, no GPU) so a weak or local model can reuse a strong method.
- **Govern.** Every change is quarantined, carries held-out evidence, and needs a human approval;
promotion is atomic, snapshotted, reversible, and audited.
- **Serve.** An MCP server routes each task to the current live revision of the right skill (embedding
routing on CPU, no GPU) and supplies the current content-addressed instructions.
- **Govern.** Optimizer-generated challengers are quarantined, carry held-out evidence, and need
human promotion. Promotion is snapshotted and recoverable.
- **Improve.** SkillOpt integration mines real traces for failing skills, trains bounded instruction
edits with its reflective optimizer, and A/Bs the result on held-out tasks, leaving a reviewable
proposal that only a human can activate.
Expand All @@ -192,3 +236,10 @@ The component map is in [docs/how-it-works.md](docs/how-it-works.md); deeper des
## License

MIT, see [LICENSE](LICENSE).

## Project status and support

Ingot supports local development for one trusted operator. Only the latest `master` revision
receives security fixes. Report bugs through [GitHub Issues](https://github.com/SlanchaAI/ingot/issues),
ask usage questions on [Discord](https://discord.gg/TtYVvwHJRQ), and report vulnerabilities through
the private process in [SECURITY.md](SECURITY.md).
Loading