fixture-bank

A DSL and generation engine for load-test fixture data, investigated and generated by agents exploring code and schemas

What is this?

fixture-bank is a tool that lets an agent (e.g. Claude Code) investigate and design the "prerequisite data" needed for load testing, then generate it deterministically in bulk.

The flow is simple:

1. The user or agent describes a load-test scenario (e.g. "I want to load test with level 50 users who have a premium pass")
2. The agent investigates the code and schema to understand the shape of data the scenario needs
3. Based on that investigation, the agent generates a Fixture Bank DSL
4. Using the DSL as input, fixture-bank generates any number of records as SQL (applied directly to the DB) or JSON (for inspecting the contents)

The key point is that the LLM is only invoked in step 3, writing the DSL. Whether you generate 1 record or 100,000, step 4 runs as an ordinary deterministic program, so there's no token cost or variance in the output.

Why it's needed

• 📉 Can't parallelize work: When scenarios run in series, downstream testing is blocked until upstream bottlenecks are resolved
• 🧩 Setting up prerequisite data pollutes the system under test: When initialization and the measured process share the same code path, accurate measurement becomes impossible
• 🕳️ Coverage doesn't improve: Scenarios tend to always start from a fresh-user-like state, leaving accumulated data states (aged state) unverified

fixture-bank addresses these by extracting the logic for generating prerequisite data into a DSL and turning it into a reusable asset. See DESIGN.md for the detailed design, and MCP_TOOLS.md for the MCP tool interface.

"Couldn't the agent just write a seed script?"

Agents can write factory_bot or Python seed scripts directly, so what does a DSL buy you? Four things:

• Determinism: A free-form script varies in its details (randomness, null rates, distributions) every time it's authored, which makes before/after load-test comparisons meaningless. The DSL restricts generation to deterministic generators — same DSL + same seed always yields the same data, guaranteed by the tool rather than by how carefully the script happened to be written.
• Tokens don't scale with record count: The LLM authors a few dozen declarative lines once; materialize is an ordinary program. Previewing 1 record and inserting 100,000 cost the same zero additional LLM tokens — no write/run/debug loop over a growing script.
• Three-stage validation: A script can only be validated by running it. A DSL has a fixed structure, so it's checked mechanically: syntax → schema integrity against the real database (introspect_schema) → a sandboxed trial insert (always rolled back) that catches real constraint violations. And since materialize only interprets the DSL, there's no arbitrary code execution — no surprise side effects against your database.
• Fixtures become reusable assets: Seed scripts are one-offs tied to a language and a point-in-time schema. A tagged DSL (save_fixture) is short, diffable, replayable by anyone via --fixture <tag>, and stale fixtures are detected mechanically by re-running validation after schema changes.

The full argument is in DESIGN.md §3.

How does this compare to other data-generation tools?

None of these do exactly what fixture-bank does, but the neighborhood is crowded enough to be explicit about it:

• Seedfast is the closest: an AI-powered CLI for PostgreSQL that reads your live schema and generates relationally-consistent test data from natural language each run, with an MCP server for Claude Code and similar clients. The agent × schema-introspection × Postgres × MCP shape overlaps heavily with fixture-bank. The real difference is architectural: Seedfast uses the LLM as the generator on every invocation, and by its own account optimizes for data that "looks realistic" over byte-identical reproducibility. fixture-bank's DSL exists to reject that tradeoff — the LLM authors a DSL once, and materialize reproduces it deterministically afterward with no further LLM involvement.
• Tonic Fabricate (commercial) generates synthetic data through an AI agent and chat UI, is usable from MCP clients, and explicitly names load testing as a use case. It's the best-funded adjacent tool, but it's built around enterprise test-data-management broadly rather than a small, git-committable, reviewable DSL artifact.
• Snaplet Seed (now maintained as supabase-community/seed) shared the determinism instinct via its Copycat generator, but the company behind it shut down in 2024 and it's effectively in maintenance mode — a TypeScript client rather than a DSL, and it predates MCP.
• Synth is open-source prior art for "declarative, git-reviewable data model, constraint-based, scales to millions of rows" — roughly half of fixture-bank's DSL argument — but development has stalled, and it has no agent integration or pre-execution DB validation.

Every individual ingredient here (a declarative DSL, deterministic generation, MCP, schema introspection) has prior art somewhere. fixture-bank's claim is the combination: the LLM only authors the DSL, generation is validated in three stages before anything touches real data (see below), and the explicit job is load-test prerequisite data, including "aged state" scenarios. The sandboxed trial-insert validation stage in particular doesn't appear to exist in any of the tools above.

Quick picture

# Generate 1000 records as SQL from a saved DSL and apply to the DB
$ fixture-bank materialize --fixture user:level50:has_premium_pass --count 1000 --format sql | psql mydb

# Just want to check the contents? Use JSON
$ fixture-bank materialize --fixture user:level50:has_premium_pass --count 3 --format json

Current status

🚧 Implementation phase. The Go implementation already covers the DSL parser, the full generator suite, materialize (SQL/JSON output), fixture save/tag management, PostgreSQL integration (pool_ref/unique: db), and the MCP server (introspect_schema/draft_dsl/materialize/save_fixture). See TODO.md for details.

Build & run

$ go build -o fixture-bank ./cmd/fixture-bank

# Generate JSON from a DSL file
$ ./fixture-bank materialize --dsl fixture.yaml --count 3 --format json

# Generate as SQL (INSERT statements)
$ ./fixture-bank materialize --dsl fixture.yaml --count 1000 --format sql --db-url "$DATABASE_URL"

# Save a fixture with a tag, then generate from the tag
$ ./fixture-bank fixture save --dsl fixture.yaml --tag user:level50:has_premium_pass
$ ./fixture-bank materialize --fixture user:level50:has_premium_pass --count 500 --format sql

Using fixture-bank as an MCP server

fixture-bank mcp runs fixture-bank as an MCP server over stdio, so an agent (Claude Code, Claude Desktop, or any other MCP client) can investigate your schema and author/validate a DSL itself, instead of you writing the DSL by hand. This is what DESIGN.md calls the "author" side of the author/materialize split — the agent designs the generation logic once, and materialize reproduces it deterministically for any record count.

$ ./fixture-bank mcp --db-url "$DATABASE_URL" --store-dir ./fixtures

Flags:

Flag	Default	Purpose
--db-url	(none)	PostgreSQL connection string. Required for introspect_schema, for draft_dsl's schema/db_execution validation stages, and for materialize/draft_dsl when the DSL uses pool_ref or unique: db. If omitted, calls that need it return an error_type: db_error result rather than failing at startup
--store-dir	./fixtures	Where save_fixture writes tagged DSL files

Registering with an MCP client

Claude Code:

$ claude mcp add fixture-bank -- /path/to/fixture-bank mcp --db-url "$DATABASE_URL" --store-dir ./fixtures

Claude Desktop (and other clients using the same claude_desktop_config.json style config):

{
  "mcpServers": {
    "fixture-bank": {
      "command": "/path/to/fixture-bank",
      "args": ["mcp", "--db-url", "postgres://user:pass@localhost:5432/mydb", "--store-dir", "./fixtures"]
    }
  }
}

GitHub Copilot CLI:

Add an entry to ~/.copilot/mcp-config.json (create the file if it doesn't exist yet):

{
  "mcpServers": {
    "fixture-bank": {
      "command": "/path/to/fixture-bank",
      "args": ["mcp", "--db-url", "postgres://user:pass@localhost:5432/mydb", "--store-dir", "./fixtures"]
    }
  }
}

Restart copilot, then confirm it's connected with /mcp show.

Gemini CLI:

Add an entry to ~/.gemini/settings.json (global) or .gemini/settings.json (project-local):

{
  "mcpServers": {
    "fixture-bank": {
      "command": "/path/to/fixture-bank",
      "args": ["mcp", "--db-url", "postgres://user:pass@localhost:5432/mydb", "--store-dir", "./fixtures"]
    }
  }
}

Available tools

Tool	What it does
introspect_schema	Inspects tables/columns/constraints (PRIMARY KEY, UNIQUE, FOREIGN KEY) in the public schema, so an agent can learn the real shape of the database before writing a DSL
draft_dsl	Validates a candidate DSL in the three stages from DSL_SPEC.md ¤5: syntax → schema integrity (against introspect_schema) → sandbox DB execution (a trial insert inside a transaction that is always rolled back, so no data is ever persisted). Returns valid: true, or valid: false plus which stage failed and why
materialize	Generates records from a DSL as JSON or SQL, with optional count/seed overrides — the same engine the materialize CLI command uses
save_fixture	Saves a DSL under a tag (e.g. user:level50:has_premium_pass) so it can be replayed later via fixture-bank materialize --fixture <tag>

A typical agent workflow: call introspect_schema to learn the tables, iterate on a DSL with draft_dsl until valid: true, then call materialize to preview a few JSON records, and save_fixture once it looks right.

See docs/MCP_TOOLS.md for the full input/output schema and error_type reference for each tool.

License

MIT — see LICENSE