Zero-dependency, SDK-agnostic recorder and replayer for LLM agent. Record once, test the trajectory forever.
LLM agents are non-deterministic at the model level, but their tool call trajectory i.e. which tools they call, in what order, with what arguments is the real observable behavior.
Existing approaches' shortcomings:
- Live APIs in every test run: Slow and expensive. Worse: varying tool responses mean the agent takes different paths each run — trajectory assertions are unreliable.
- Hand-written mocks: You write the return value before seeing what the real agent produces. If the agent never calls the tool, the mock never fails. You're testing a fiction.
- Network-level recording: Records all HTTP — including every LLM request. Fixtures balloon in size and break on any SDK update, header change, or streaming format shift.
toolsnap takes a different approach: record the real trajectory once, then replay and assert on it forever.
- A trajectory recorder and assertion library for LLM agent tool calls
- A way to pin what the agent does so prompt or code changes that alter behavior surface immediately
- Most valuable when tools call external services you cannot or should not hit in CI
- A mock framework: you never invent return values by hand
- A network recorder: it operates at the Python function boundary, not HTTP
- A way to eliminate LLM calls: the LLM still runs in replay mode; only the tool backends are fixed
- An evaluation framework: it does not measure response quality or semantic correctness
- Tools that call external APIs, databases, clocks, or any service you don't want in CI
- Multi-step agents where tool call order matters
- Teams that want to catch when a prompt change silently alters agent behavior
- Any scenario where "does the agent still call the right tools?" is the key question
- Tools that are already pure functions with no external calls — just unit test them directly
- Testing LLM output quality or reasoning — use evals for that
- HTTP-level fidelity (exact headers, status codes) — use vcrpy or pytest-recording
pip install toolsnap# main.py — run once against live APIs
from toolsnap import SnapSession
def search(query: str) -> list[str]:
return real_search_api(query)
def get_weather(city: str) -> dict:
return real_weather_api(city)
# wraps both tools, saves all calls to one file in call order
with SnapSession.snap("fixtures/session.jsonl") as s:
agent = make_agent(tools=[s.wrap(search), s.wrap(get_weather)])
agent.run("what's the weather in london and find llm docs")
# fixtures/session.jsonl writtenpython main.py# test_agent.py — tool backends don't run; the LLM still runs
from toolsnap import SnapSession, contains
def test_research_agent_trajectory():
with SnapSession.replay("fixtures/session.jsonl") as s:
agent = make_agent(tools=[s.wrap(search), s.wrap(get_weather)])
agent.run("what's the weather in london and find llm docs")
# search() and get_weather() returned their recorded responses
s.assert_called("search", times=1)
s.assert_called_with("get_weather", city=contains("london"))
s.assert_call_order(["get_weather", "search"])
s.assert_no_errors()pytest test_agent.py # tool backends free, LLM still runs, trajectory deterministicBest when you have one tool and want the simplest possible setup.
# Record — search() runs for real and saves the result
@snap("fixtures/search.jsonl")
def search(query: str) -> list[str]:
return real_search_api(query)
agent = make_agent(tools=[search])
agent.run("find llm docs")
# fixtures/search.jsonl written
# Test — search() returns the recorded result; real backend never called
@replay("fixtures/search.jsonl")
def search(query: str) -> list[str]: ...
def test_agent_uses_search():
agent = make_agent(tools=[search])
result = agent.run("find llm docs") # search() returns recorded response
assert result is not NoneBest for agents that coordinate several tools. Wraps them all under one fixture and provides the full assertion API.
from toolsnap import SnapSession, contains
# Record
with SnapSession.snap("fixtures/session.jsonl") as s:
agent = make_agent(tools=[s.wrap(search), s.wrap(summarize)])
agent.run("find and summarize llm docs")
# Test
def test_multi_tool_agent():
with SnapSession.replay("fixtures/session.jsonl") as s:
agent = make_agent(tools=[s.wrap(search), s.wrap(summarize)])
agent.run("find and summarize llm docs")
s.assert_called("search", times=1)
s.assert_called_with("search", query=contains("llm"))
s.assert_call_order(["search", "summarize"])
s.assert_no_errors()Best for teams. Record and replay modes are controlled from the command line — no code changes needed.
# conftest.py
pytest_plugins = ["toolsnap.pytest_plugin"]
# test_agent.py
@pytest.mark.toolsnap_fixture("fixtures/session.jsonl")
def test_agent_trajectory(toolsnap_session):
agent = make_agent(tools=[
toolsnap_session.wrap(search),
toolsnap_session.wrap(summarize),
])
agent.run("find and summarize llm docs")
toolsnap_session.assert_called("search", times=1)
toolsnap_session.assert_call_order(["search", "summarize"])pytest tests/ # replay — tool backends free, LLM still runs
pytest tests/ --toolsnap-record # re-record after prompt/code changes
pytest tests/ --toolsnap-strict=false # allow unexpected calls to fall throughAll assertion methods accept predicate objects for structural matching:
| Predicate | Matches when |
|---|---|
contains("llm") |
value contains the substring |
matches(r"\d{4}-\d{2}") |
value matches the regex |
any_of("london", "paris") |
value is one of the given options |
gt(0) / lt(100) |
value is greater / less than threshold |
s.assert_called_with("search", query=contains("london"))
s.assert_called_with("embed", n_tokens=lt(512))All assertion methods on SnapSession:
| Method | What it checks |
|---|---|
assert_called(name) |
called at least once |
assert_called(name, times=N) |
called exactly N times |
assert_not_called(name) |
never called |
assert_called_with(name, **kwargs) |
at least one call matches the kwargs / predicates |
assert_called_with(name, call=N, **kwargs) |
the Nth call matches |
assert_call_order([...]) |
names appear in this order across the timeline |
assert_no_errors() |
no tool call raised an exception |
assert_raised(name, "ErrorType") |
at least one call raised this exception type |
After a prompt change, toolsnap diff shows exactly what shifted in the agent's trajectory:
toolsnap diff fixtures/before.jsonl fixtures/after.jsonl
# Diff: fixtures/before.jsonl → fixtures/after.jsonl
# ────────────────────────────────────────────────────
# search call 0 args unchanged result CHANGED (3 items → 2 items)
# + summarize call 0 ADDED
toolsnap show fixtures/session.jsonl # pretty-print all records
toolsnap stats fixtures/session.jsonl # call counts, avg/p95 latency, errors
toolsnap validate fixtures/session.jsonl
toolsnap list