RiddleRun

Docker-first AI end-to-end browser testing.

RiddleRun is a CLI and optional self-hosted web app for teams that want automated agentic end2end test. Describe a user journey in JSON, run it in Docker, and let a Playwright/browser-use agent execute your test case for you!

Examples

The example runs below come from the example tests in this repository and were recorded from the self-hosted RiddleRun web app.

Successful run

Failed run

The repo ships with Wikipedia examples you can run without hosting your own app first:

Test ID	Steps	Purpose
`wikipedia-python-language-es`	3	Switch the Python article from English to Spanish.
`wikipedia-search-toc-references`	7	Search Wikipedia, use the table of contents, and verify references.
`wikipedia-cross-article-links`	8	Follow in-article links from Python to related topics.
`wikipedia-python-language-fail`	3	Intentional failure for checking error reporting.

Why End-to-End Tests?

End-to-end testing is where product quality often becomes expensive:

Vibecoding. Your codebase changes faster than you can test.
Manual E2E testing does not scale. Every release needs core flows checked again, and the cost grows quickly as your product, supported browsers, and user journeys expand.
Traditional Playwright and Selenium suites can be brittle and slow to maintain. Selector-heavy tests break when UI structure changes, while modern AI-assisted development increases the speed at which applications change and therefore the speed at which teams need feedback.

Quick Start

1. Clone the examples

The CLI is published as a Docker image. Clone this repo for the example project and test files, or mount your own JSON files once you are ready.

git clone https://github.com/raeudigerRaeffi/riddlerun.git
cd riddlerun

2. Run with the published Docker image

Set an API key in your shell, pull the image, and run one example test:

$env:OPENAI_API_KEY = "sk-..."

docker pull riddlerun/riddlerun:latest

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e OPENAI_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --test-id wikipedia-python-language-es

Runs write JSON artifacts under /data/artifacts/runs by default. Browser-use logs are hidden by default so the CLI can show a live results table; add --verbose when debugging.

3. Or build the CLI image from source

Use this option when you want to develop RiddleRun itself or avoid pulling the published image.

cd riddlerun
docker build -t riddlerun:local .

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e OPENAI_API_KEY `
  riddlerun:local `
  run /workspace/examples/tests --test-id wikipedia-python-language-es

4. Create your own project and tests

RiddleRun needs one project JSON file and one or more test case JSON files.

A project describes the application under test:

{
  "id": "local-demo",
  "name": "Local demo app",
  "baseUrl": "http://host.docker.internal:3000",
  "auth": {
    "username": "demo@example.com",
    "email": "demo@example.com",
    "password": "change-me"
  },
  "browser": {
    "mode": "local",
    "headless": true,
    "timeoutMs": 120000
  }
}

A test case belongs to a project and contains ordered natural-language steps:

{
  "id": "login-smoke",
  "name": "Login smoke test",
  "projectId": "local-demo",
  "description": "Verify that a user can log in.",
  "tags": ["smoke", "auth"],
  "steps": [
    {
      "description": "Open the project base URL."
    },
    {
      "description": "Log in using the saved project credentials.",
      "expected_result": "The authenticated dashboard is visible."
    }
  ]
}

Field notes:

auth is optional if testing requires an account
baseUrl is the starting URL for the app under test. From Docker on macOS and Windows, host.docker.internal points back to your host machine.
projectId in each test must match the project id.
description tells the browser agent what to do.
expected_result is optional and tells the agent what outcome to validate.
tags are optional and make it easier to run subsets of tests.

See docs/test-spec.md, schemas/project.schema.json, and schemas/testcase.schema.json for the full JSON contract.

Run your own files by mounting their directory and pointing RIDDLE_RUN_PROJECT at your project file:

docker run --rm `
  -v "${PWD}/my-riddlerun-tests:/workspace/tests:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/tests/projects/local-demo.json `
  -e OPENAI_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/tests/cases --tag smoke

Run Options

Each test case is executed as one browser-use agent task. The options below control which tests are selected, how long the agent can work, which LLM is used, and what gets saved afterward.

Common commands

Run every Wikipedia example:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e OPENAI_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --tag wikipedia

Run the intentional failure example:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e OPENAI_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --test-id wikipedia-python-language-fail

Validate JSON files without opening a browser:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  riddlerun/riddlerun:latest `
  validate /workspace/examples/tests

Record a run video:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e OPENAI_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --test-id wikipedia-python-language-es --video-dir /data/artifacts/videos

Run a longer flow with a higher agent step budget:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e OPENAI_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests `
  --test-id wikipedia-cross-article-links `
  --max-steps 45 `
  --verbose

CLI flags

Flag	Default	Description
`--project`, `-p`	`RIDDLE_RUN_PROJECT`	Project metadata JSON file. Required unless the env var is set.
`--test-id`	all tests	Run only the given test IDs. Repeat the flag to select multiple tests.
`--tag`	all tests	Run only tests that include one of the given tags. Repeat to match any listed tag.
`--max-steps`	`29`	Maximum browser-use agent actions allowed per test. Increase this for longer multi-step journeys.
`--dry-run`	off	Validate the selected tests and print skipped results without opening a browser.
`--artifact-dir`	`/data/artifacts` in Docker	Directory where JSON run result files are written.
`--video-dir`	off	Enable screen recording and save videos under this directory.
`--verbose`	off	Show browser-use logs instead of only the live results table.
`--llm-provider`	`openai`	LLM provider passed to browser-use.
`--llm-model`	`o4-mini`	Model name for the selected provider. Important the test works best when Vision Language Models are used
`--llm-api-key`	provider env var	API key override. Prefer provider-specific env vars in Docker.
`--llm-base-url`	provider default	Custom base URL for local or compatible providers such as Ollama.

Environment variables

All settings can also be set with the RIDDLE_RUN_ prefix. See .env.example for a starting point.

Variable	Default	Description
`RIDDLE_RUN_PROJECT`	—	Default project JSON path for `run`.
`RIDDLE_RUN_MAX_STEPS`	`29`	Default agent step cap per test.
`RIDDLE_RUN_ARTIFACT_DIR`	`artifacts`	Default directory for run JSON artifacts.
`RIDDLE_RUN_VIDEO_DIR`	off	When set, enables video recording to this directory.
`RIDDLE_RUN_VERBOSE`	`false`	Show browser-use logs by default.
`RIDDLE_RUN_LLM_PROVIDER`	`openai`	Default LLM provider.
`RIDDLE_RUN_LLM_MODEL`	`o4-mini`	Default LLM model.
`RIDDLE_RUN_LLM_API_KEY`	—	Generic API key override for any provider.
`RIDDLE_RUN_LLM_BASE_URL`	—	Default custom provider base URL.

Project browser settings

Browser behavior for a project is defined in the project JSON browser block:

Field	Default	Description
`mode`	`local`	Browser execution mode. Remote browser connections are not supported yet.
`headless`	`true`	Whether Chromium runs headless inside the Docker container.
`timeoutMs`	`120000`	Browser session timeout in milliseconds.

RiddleRun launches Chromium inside the Docker container for each run. The browser is not displayed on your host machine; use --video-dir when you need a visual recording.

LLM Providers and API Keys

RiddleRun uses browser-use with a configurable LLM provider. OpenAI is the default.

Set keys in your shell

PowerShell:

$env:OPENAI_API_KEY = "sk-..."

macOS and Linux:

export OPENAI_API_KEY="sk-..."

Then pass the variable into Docker with -e OPENAI_API_KEY. Provider-specific keys such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, MISTRAL_API_KEY, and GROQ_API_KEY are read automatically when no CLI flag or RIDDLE_RUN_LLM_API_KEY override is set.

For Docker Compose development, put keys in .env:

RIDDLE_RUN_LLM_PROVIDER=openai
RIDDLE_RUN_LLM_MODEL=o4-mini
OPENAI_API_KEY=sk-...

Provider examples

Anthropic:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e ANTHROPIC_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --test-id wikipedia-python-language-es --llm-provider anthropic --llm-model claude-sonnet-4-5

Local Ollama:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --test-id wikipedia-python-language-es --llm-provider ollama --llm-model llama3.1 --llm-base-url http://host.docker.internal:11434

LiteLLM provider, for example Groq:

docker run --rm `
  -v "${PWD}/examples:/workspace/examples:ro" `
  -v "${PWD}/artifacts:/data/artifacts" `
  -e RIDDLE_RUN_PROJECT=/workspace/examples/projects/wikipedia.json `
  -e GROQ_API_KEY `
  riddlerun/riddlerun:latest `
  run /workspace/examples/tests --test-id wikipedia-python-language-es --llm-provider groq --llm-model groq/llama-3.3-70b-versatile

Supported providers include openai, anthropic, google, mistral, ollama, browser-use, vercel, and LiteLLM aliases such as groq, openrouter, together, azure, bedrock, lmstudio, and openai-compatible.

Self-Hosted Web App

The web UI and API are built from this repository only; they are not published as Docker Hub images. Run the full stack with Docker Compose:

docker compose up --build

Service	URL
Frontend	http://localhost:3000
API docs	http://localhost:8000/docs
Postgres	localhost:5432

On first visit, create the local admin account at /setup. The web UI lets you manage projects, author test cases, store provider keys, and inspect run history.

Run CLI commands through Compose:

docker compose --profile cli run --rm cli validate /workspace/examples/tests
docker compose --profile cli run --rm cli run /workspace/examples/tests

Run package tests:

docker compose --profile cli run --rm -v ./tests:/app/tests cli sh -lc 'python -m pip install -e ".[dev]" && python -m pytest'

Repository Layout

.
├── riddlerun/core/       # Shared Playwright / browser-use execution engine
├── backend/              # FastAPI API and persistence layer
├── cli/                  # Docker-first command-line entrypoint
├── docs/                 # Project and JSON spec documentation
├── examples/             # Example projects and test case files
├── frontend/             # Next.js TypeScript web UI
├── media/                # Logo, banner, and example run recordings
├── schemas/              # JSON schemas for projects and test cases
├── docker-compose.yml    # Docker-only development and self-hosted stack
├── Dockerfile            # CLI image
├── Dockerfile.api        # API image
└── pyproject.toml        # Python package metadata for container builds

Production Deployment

A production-oriented Compose stack lives in docker-compose.prod.yml. Copy .env.production.example to .env.production, replace every placeholder secret, set the public frontend and API URLs, then start the stack:

cp .env.production.example .env.production
docker compose --env-file .env.production -f docker-compose.prod.yml up --build -d

When RIDDLE_RUN_ENV=production, the API refuses to start if RIDDLE_RUN_APP_SECRET is still a placeholder or shorter than 32 characters. The production stack uses named volumes for Postgres data and run artifacts; put TLS and public host routing in front of the exposed API and frontend ports with your reverse proxy of choice.

License

RiddleRun is licensed under AGPL-3.0-only. A commercial license may be offered later for organizations that cannot use AGPL software.

Contribution

Contributions are welcome. We are especially happy to accept improvements that expand test coverage, add realistic example flows, or make RiddleRun easier to run in CI and self-hosted environments. For questions: raffaelschoERASECHARSCAPITALIZED@gmail.com

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

RiddleRun

Examples

Successful run

Failed run

Why End-to-End Tests?

Quick Start

1. Clone the examples

2. Run with the published Docker image

3. Or build the CLI image from source

4. Create your own project and tests

Run Options

Common commands

CLI flags

Environment variables

Project browser settings

LLM Providers and API Keys

Set keys in your shell

Provider examples

Self-Hosted Web App

Repository Layout

Production Deployment

License

Contribution

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.github/workflows		.github/workflows
backend		backend
cli		cli
docs		docs
examples		examples
frontend		frontend
media		media
migrations		migrations
riddlerun		riddlerun
schemas		schemas
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.env.production.example		.env.production.example
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.api		Dockerfile.api
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
alembic.ini		alembic.ini
docker-compose.prod.yml		docker-compose.prod.yml
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

RiddleRun

Examples

Successful run

Failed run

Why End-to-End Tests?

Quick Start

1. Clone the examples

2. Run with the published Docker image

3. Or build the CLI image from source

4. Create your own project and tests

Run Options

Common commands

CLI flags

Environment variables

Project browser settings

LLM Providers and API Keys

Set keys in your shell

Provider examples

Self-Hosted Web App

Repository Layout

Production Deployment

License

Contribution

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages