sbproxy-bench

Last modified: 2026-04-27

A reproducible benchmark harness for HTTP proxies and AI gateways.

The harness runs sbproxy and twelve other engines (LiteLLM, Portkey, Kong AI, Bifrost, Helicone, plus the standard reverse proxies: nginx, HAProxy, Caddy, Envoy, Traefik, KrakenD, Kong) through the same scenarios on the same hardware, then writes the raw output to results/. Methodology and per-engine setup live under docs/competitors/.

If you want to know how an AI gateway holds up under sustained load on your hardware, this is the harness to run.

Layout

sbproxy-bench/
  README.md              # this file
  Makefile               # top-level orchestration
  terraform/             # ephemeral cloud environment (single-zone)
  scripts/
    local-smoke.sh       # build the SUT, run a tiny passthrough locally
    competitor-smoke.sh  # same, against any registered competitor image
  scenarios/             # per-scenario config + client (P00-P20, A00-A11)
  harness/               # run-matrix.sh + per-scenario drivers
  images/                # Dockerfiles for the bench images
    Dockerfile.engine-rust
    Dockerfile.origin
    Dockerfile.loadgen
    competitors/         # one Dockerfile per registered competitor
  plot/                  # plot scripts and chart templates
  results/               # local JSONL output (gitignored)
  docs/
    REPRODUCIBILITY.md         # how to reproduce a run end to end
    GCP_SETUP.md               # one-time cloud bootstrap
    competitors/               # multi-engine benchmark (the public face)
      METHODOLOGY.md
      PLAN.md
      FEATURE_MATRIX_AI_GATEWAY.md
      FEATURE_MATRIX_PROXY.md
      ai-gateways/             # per-engine setup notes
      proxies/                 # per-engine setup notes

Quick reference

# Local pre-flight, no cloud spend
./scripts/local-smoke.sh

# Smoke test any registered competitor by name
./scripts/competitor-smoke.sh litellm

# Full cloud matrix (see docs/GCP_SETUP.md for one-time setup)
cd terraform
./scripts/bootstrap.sh
make init && make up
make matrix
make down

What you need to run it

• A Linux or macOS machine with Docker for the local smoke
• A cloud project (GCP today; the terraform is portable) for the full matrix
• The competitor images you want to test, built locally or pulled from your own registry

What lives elsewhere

• The proxy itself. Source for sbproxy is ../sbproxy/. This repo invokes the binary; it does not vendor it.
• The marketing summary. Plain-language results for non-engineers are at sbproxy.dev/benchmark.
• Methodology and per-engine setup. docs/competitors/.

License and contributing

Apache 2.0. See LICENSE and NOTICE. Patches welcome; read CONTRIBUTING.md first.