Ethereum JSON-RPC Benchmarking Suite

A comprehensive benchmarking framework for Ethereum JSON-RPC clients, designed to evaluate and compare the performance and behavior of different client implementations like Geth and Nethermind.

Overview

This project runs predefined RPC tests derived from the official Ethereum Execution APIs spec, generates standardized performance metrics, checks for response consistency, and provides both historic tracking and real-time analysis through a modern web dashboard.

Features

Performance Benchmarking: Benchmark and compare Ethereum clients under realistic load using k6
Historic Tracking: Store and analyze performance trends over time with PostgreSQL + Grafana integration
Real-time Dashboard: Modern React UI for viewing results, trends, and comparisons
Response Validation: Validate RPC response compatibility with ethereum/execution-apis
Schema Validation: Check responses against official Ethereum JSON-RPC specifications
Baseline Management: Set performance baselines and detect regressions automatically
Multiple Output Formats: Generate HTML reports, CSV exports, and JSON data
WebSocket Updates: Real-time updates for live monitoring
Grafana Integration: Pre-built dashboards for time-series analysis and alerting

Project Structure

json-bench/
│
├── config/                       # YAML test configurations
│   ├── clients/                  # RPC client registries
│   │   └── clients.yaml
│   ├── benchmark/                # Benchmark workload definitions
│   │   ├── mixed.yaml
│   │   └── read-heavy.yaml
│   ├── compare/                  # `runner compare` request lists
│   │   ├── defaults.yaml
│   │   └── example.yaml
│   ├── compare-openrpc/          # OpenRPC-driven comparison inputs
│   │   └── param_variations.yaml
│   └── storage/                  # Historic storage configuration
│       ├── storage-example.yaml
│       └── storage-docker.yaml
│
├── runner/                   # Go benchmark runner with historic tracking
│   ├── main.go              # Thin entry point - delegates to cmd.Execute()
│   ├── cmd/                 # Cobra subcommands (benchmark, api, historic,
│   │                        #                    compare, compare-openrpc)
│   ├── api/                 # HTTP API server and WebSocket support
│   ├── storage/             # PostgreSQL integration
│   ├── analysis/            # Trend analysis and regression detection
│   └── generator/           # K6 script generation and HTML reports
│
├── dashboard/               # React dashboard for historic analysis
│   ├── src/
│   │   ├── pages/          # Dashboard pages (trends, comparisons, baselines)
│   │   ├── components/     # Reusable UI components
│   │   └── api/            # API client for backend integration
│   └── dist/               # Built dashboard files
│
├── metrics/                 # Postgres + Prometheus + Grafana configuration
│   ├── grafana-provisioning/
│   └── dashboards/         # Pre-built Grafana dashboards
│
└── cmd/                     # Legacy debug helpers (not built by default)

Getting Started

Prerequisites

Docker and Docker Compose (for client nodes and infrastructure)
Go 1.20+ (for the benchmark runner)
Node.js 18+ (for the React dashboard)
k6 (for load testing - install from https://k6.io/)
PostgreSQL (for historic tracking - included in Docker Compose)

Quick Start

Clone the repository:

git clone <repository-url>
cd json-bench

Start the infrastructure (PostgreSQL, Prometheus and Grafana):

# Start Grafana, PostgreSQL and Prometheus
docker compose up -d grafana postgres prometheus

# You can also start the runner API and the dashboard
docker compose up -d runner dashboard

# Or just start everything
docker compose up -d

Set up your client nodes:

# Configure your own endpoints in the config files
nano config/clients/clients.yaml

Run a benchmark:

# Basic benchmark (no historic tracking)
go run ./runner benchmark --config ./config/benchmark/mixed.yaml --clients ./config/clients/clients.yaml
# With historic tracking (requires PostgreSQL)
go run ./runner benchmark --config ./config/benchmark/mixed.yaml --clients ./config/clients/clients.yaml --historic --storage-config ./config/storage/storage-example.yaml

# View results
open outputs/report.html

NOTE: storage-example.yaml works out of the box with the docker containers deployed in the compose file.

Access the services:
- PostgreSQL: localhost:5432 (postgres/postgres)
- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000 (admin/admin)
- RunnerAPI: http://localhost:8082 (if started)
- Dashboard: http://localhost:8080 (if started)

Usage

The runner binary exposes its functionality through subcommands. Running runner with no subcommand prints usage and exits with status 2.

runner benchmark        Run a benchmark (k6 -> Prometheus -> reports)
runner compare          One-shot cross-client JSON-RPC response comparison
runner compare-openrpc  Cross-client comparison driven by an OpenRPC specification
runner api              Start the HTTP API server
runner historic         Generate a historic-analysis report from PostgreSQL

Global flags accepted by every subcommand:

--output (default outputs/) - where artefacts are written.
--log-level - debug, info, warn, or error. Falls back to the LOG_LEVEL environment variable, then info.

Basic Benchmarking

# Run a mixed workload benchmark (no Prometheus; metrics come from k6's summary.json)
go run ./runner benchmark --config ./config/benchmark/mixed.yaml --clients ./config/clients/clients.yaml

# Run a read-heavy benchmark
go run ./runner benchmark --config ./config/benchmark/read-heavy.yaml --clients ./config/clients/clients.yaml

# Opt into Prometheus remote-write by passing an endpoint
go run ./runner --output ./custom-results benchmark \
  --config ./config/benchmark/mixed.yaml \
  --clients ./config/clients/clients.yaml \
  --prometheus http://prometheus:9090

# Override the remote-write path (defaults to /api/v1/write)
go run ./runner benchmark \
  --config ./config/benchmark/mixed.yaml \
  --clients ./config/clients/clients.yaml \
  --prometheus http://prometheus:9090 \
  --prometheus-rw-path /custom/remote-write/path

# Generate an HTML report alongside the JSON and CSV exports (off by default)
go run ./runner benchmark \
  --config ./config/benchmark/mixed.yaml \
  --clients ./config/clients/clients.yaml \
  --html-report

--prometheus is optional and disabled by default. When it is omitted (or empty), k6 remote-write is not enabled and per-client metrics are collected from k6's summary.json instead. Passing an endpoint opts into remote-write and post-run PromQL queries.

benchmark always writes outputs/results.json and outputs/results.csv. The HTML report at outputs/report.html is opt-in via --html-report. compare and compare-openrpc always produce their HTML report.

Historic Tracking & Analysis

Enable historic tracking to store results in PostgreSQL and analyze trends over time:

# Persist this benchmark run to PostgreSQL
go run ./runner benchmark \
  --config ./config/benchmark/mixed.yaml \
  --clients ./config/clients/clients.yaml \
  --historic \
  --storage-config ./config/storage/storage-example.yaml

# Generate a historic-analysis report (no new benchmark)
go run ./runner historic \
  --config ./config/benchmark/mixed.yaml \
  --storage-config ./config/storage/storage-example.yaml

API Server for Real-time Access

Start the HTTP API server for real-time data access and WebSocket updates:

# Start the API server with historic storage
go run ./runner api \
  --storage-config ./config/storage/storage-example.yaml \
  --api-addr :8081

# API will be available at http://localhost:8081

One-shot Cross-Client Comparison

runner compare runs a one-shot JSON-RPC response comparison across a set of clients defined in clients.yaml. It reads the request list from a YAML config and is filesystem-only: results are not written to the historic-runs database.

The compare config is a curated list of {method, params} calls grouped by method. Each entry may carry an optional id (must be [a-zA-Z0-9_-]+ and unique within its method); when omitted, entries are numbered variant1, variant2, ...

# config/compare/example.yaml
name: "Ethereum API comparison"
description: "Cross-client sanity checks"
calls:
  eth_blockNumber:
    - params: []
  eth_getBalance:
    - id: "vitalik-latest"
      params: ["0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045", "latest"]
    - id: "vitalik-at-block-4096"
      params: ["0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045", "0x1000"]
  eth_call:
    - id: "weth-balanceOf"
      params:
        - to: "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
          data: "0x70a08231000000000000000000000000000000000000000000000000000000000000000a"
        - "latest"

go run ./runner compare \
  --config ./config/compare/example.yaml \
  --clients ./config/clients/clients.yaml \
  --client-refs geth,nethermind \
  --output ./comparison-results

These artefacts are always produced:

<output>/comparison-results.json
<output>/comparison-report.html
<output>/comparison-provenance.json — the effective config (client refs, active rules, block override, output/retry settings, sample counts) so a report is self-describing and reproducible.

A config/compare/defaults.yaml ships with the repo and reproduces the old baseline of eight common methods (eth_blockNumber, eth_getBalance, eth_call, eth_getBlockByNumber, eth_getTransactionCount, eth_chainId, eth_gasPrice, net_version) with the parameters the CLI previously used implicitly.

Declaring expected differences (`comparison:` block)

Cross-client (and cross-version) runs surface many expected differences — a serialization change like totalDifficulty, sub-percent eth_estimateGas drift, or a namespace that is disabled on one node. Declare these in an optional comparison: block so they stop flooding the report:

comparison:
  # Rewrite latest/pending tags to a static block and append a block arg to
  # calls that omit one, so archive nodes at different heads are comparable.
  block_override: "0x1406f40"
  rules:
    # Treat two hex quantities as equal within abs units OR rel fraction.
    # (eth_estimateGas already gets a built-in 10% tolerance by default.)
    - method: eth_estimateGas
      path: result
      kind: numeric_tolerance
      abs: 32
      rel: 0.10
    # Drop a path entirely; `[*]` matches any array index.
    - path: result.totalDifficulty
      kind: ignore
    - method: eth_getBlockByNumber
      path: result.transactions[*].v
      kind: ignore
    # Compare only the error code, not the (benign) message wording.
    - method: eth_call
      kind: error_code_only    # also: error_presence_only

Rule kinds: ignore, numeric_tolerance (abs/rel), error_code_only, error_presence_only. A rule with no method applies to all methods; no path applies to the whole response value. Everything in the block is optional and defaults reproduce the prior behavior.

The same block can live in a standalone file passed with --rules, which merges over the config's own rules and works in corpus mode too:

go run ./runner compare \
  --from-jsonl ./rpc-calls --sample 400 \
  --rules ./config/compare/archive-rules.yaml \
  --block-override 0x1406f40 \
  --clients ./config/clients/clients.yaml --client-refs old,new \
  --diff-only --fail-on-diff --concurrency 4

Building a config from a corpus (`--from-jsonl`)

Instead of --config, point --from-jsonl <dir> at a corpus directory. It recurses and reads both line-delimited *.jsonl files and *.json files holding a JSON array of {method, params} objects. --sample N keeps at most N calls per method (deterministic; control the seed with --sample-seed). Head-dependent and unstorable methods (eth_getProof, eth_gasPrice, eth_syncing, eth_blockNumber, eth_maxPriorityFeePerGas, the debug_ namespace) are excluded; eth_feeHistory is kept only when a block override pins its newestBlock.

Smaller reports and CI gating

--diff-only excludes identical calls. To keep the report small it also caps response bodies (4096 bytes) unless you pass --keep-response-bodies to retain them or --omit-matching-responses to drop them entirely. --max-response-bytes N sets an explicit cap.
The run prints a category summary: identical / differ (real) / differ (env/expected) / transport-error / schema-error / skipped, plus per-class environment/capability error counts.
--fail-on-diff exits non-zero on real differences only (environment/ capability differences are excluded by default); add --fail-on-env-diff for strict mode that also fails on those.
--skip-above-head queries each client's head and skips calls pinned to a higher block, so a less-synced node does not produce false differences.

Robustness against throttling nodes

Transport errors and 5xx responses are retried with exponential backoff. --max-retries sets the attempt budget (falls back to a client's max_retries in clients.yaml, or 5 if unset) and --retry-base-delay the base backoff. A call that still fails is recorded per-call and the run continues, so a single dead endpoint never discards an otherwise good run. Keep --concurrency low (≤4) against nodes that throttle connection bursts.

OpenRPC-Driven Comparison

runner compare-openrpc loads the method set from an OpenRPC specification and runs the same cross-client comparison as compare, optionally expanding individual methods into per-parameter variations defined in a YAML file. Like compare, it is filesystem-only and does not write to the historic-runs database.

go run ./runner compare-openrpc \
  --spec ./openrpc.json \
  --variations ./config/compare-openrpc/param_variations.yaml \
  --clients ./config/clients/clients.yaml \
  --client-refs geth,nethermind \
  --filter eth_blockNumber,eth_getBlockByNumber \
  --output ./openrpc-results

Useful debug aids:

--filter <m1,m2,...> - restrict the comparison to a whitelist of method names. Matches the base method name, so --filter eth_getBlockByNumber keeps every eth_getBlockByNumber_variantN generated from the variations file.
--curl - log a curl-equivalent command for every JSON-RPC request the comparator makes, useful for reproducing a single call against a client by hand.

Both comparison-results.json and comparison-report.html are written to --output.

Migrating from the pre-refactor CLI

The flag-modal entry point (runner -api, runner -historic-mode, single-dash long flags) has been replaced with explicit Cobra subcommands. The old invocations will fail at flag-parse time with a clear error. The mapping is:

Old	New
`runner -config ... -prometheus-rw ...`	`runner benchmark --config ... --prometheus ...`
`runner -api -storage-config ... -api-addr ...`	`runner api --storage-config ... --api-addr ...`
`runner -historic-mode -storage-config ... -config ...`	`runner historic --storage-config ... --config ...`
`runner -historic -storage-config ...` (with `-config ...`)	`runner benchmark --historic --storage-config ...`
`-config`, `-clients`, `-output`, `-prometheus-rw`, ... (single dash)	`--config`, `--clients`, `--output`, `--prometheus`, ... (double dash)
`--prometheus-rw http://host:9090/api/v1/write` (full URL)	`--prometheus http://host:9090` (base only). Override the appended path with `--prometheus-rw-path /api/v1/write` if your deployment is non-standard.

Additional behaviour changes worth noting:

--prometheus is now optional and disabled by default. Omit it to run without remote-write (metrics are read from k6's summary.json); pass an endpoint to opt into Prometheus.
The benchmark report.html is opt-in via --html-report. JSON and CSV exports remain on by default.
The legacy endpoints + frequency YAML schema is no longer accepted. Configs using it must be migrated by hand to the calls: schema (see config/benchmark/mixed.yaml for the canonical shape). No migrator is provided.
The jsonrpc-benchmark.json Grafana dashboard has been removed. Its Prometheus queries (method_calls_*, rpc_errors_*, rpc_calls_*) reference custom counters from the pre-refactor k6 template that are no longer emitted. The other three dashboards (k6-dashboard.json, jsonrpc-benchmark-enhanced.json, baseline-comparison.json) remain in place.

Available API endpoints:

GET /api/runs - List historic benchmark runs
GET /api/runs/:id - Get specific run details
GET /api/trends - Get performance trend data
GET /api/baselines - List performance baselines
GET /api/compare?run1=:id1&run2=:id2 - Compare two runs
POST /api/runs/:id/baseline - Set run as baseline
WS /api/ws - WebSocket for real-time updates

React Dashboard

The modern React dashboard provides an intuitive interface for analyzing benchmark results and trends:

# Install dashboard dependencies
cd dashboard
npm install

# Start development server
npm run dev
# Dashboard available at http://localhost:3000

# Build for production
npm run build
npm run preview

Dashboard Features:

Dashboard Page: Overview of recent runs and performance trends
Run Details: Detailed analysis of individual benchmark runs
Comparison View: Side-by-side comparison of multiple runs
Baseline Management: Set and manage performance baselines
Trend Analysis: Interactive charts showing performance over time
Regression Alerts: Automatic detection of performance regressions

Grafana Integration

For advanced time-series analysis and alerting, you can use Grafana:

Setup Grafana locally or use Docker:
```
docker compose up -d grafana
```
Open Grafana: http://localhost:3000 (admin/admin)
Provisioned data sources:
- Prometheus:
  - URL: http://localhost:9090 (or http://prometheus:9090 if using Docker network)
  - Access: Server (default)
- PostgreSQL (for historic data):
  - Host: localhost:5432 (or postgres:5432 if using Docker network)
  - Database: jsonrpc_bench
  - User: postgres
  - Password: postgres
  - SSL Mode: disable
Provisioned dashboards from metrics/dashboards/
- K6 Performance
- Client performance comparison
- Method-specific latency trends
- Error rate monitoring
- System resource usage
- Historic trend analysis
Set up alerting for performance regressions and system issues

Scraping node-side metrics

When Prometheus is enabled, this tool only ships k6 client-side metrics (k6_http_req_*) to it. It does not scrape the Geth/Nethermind/etc. clients under test — bring your own observability for server-side metrics. To add them, point your own Prometheus at the node's metrics endpoint (each EL client publishes one):

# prometheus.yml — example for a local Geth instance
scrape_configs:
  - job_name: 'geth'
    metrics_path: /debug/metrics/prometheus
    static_configs:
      - targets: ['localhost:6060']

If you compose your own Prometheus alongside this stack, add the scrape job to that config; the bundled metrics/prometheus.yml only handles k6's remote-write target.

Storage Configuration

Configure PostgreSQL storage for historic tracking. Choose the appropriate configuration file based on your setup:

Local Development (outside Docker):

# Use storage-example.yaml for local PostgreSQL connection
go run ./runner benchmark --config ./config/benchmark/mixed.yaml --historic --storage-config ./config/storage/storage-example.yaml

Docker Environment:

# Use storage-docker.yaml when running inside Docker containers
# (This config uses 'postgres' hostname which only exists in Docker network)
docker run ... api --storage-config ./config/storage/storage-docker.yaml

Configuration Examples:

# config/storage/storage-example.yaml (for local development)
historic_path: "./historic"
enable_historic: true

postgresql:
  host: "localhost"          # Use localhost when running outside Docker
  port: 5432
  database: "jsonrpc_bench"
  username: "postgres"
  password: "postgres"
  ssl_mode: "disable"
  
  grafana:
    metrics_table: "benchmark_metrics"
    runs_table: "benchmark_runs"
    retention_policy:
      metrics_retention: "30d"
      aggregated_retention: "90d"

# config/storage/storage-docker.yaml (for Docker environment)
historic_path: "/app/historic"
enable_historic: true

postgresql:
  host: "postgres"           # Use service name when running in Docker
  port: 5432
  database: "jsonrpc_bench"
  username: "postgres"
  password: "postgres"
  ssl_mode: "disable"

Advanced Features

Baseline Management

Set performance baselines to detect regressions:

# Set a run as baseline via API
curl -X POST http://localhost:8080/api/runs/20250103-120000-abc123/baseline \
  -H "Content-Type: application/json" \
  -d '{"name": "Production Baseline", "description": "Post-optimization baseline"}'

# Compare current run against baseline
curl "http://localhost:8080/api/compare?run1=baseline&run2=20250103-130000-def456"

Custom Test Configurations

Create custom benchmark configurations:

# Custom benchmark configuration
test_name: "Custom Load Test"
description: "High-load test for production validation"

clients:
  - name: "geth"
    url: "http://localhost:8545"
  - name: "nethermind"
    url: "http://localhost:8546"

rps: 500
duration: "5m"

calls:
  - name: "my_method_1"
    method: "eth_blockNumber"
    weight: 20
  - name: "my_method_2"
    method: "eth_getBalance"
    params: ["0x742d35Cc641C0532a7D4567bb19f68cE3FdD72cD", "latest"]
    weight: 40

Parameter Variations

Test methods with different parameter sets:

# config/compare-openrpc/param_variations.yaml
eth_call:
  - [{"to": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"}, "latest"]
  - [{"to": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"}, "pending"]

eth_getBalance:
  - ["0x742d35Cc641C0532a7D4567bb19f68cE3FdD72cD", "latest"]
  - ["0x742d35Cc641C0532a7D4567bb19f68cE3FdD72cD", "pending"]

Docker Deployment

Deploy the entire stack using Docker:

# Full stack deployment
docker-compose up -d

Environment Variables

Configure the application using environment variables:

# Create a copy of the .env.example file
cp .env.example .env

# Edit environment configuration
nano .env

Monitoring and Alerting

Grafana Alerts

Set up alerts for performance regressions:

Configure notification channels (Slack, Discord, email)
Set alert rules for:
- Latency increases > 20%
- Error rate > 1%
- Throughput decreases > 15%
Enable alert evaluation in Grafana settings

Webhook Integration

Configure webhooks for CI/CD integration:

# Environment variables for webhook notifications
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."
export GITHUB_WEBHOOK_URL="https://api.github.com/repos/owner/repo/dispatches"

Output Formats

The benchmark runner generates multiple output formats:

HTML Reports: Interactive reports with charts and metrics
JSON Data: Machine-readable benchmark results
CSV Exports: For spreadsheet analysis
Grafana Dashboards: Time-series visualizations
Historic Analysis: Trend reports and regression detection

Contributing

Follow Go conventions for backend code
Use TypeScript for frontend development
Add tests for new features
Update documentation for API changes
Ensure backward compatibility for configuration files

License

This project is licensed under the MIT License - see the LICENSE file for details.

Name		Name	Last commit message	Last commit date
Latest commit History 88 Commits
.github		.github
config		config
dashboard		dashboard
metrics		metrics
rpc-calls		rpc-calls
runner		runner
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
.trivyignore		.trivyignore
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
USAGE.md		USAGE.md
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum
openrpc.json		openrpc.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation