Add RPC ingestion load test driven by synthetic apply-load ledger bundles by cjonas9 · Pull Request #741 · stellar/stellar-rpc

cjonas9 · 2026-05-15T22:40:07Z

What

This is a PR implementing a repeatable CI ingestion load test on a full database of 7 days of ledgers. The approximate design is here:

This GHA workflow for this test, currently, is triggered on pushes to this branch (apply-load), but will later be modified to trigger on any release or on PR comments stating "run load test".

The workflow benchmarks RPC ingestion end-to-end on an ephemeral c5.2xlarge: it launches the box, pulls a mainnet-scale golden DB (~307GB, 1-week retention window), a BUILD_TESTS stellar-core, and three apply-load ledger bundles from S3 (sha-verified). After the box downloads and decompresses this data, its gp3 volume is throttled to 125 MiB/s, ingests the bundles, and posts a per-profile results table to the run summary / PR.

Main Pieces:

integrationtest/ingest_loadtest_test.go::TestIngestSyntheticLedgers: byte-concatenates N bundles into one continuous stream (the backend rebases ledger seqs per ledger, so per-bundle seq resets are harmless), ingests onto the golden DB with retention trimming live, verifies exact classic/soroban op counts via parallel getTransactions walkers, and reports per-profile wall-clock/ledgers-sec/ms-ledger/latency quantiles.
loadtest/testdata/apply-load-v27-*-cfg: config files specifying three O3 target tx profiles, 1,000 ledgers each: sac (1,000 soroban TPL), oz (900), soroswap (250). All generate these + 1,000 classic payments/ledger to create ledger bundles (for local usage or S3) offline by stellar-core apply-load.
.github/workflows/load-test.yml: push-triggered orchestrator. OIDC-assumes into AWS, launches an ephemeral c5.2xlarge (Ubuntu 22.04, 500GB gp3) with the runner script as user-data (shipped verbatim, TARGET_SHA/RUN_ID passed via a two-line env preamble), waits for SSM registration, delegates polling to the script, writes the results table to the step summary (and PR comment when one exists), fails the job on a fail verdict or timeout, and always terminates the instance.
run-load-test.sh: both halves of the run in one self-contained script, coordinated by a /tmp marker protocol.
- instance (user-data on the box): installs the toolchain, streams the golden DB + BUILD_TESTS core + all bundles from S3, builds the repo at the target SHA, throttles root volume MiB/s, runs the test, and writes results.md plus an ok/fail verdict.
- orchestrate (on the GHA runner): polls the box over SSM, drives the gp3 downshift handshake (500 -> 125 MiB/s after downloads complete, so fetches are fast but the benchmark runs on throttled I/O), and relays verdict + results as step outputs.

Why

CI testing of RPC ingestion performance; benchmarking. This also serves as an automated regression testing framework, though future work should expand this to report some metric that allows one to compare a run's results to historical results.

Known limitations

This is purely intended as a test of RPC's ingestion pipeline and seeks to see how it handles load in isolation (i.e. without captive core running). Future work should also seek to automatically refresh the S3 DB + ledger bundles on some pre-determined cadence.

… real DB test

github-actions · 2026-05-15T22:42:08Z

⏳ Load test launching on i-063ed1e3a29f001e3 (commit 241bdf833edbfc18d3c312a7109168d171324aa1).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25944865928
Posting results when the run finishes (~15 min).

github-actions · 2026-06-22T22:59:10Z

📈 Ingest load test — `1f974a4`

Profile	Ledgers	Wall-clock	Ledgers/sec	ms/ledger	p50 / p95 / p99 ms
apply-load-v27-oz	1000	1234.224s	0.81	1235.46	1150 / 1674.999 / 1925.001
apply-load-v27-sac	1000	1138.025s	0.88	1138.02	1149.999 / 1225 / 1275
apply-load-v27-soroswap	1000	829.299s	1.21	829.30	849.994 / 900.001 / 975

Metric	Value
Ledgers replayed	3000
Initial DB ledger count	120960
Overall throughput	0.94 ledgers/sec
Overall ingest wall-clock	3201.548s
Per-ledger p50 / p95 / p99	1099.998 / 1449.999 / 1824.999 ms
Golden DB fetch+decompress	2436s
stellar-core	`v27.0.0`
Workflow run	#27982167923

github-actions · 2026-06-22T23:44:39Z

📈 Ingest load test — `19a3115`

Profile	Ledgers	Wall-clock	Ledgers/sec	ms/ledger	p50 / p95 / p99 ms
load-test-ledgers-v27-oz	1000	1234.299s	0.81	1235.54	1150 / 1699.998 / 1924.999
load-test-ledgers-v27-sac	1000	1137.524s	0.88	1137.52	1149.999 / 1225 / 1275.001
load-test-ledgers-v27-soroswap	1000	829.149s	1.21	829.15	825.001 / 900.002 / 974.999

Metric	Value
Ledgers replayed	3000
Initial DB ledger count	120960
Overall throughput	0.94 ledgers/sec
Overall ingest wall-clock	3200.973s
Per-ledger p50 / p95 / p99	1099.998 / 1450 / 1824.999 ms
Golden DB fetch+decompress	2438s
stellar-core	`v27.0.0`
Workflow run	#27984593953

…figs

github-actions · 2026-06-23T04:34:31Z

⏳ Load test launching on i-04965e211d076ce6f (commit 103b64849c20790426cd22eba966ffbfdd96c127).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/28002422012
Posting results when the run finishes.

github-actions · 2026-06-23T07:04:46Z

📈 Ingest load test — `103b648`

Profile	Ledgers	Wall-clock	Ledgers/sec	ms/ledger	p50 / p95 / p99 ms
load-test-ledgers-v27-oz	1000	1235.075s	0.81	1236.31	1150 / 1675.001 / 1925
load-test-ledgers-v27-sac	1000	1138.124s	0.88	1138.12	1149.999 / 1225 / 1275
load-test-ledgers-v27-soroswap	1000	829.550s	1.21	829.55	849.997 / 900.004 / 974.999

Metric	Value
Ledgers replayed	3000
Initial DB ledger count	120960
Overall throughput	0.94 ledgers/sec
Overall ingest wall-clock	3202.749s
Per-ledger p50 / p95 / p99	1099.998 / 1450 / 1824.999 ms
Golden DB fetch+decompress	2454s
stellar-core	`v27.0.0`
Workflow run	#28002422012

# Conflicts: # go.mod # go.sum

github-actions · 2026-06-23T19:13:49Z

⏳ Load test launching on i-0bfa652ea22b1901d (commit a6241f66163e8fc6d68a7b63c737867b04454471).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/28050597017
Posting results when the run finishes.

github-actions · 2026-06-23T21:43:21Z

📈 Ingest load test — `a6241f6`

Profile	Ledgers	Wall-clock	Ledgers/sec	ms/ledger	p50 / p95 / p99 ms
load-test-ledgers-v27-oz	1000	1235.175s	0.81	1236.41	1150 / 1699.997 / 1924.999
load-test-ledgers-v27-sac	1000	1137.950s	0.88	1137.95	1149.999 / 1225 / 1275
load-test-ledgers-v27-soroswap	1000	829.075s	1.21	829.07	825.002 / 924.997 / 974.996

Metric	Value
Ledgers replayed	3000
Initial DB ledger count	120960
Overall throughput	0.94 ledgers/sec
Overall ingest wall-clock	3202.199s
Per-ledger p50 / p95 / p99	1099.991 / 1450 / 1825.001 ms
Golden DB fetch+decompress	2439s
stellar-core	`v27.0.0`
Workflow run	#28050597017

…M risks

github-actions · 2026-06-24T19:56:46Z

⏳ Load test launching on i-0e4266d57523538ec (commit 21b80454838bc00535f8e212cff823b484efa577).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/28125731936
Posting results when the run finishes.

github-actions · 2026-06-24T22:18:48Z

📈 Ingest load test — `21b8045`

Profile	Ledgers	ms/ledger	p50 / p95 / p99 ms	max ms
load-test-ledgers-v27-oz	1000	1198.542	1117.151 / 1628.251 / 1918.960	2787.028
load-test-ledgers-v27-sac	1000	1096.993	1106.204 / 1182.071 / 1239.080	1304.020
load-test-ledgers-v27-soroswap	1000	792.811	804.091 / 867.349 / 925.778	1034.136

Metric	Value
Ledgers replayed	3000
Initial DB ledger count	120960
Throughput	0.94 ledgers/sec
Elapsed wall-clock	3188.772s
Ingest busy-time	3088.347s (96.9% utilization)
Per-ledger p50 / p95 / p99	1052.177 / 1422.112 / 1732.868 ms
Golden DB fetch+decompress	2439s
stellar-core	`v27.0.0`
Workflow run	#28125731936-1

Resolve go.mod/go.sum SDK conflict in favor of 6181cdf8bda5 (descendant of main's fc9acb3f1ba6), which carries the merged loadtest API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Drop the temporary @789121c0... pin added during load-test CI iteration; main tracks system-test master per repo convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cjonas9 added 26 commits May 8, 2026 21:32

pull initial work from branch load-testing

3392f09

add ledger generation test adapted for RPC

ffe27bc

add apply load config

65da3b5

add generated ledger output to infrastructure/testdata/

34f086d

add basic ingestion of synthetic ledgers phase

80c982d

disable debug logs for load test for timeout reasons

94c69b7

add functions for snapshotting + restoring test DB

f4a16f9

improve ad restructure db restoration helpers/API

1d53b96

finish DB restoration logic flow and wiring

baf2255

skip migrations/fee-stats in load test mode

1647464

ingest test: refactor, minor semantic fixes

2f14765

test.go: add retention window to config, fix fake history archive for…

d7c90a9

… real DB test

minor db restore/trim helper fixes

0390757

rename restore backed-up ledgers function for accuracy

e0a86e7

refactor, add env vars, change DB helpers to take sequences

f151a35

remove db restoration functionality

bffb101

add performance metrics json emission functionality

e04d51d

migrate to polling getHealth, change ingest test limits to 1000 ledgers

bd8c784

remove ledger fixtures

c7bc001

add workflow and script

786423d

fix yaml referencing wrong path for script

1606829

fix yml parsing indentation bug

7d41b1a

use head-object for metadata rather than tags

b701108

refine workflow + instance script

b1cec1d

add apply load cfg

b9ef27e

testing: on-push runs

73df1e7

Copilot AI review requested due to automatic review settings May 15, 2026 22:40

Copilot started reviewing on behalf of cjonas9 May 15, 2026 22:40 View session

minor yml syntax fixes

241bdf8

remove ingest phase dependence on unnecessary supplied apply load con…

103b648

…figs

sydneynotthecity added this to Platform Scrum Jun 23, 2026

github-project-automation Bot moved this to To Do in Platform Scrum Jun 23, 2026

sydneynotthecity added this to the platform sprint 73 milestone Jun 23, 2026

cjonas9 added 4 commits June 23, 2026 14:31

bump go sdk to v0.6.x

7e13bf3

bump go sdk again, merge SDK main into loadtest-patch

8333974

Merge remote-tracking branch 'origin/main' into apply-load

f3f3d58

# Conflicts: # go.mod # go.sum

update comments in light of changes

a6241f6

cjonas9 mentioned this pull request Jun 23, 2026

ingest/loadtest: expand loadtest functionality to handle multiple ledger bundles stellar/go-stellar-sdk#5959

Merged

7 tasks

tamirms reviewed Jun 23, 2026

View reviewed changes

cjonas9 added 2 commits June 24, 2026 15:05

rework handshake into instance->s3 results push, fix minor leaks + OO…

8d5f523

…M risks

replace polling-based ledger timing computation with daemon hook

21b8045

tamirms approved these changes Jun 24, 2026

View reviewed changes

cjonas9 and others added 6 commits June 25, 2026 00:18

remove trigger-on-push behavior

a8931fe

bump SDK to latest commit to main

6666e63

Merge branch 'main' into apply-load

53557d4

Resolve go.mod/go.sum SDK conflict in favor of 6181cdf8bda5 (descendant of main's fc9acb3f1ba6), which carries the merged loadtest API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

e2e: restore system-test workflow ref to @master

c9ce090

Drop the temporary @789121c0... pin added during load-test CI iteration; main tracks system-test master per repo convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cleaned up comments for readability

ff3bfef

Merge branch 'main' into apply-load

c29fcde

cjonas9 merged commit 539b093 into main Jun 26, 2026
17 checks passed

cjonas9 deleted the apply-load branch June 26, 2026 17:38

github-project-automation Bot moved this from To Do to Done in Platform Scrum Jun 26, 2026

Uh oh!

Conversation

cjonas9 commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Known limitations

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

📈 Ingest load test — 1f974a4

Uh oh!

github-actions Bot commented Jun 22, 2026

📈 Ingest load test — 19a3115

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

📈 Ingest load test — 103b648

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

📈 Ingest load test — a6241f6

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

📈 Ingest load test — 21b8045

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cjonas9 commented May 15, 2026 •

edited

Loading

📈 Ingest load test — `1f974a4`

📈 Ingest load test — `19a3115`

📈 Ingest load test — `103b648`

📈 Ingest load test — `a6241f6`

📈 Ingest load test — `21b8045`