feat(loadtest): load testing framework#420
Draft
JeroenSoeters wants to merge 4 commits into
Draft
Conversation
Emit an OTel histogram (formae.datastore.query.duration_ms) per postgres query path so we can spot slow datastore operations during load tests and in production. Initialized alongside the existing pool metrics; a failure to create the histogram is non-fatal and leaves query duration unmetered.
Extend generate-stress-test-env.py to produce Azure and GCP resources alongside AWS, selectable via --clouds and configurable per-cloud via --subscription-azure / --project-gcp. Also wire up scale profiles so we can generate consistent small/medium/large environments without hand-picking counts. Used to seed Suite B regression baselines across the tier-1 cloud providers.
Add a Go-based harness that drives the Suite B load test lifecycle: apply, verify, soak, destroy. Each phase lives in its own file so they can be exercised independently; the harness composes them and reports results via the shared report type. Includes helpers for metric collection (scraped from the observability stack) and log checking, plus unit tests for config and report rendering.
Add the Suite B observability stack as a formae forma: a VPC-local LGTM deployment (Loki, Grafana, Tempo, Mimir, OTel collector) on ECS Fargate with an ALB, S3 backing buckets, EFS for shared state, and the IAM wiring to let the task pull images and write logs. Routes depend on the VPCGatewayAttachment resolvable rather than the internet gateway directly so the create/destroy ordering stays correct. This is the resident observability environment the harness scrapes metrics and logs from during load tests; applying it successfully is also our end-to-end validation of running 0.84 in production.
JeroenSoeters
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Suite B is our regression-detection framework for multi-cloud resource management. This PR lands the initial scaffolding so we can start gathering baselines.
scripts/generate-stress-test-env.pyto produce Azure and GCP resources alongside AWS, with scale profiles so we can regenerate consistent small/medium/large envs.tests/loadtest/harnesswith apply / verify / soak / destroy phases, metric collection against the observability stack, log checking, and a report renderer. Composed bytests/loadtest/loadtest_test.go.Draft while we finish wiring up the resident agent and apply the LGTM forma end-to-end (blocked on #419). Not expected to land before Suite B baselines have been captured against it.