From d5713252ad6889442243299f7abbfd78a9e3f6bd Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Thu, 16 Jul 2026 21:45:31 +0300 Subject: [PATCH] docs(concepts): consolidated Performance page Add docs/concepts/performance.md, a hub tying the three 0.11.0 performance levers together: NOTIFY dedup, batched terminal flush, and autovacuum tooling. Owns the synthesis (a which-knob-for-which-workload table and a two-column benchmark table separating CI-gated statement counts from illustrative loopback wall-clock); links to the existing mechanics rather than moving them. Back-links added from the subscriber batching and Alembic autovacuum sections. Design: planning/changes/2026-07-16.03-performance-docs-page.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/concepts/performance.md | 105 ++++++++++++++++++ docs/operations/alembic.md | 3 + docs/usage/subscriber.md | 3 + mkdocs.yml | 1 + .../2026-07-16.03-performance-docs-page.md | 88 +++++++++++++++ 5 files changed, 200 insertions(+) create mode 100644 docs/concepts/performance.md create mode 100644 planning/changes/2026-07-16.03-performance-docs-page.md diff --git a/docs/concepts/performance.md b/docs/concepts/performance.md new file mode 100644 index 0000000..16c6cd5 --- /dev/null +++ b/docs/concepts/performance.md @@ -0,0 +1,105 @@ +# Performance + +The outbox transport is Postgres rows, so its performance is governed by two +things: **round-trips** (every publish and every terminal completion is a +statement against the database) and **table churn** (every message is one +`INSERT`, one lease `UPDATE`, and one terminal `DELETE`, so dead tuples pile up at +roughly twice the message rate and autovacuum has to reclaim them). Three levers +address those. Two are automatic or opt-in code knobs; one is table hygiene you +apply once in a migration. + +This page is the map: what each lever buys, which one your workload needs, and +where the mechanics live. If you already know which knob you want, jump to its +detail page from the [three levers](#the-three-levers) below. + +## What each lever buys + +The left column is what the [benchmark harness](#measuring-it-yourself) locks in +CI — deterministic statement counts, identical on any machine. The right column +is illustrative wall-clock from a one-off run on loopback Postgres: real, but +**machine-dependent**, so treat it as a shape, not a promise. + +| Lever | CI-gated (deterministic) | Illustrative (loopback, one-off) | +|---|---|---| +| Producer NOTIFY dedup | `pg_notify` statements per bulk publish: **5000 → 1** | ~1.9× bulk-publish throughput; grows with DB round-trip latency | +| Batched terminal flush | terminal `DELETE`s: **5000 → 50** (100×) at `terminal_flush_batch_size=100` | one batched worker out-throughputs a four-worker per-row subscriber | +| Autovacuum tuning | *(no benchmark — time-and-throughput dependent)* | churn demo: throughput knob bounds bloat under sustained load | + +**Gated vs illustrative.** The gated counts come from `benchmarks/baseline.json` +and fail CI if they regress, so they are guarantees about *statement count*, not +speed. The wall-clock figures were measured once and depend on your CPU, disk, and +especially network latency to Postgres — reproduce them on your own hardware +before quoting them. Autovacuum has no gated number at all: its benefit is +time-and-throughput dependent and cannot be captured as a deterministic count (see +[Autovacuum tuning](../operations/alembic.md#autovacuum-tuning-recommended) for +why). + +## Which knob for which workload + +| Your workload | Reach for | Why | +|---|---|---| +| Many rows to one queue in one transaction (bulk / `publish_batch`) | Nothing — **NOTIFY dedup is automatic** | The redundant `pg_notify` round-trips are already gone; delivery is unchanged. | +| High terminal throughput, **idempotent** handlers, few workers | `terminal_flush_batch_size` > 1 | Collapses per-message `DELETE`s into one per batch; reaches high throughput without spending more workers' connections. | +| Exactly-once-sensitive or low-volume queue | Leave `terminal_flush_batch_size=1` | Batching widens the crash-redelivery window — not worth it here. | +| Table bloating / vacuum can't keep up under sustained churn | `outbox_autovacuum_ddl(...)`, and raise the throughput knob | Eligibility settings fire vacuum sooner; `vacuum_cost_delay` lets it keep pace. | +| Need more parallel handler capacity | `max_workers` / `fetch_batch_size` | More concurrency — but mind the [connection budget](../usage/subscriber.md#connection-budget). | + +Single-publish-per-transaction workloads see no change from NOTIFY dedup (one row +still emits one NOTIFY), and the batched-flush win is largest at low `max_workers` +(where per-row deletes serialise) and narrows as worker parallelism rises. + +## The three levers + +### Producer: NOTIFY dedup (automatic) + +`broker.publish` / `publish_batch` used to emit one `SELECT pg_notify(...)` per +call, so N publishes to the same queue in one transaction cost N NOTIFY +round-trips — N−1 of them pure waste, since Postgres already coalesces identical +notifications per transaction at delivery. Since 0.11.0 the producer emits **one +`pg_notify` per (transaction, queue)**. It is default-on, has no knob, and is +behavior-preserving: the subscriber still gets its wake, fired inline at the first +publish. You do not configure this — it just makes bulk publishing cheaper. See +[How it works](../introduction/how-it-works.md) for the write path. + +### Subscriber: batched terminal flush (opt-in) + +By default each processed row is deleted with its own `DELETE` — one round-trip +per message, which is the throughput ceiling at low `max_workers`. Set +`terminal_flush_batch_size` above `1` to coalesce completed rows into one +`DELETE … RETURNING` per batch. The tradeoff is a **wider crash-redelivery +window**: on an ungraceful crash, up to a full batch of already-handled rows are +redelivered, so handlers must be idempotent. Enable it per subscriber for +high-throughput idempotent queues; leave it off otherwise. Full mechanics, +lease-ceiling sizing, and backlog-depth effects are in +[Batching terminal deletes](../usage/subscriber.md#batching-terminal-deletes). + +### Storage: autovacuum tuning (apply once) + +A high-churn queue table defeats Postgres' default autovacuum: the +`scale_factor = 0.2` bar is size-dependent and, on a table whose `reltuples` +estimate has gone stale, fires rarely — the classic queue-table death-spiral. +`outbox_autovacuum_ddl("outbox")` renders the `ALTER TABLE … SET (autovacuum_*)` +statement to drop into a migration; it sets `scale_factor = 0` with a constant +threshold (**eligibility** — fire on a fixed dead-tuple count, size-independent). +Under heavy sustained churn the binding constraint is instead vacuum **throughput** +— pass `vacuum_cost_delay` / `vacuum_cost_limit` to let vacuum keep pace. A +`validate_schema(check_autovacuum=True)` probe can gate that the eligibility +settings are applied. Full guidance, the eligibility-vs-throughput split, and the +I/O caution on `vacuum_cost_delay=0` are in +[Autovacuum tuning](../operations/alembic.md#autovacuum-tuning-recommended). + +## Measuring it yourself + +The gated numbers above come from the repository's benchmark harness. Contributors +can run it against a local Postgres: + +- `just bench` — runs the producer/consumer workload sweep and prints per-message + DB counters (split by leading SQL keyword). +- `just bench-check` — gates the deterministic counts against + `benchmarks/baseline.json`; this is what CI runs. + +The statement-count reductions (`select_calls`, `delete_calls`, `insert_calls`, +…) are machine-independent and exact. Wall-clock throughput is not — it scales +with your hardware and, most of all, round-trip latency to the database, so a +networked Postgres will show a larger absolute win from the round-trip reductions +than the loopback figures here. diff --git a/docs/operations/alembic.md b/docs/operations/alembic.md index cdcb00d..06bd9ff 100644 --- a/docs/operations/alembic.md +++ b/docs/operations/alembic.md @@ -174,6 +174,9 @@ The outbox is a high-churn queue table: every message is one `INSERT`, one lease the message rate**, and autovacuum has to reclaim them. Two independent levers matter, and they do different things. +For how this fits with the outbox's other performance levers, see the +[Performance](../concepts/performance.md) guide. + ### Eligibility — when autovacuum fires Postgres' default `autovacuum_vacuum_scale_factor = 0.2` fires vacuum only after a diff --git a/docs/usage/subscriber.md b/docs/usage/subscriber.md index 573e699..e1e06b7 100644 --- a/docs/usage/subscriber.md +++ b/docs/usage/subscriber.md @@ -204,6 +204,9 @@ per-row path). Enable it per subscriber when the queue is high-throughput and its handler is idempotent; leave it off for low-volume or exactly-once-sensitive queues. +For where this sits among the outbox's tuning levers and which workloads want it, +see the [Performance](../concepts/performance.md) guide. + ## Ack policy The default is `AckPolicy.NACK_ON_ERROR`: on a handler exception, the retry diff --git a/mkdocs.yml b/mkdocs.yml index 2e99dfb..68f1f55 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -12,6 +12,7 @@ nav: - 'Tutorial: Add a Kafka relay': tutorials/add-kafka-relay.md - Concepts: - How it works: introduction/how-it-works.md + - Performance: concepts/performance.md - Comparison: concepts/comparison.md - Instrumentation seams: concepts/instrumentation-seams.md - Guides: diff --git a/planning/changes/2026-07-16.03-performance-docs-page.md b/planning/changes/2026-07-16.03-performance-docs-page.md new file mode 100644 index 0000000..e211e53 --- /dev/null +++ b/planning/changes/2026-07-16.03-performance-docs-page.md @@ -0,0 +1,88 @@ +--- +summary: Add a consolidated user-facing Performance page (docs/concepts/performance.md) that tells the three-lever tuning story (NOTIFY dedup, batched terminal flush, autovacuum) with a which-knob-for-which-workload table and a two-column benchmark table (CI-gated counts vs illustrative wall-clock). Hub that links to the existing detail; no content moves. +--- + +# Design: Consolidated Performance docs page + +## Summary + +The outbox's performance story ships across 0.11.0 as three independent levers — +producer NOTIFY dedup, the subscriber's opt-in batched terminal flush, and the +`outbox_autovacuum_ddl` table-hygiene helper — but a reader has no single place to +understand them together or to decide which one their workload needs. Add one +user-facing page, `docs/concepts/performance.md`, that owns the synthesis: the +three levers, a which-knob-for-which-workload decision table, and a consolidated +benchmark table. It is a **hub**: the per-knob mechanics stay in their current +homes (subscriber reference, Alembic operations) and the page links to them; no +existing content moves. + +## Motivation + +After the three performance changes landed (#131, #133/#134, #135) the knowledge +is scattered: batched flush in `docs/usage/subscriber.md`, autovacuum in +`docs/operations/alembic.md`, and the NOTIFY dedup only in `architecture/producer.md` +(not surfaced in user docs as a perf item at all). Someone asking "my outbox is +slow / bloating, what do I turn?" must already know which of three pages to open. +A hub page answers that question in one place and routes to depth. + +## Design + +New `docs/concepts/performance.md`, added to the **Concepts** nav group, five +sections: + +1. **The cost model** — one paragraph: every publish and every terminal + completion is a Postgres round-trip, and a high-churn queue table bloats; the + three levers reduce round-trips or bound bloat. +2. **What each lever buys** — a two-column benchmark table. Left column = the + **CI-gated, deterministic** statement-count reductions the harness actually + locks in `benchmarks/baseline.json` (`select_calls` 5000 → 1 for the producer; + `delete_calls` 5000 → 50 for the batched consumer point). Right column = + **illustrative** wall-clock, explicitly labeled one-off / loopback / + machine-dependent (the ~1.9× bulk-publish figure; "one batched worker + out-throughputs four per-row workers"). A short note defines the two categories + and points at how to reproduce. +3. **Which knob for which workload** — the decision table (the centerpiece): + workload shape → lever(s). +4. **The three levers** — one tight subsection each (what / when, not re-explained + mechanics), each linking to its detail page: producer dedup → How it works; + batched flush → `subscriber.md#batching-terminal-deletes`; autovacuum → + `alembic.md#autovacuum-tuning-recommended`. Plus a closing note on the general + throughput knobs (`max_workers`, `fetch_batch_size`, + `subscriber.md#connection-budget`). +5. **Measuring it yourself** — pointer to the `benchmarks/` harness (`just bench` / + `just bench-check`), what `baseline.json` gates, and that wall-clock is + machine-dependent so reproduce locally. + +Two back-links added (one line each) so reference readers find the hub: the +subscriber "Batching terminal deletes" section and the Alembic "Autovacuum tuning" +section each gain a "see [Performance] for the consolidated tuning guide" pointer. + +The hub links **only within `docs/`** — never to `architecture/*.md`, which lives +outside the mkdocs tree and would fail the strict build. + +## Non-goals + +- **No content relocation.** The batching and autovacuum mechanics stay put; the + hub summarizes and links (avoids duplicated prose the drift guard can't keep in + sync). +- **No new benchmark.** The page cites existing gated numbers and clearly-labeled + one-off measurements; it commits no new harness scenario. +- **No code / behavior change.** Docs-only. + +## Testing + +- `just docs-build` (`mkdocs build --strict`) — clean; every referenced anchor + (`#batching-terminal-deletes`, `#connection-budget`, + `#autovacuum-tuning-recommended`) resolves. +- `just lint-ci` — planning validator (`planning: OK`) + link/anchor drift guard + pass. + +## Risk + +- **Number drift.** The gated counts are quoted from `baseline.json`; if a future + change moves them, the hub could go stale. Mitigation: the page frames the + gated column as "what CI locks" and points at `baseline.json` as the source of + truth, so the fix location is obvious. Low impact (docs, not behavior). +- **Anchor rot.** Linking to section anchors risks breakage if a heading is + renamed. Mitigation: the link/anchor drift guard in `lint-ci` catches exactly + this on every PR.