From 3b563192b611f4bdb82da667904239686b72e121 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Wed, 8 Jul 2026 10:38:21 +0300 Subject: [PATCH 1/5] chore(planning): swap to convention 2.0.0 tooling Apply the lesnik512/planning-convention APPLY.md 1.x->2.0.0 update: overwrite planning/index.py and the canonical templates, add the glossary.md template, and drop the retired plan.md template. Co-Authored-By: Claude Opus 4.8 (1M context) --- planning/_templates/change.md | 4 +- planning/_templates/design.md | 34 +++----- planning/_templates/glossary.md | 15 ++++ planning/_templates/plan.md | 46 ---------- planning/index.py | 147 +++++++++++++++----------------- 5 files changed, 99 insertions(+), 147 deletions(-) create mode 100644 planning/_templates/glossary.md delete mode 100644 planning/_templates/plan.md diff --git a/planning/_templates/change.md b/planning/_templates/change.md index d4c8962..5aa7e81 100644 --- a/planning/_templates/change.md +++ b/planning/_templates/change.md @@ -5,8 +5,8 @@ summary: One line — shown in the generated index. Written at creation; finaliz # Change: One-line capitalized title **Lane:** lightweight — ≲30 LOC net, ≤2 files, no new file, no public-API -change, a single straightforward test. If it outgrows this, split into -`design.md` + `plan.md`. +change, a single straightforward test. If it outgrows this, rewrite it from +the design template. ## Goal diff --git a/planning/_templates/design.md b/planning/_templates/design.md index d63e22d..17dbee1 100644 --- a/planning/_templates/design.md +++ b/planning/_templates/design.md @@ -4,6 +4,10 @@ summary: One line — shown in the generated index. Written at creation; finaliz # Design: One-line capitalized title + + ## Summary One paragraph. What changes, at the level a reader needs to decide if this @@ -12,37 +16,23 @@ spec is worth reading in full. ## Motivation Why now. What is broken or missing. Concrete observations / numbers, not -abstract complaints. Link to memory entries or earlier specs when relevant. - -## Non-goals - -What is deliberately out of scope and (when nontrivial) why. Each item is -a sentence; one line each. +abstract complaints. ## Design -### 1. - What changes, in enough detail that a reader who has not seen the codebase -can follow. Code samples / diagrams welcome. +can follow. Sketches and interface fragments welcome; never the full +diff-to-be. Reference rejected alternatives in `decisions/` instead of +retelling them. -### 2. - -... - -## Operations - -Out-of-repo steps (DNS, infra, external account changes). Omit if none. - -## Out of scope +## Non-goals -Already covered above under Non-goals if appropriate. Repeat-list of -explicitly-excluded follow-ups belongs here when the list is long. +What is deliberately out of scope and (when nontrivial) why. One line each. ## Testing -How we know it landed correctly. New pytest? Smoke check on live URL? -Lint pass? Be specific. +How we know it landed correctly. Be specific: the command and the expected +signal. ## Risk diff --git a/planning/_templates/glossary.md b/planning/_templates/glossary.md new file mode 100644 index 0000000..82385c3 --- /dev/null +++ b/planning/_templates/glossary.md @@ -0,0 +1,15 @@ +# Glossary + +The project's ubiquitous language — the domain terms that code, specs, and +capability pages share. Living prose, no frontmatter, dated by git. Each entry is +a term, what it *is* (not what it does), and the synonyms to avoid. No +implementation detail; this is a glossary, not a spec. + +**Term**: +A one-or-two-sentence definition of what it is. +_Avoid_: rejected-synonym, another-one + +**Another term**: +Define what it is, tightly. Group related terms under `##` subheadings when +natural clusters emerge; a flat list is fine when they don't. +_Avoid_: … diff --git a/planning/_templates/plan.md b/planning/_templates/plan.md deleted file mode 100644 index 132d720..0000000 --- a/planning/_templates/plan.md +++ /dev/null @@ -1,46 +0,0 @@ -# — implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps -> use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** One sentence — what shipping this plan achieves. No design -rationale; link to the spec for that. - -**Spec:** [`design.md`](./design.md) - -**Branch:** `feat/my-change` (or `fix/`, `chore/`, etc.) - -**Commit strategy:** Per-task commits / single commit / squash on merge. -Whichever fits. - ---- - -### Task 1: - -**Files:** -- Modify: `path/to/file.py` -- Create: `path/to/new.py` - -One sentence on what this task accomplishes. No deeper reasoning — that's -in the spec. - -- [ ] **Step 1: ** - - Run / edit / verify command. Expected output. - -- [ ] **Step 2: ** - - ... - -- [ ] **Step 3: Commit** - - ```bash - git add path/to/file.py - git commit -m ": " - ``` - ---- - -### Task 2: ... diff --git a/planning/index.py b/planning/index.py index 8916661..2d70ac3 100644 --- a/planning/index.py +++ b/planning/index.py @@ -1,14 +1,12 @@ -# ruff: noqa: INP001, D212 # planning/ is not a Python package; D212/D213 conflict differs from faststream-outbox -""" -Generate the planning index from frontmatter. +# ruff: noqa: INP001 # planning/ is not a Python package (this file is vendored into consumers' planning/) +"""Generate the planning index from frontmatter. -Run via ``just index``. Globs ``planning/changes/*/`` (each bundle's -``design.md``, falling back to ``change.md``) and ``planning/decisions/*.md``, -reads their frontmatter, and prints a Markdown listing to stdout — changes -then decisions, newest-first. Never writes a file: +Run via ``just index``. Globs ``planning/changes/*.md`` and +``planning/decisions/*.md``, reads their frontmatter, and prints a Markdown +listing to stdout — changes then decisions, newest-first. Never writes a file: the listing is a query over the files, not a committed artifact. -``date`` and ``slug`` are derived from the directory / file name, not +``date`` and ``slug`` are derived from the file name, not frontmatter — the name is the single source of truth for both. """ @@ -17,12 +15,10 @@ import sys -CHANGES_DIR = pathlib.Path(__file__).parent / "changes" -DECISIONS_DIR = pathlib.Path(__file__).parent / "decisions" +ROOT = pathlib.Path(__file__).parent VALID_DECISION_STATUS = {"accepted", "superseded"} -BUNDLE_RE = re.compile(r"^(?P\d{4}-\d{2}-\d{2})\.\d{2}-(?P.+)$") +CHANGE_RE = re.compile(r"^(?P\d{4}-\d{2}-\d{2})\.\d{2}-(?P.+)$") DECISION_RE = re.compile(r"^(?P\d{4}-\d{2}-\d{2})-(?P.+)$") -ALLOWED_BUNDLE_FILES = {"design.md", "plan.md", "change.md"} SPEC_REQUIRED = ("summary",) DECISION_REQUIRED = ("status", "summary") @@ -47,7 +43,7 @@ def parse_frontmatter(text: str) -> dict[str, str]: def _named(fields: dict[str, str], name: str, pattern: re.Pattern[str]) -> dict[str, str]: - """Inject ``date``/``slug`` derived from a dir/file name into ``fields``.""" + """Inject ``date``/``slug`` derived from a file name into ``fields``.""" match = pattern.match(name) if match: fields["date"] = match.group("date") @@ -55,30 +51,29 @@ def _named(fields: dict[str, str], name: str, pattern: re.Pattern[str]) -> dict[ return fields -def load_bundles() -> list[dict[str, str]]: - """Read each bundle's summary; derive date/slug from the directory name.""" - bundles: list[dict[str, str]] = [] - for bundle in sorted(CHANGES_DIR.iterdir()): - if not bundle.is_dir(): - continue - spec = bundle / "design.md" - if not spec.exists(): - spec = bundle / "change.md" - if not spec.exists(): +def load_changes(root: pathlib.Path) -> list[dict[str, str]]: + """Read each change file's summary; derive date/slug from the file name.""" + changes_dir = root / "changes" + changes: list[dict[str, str]] = [] + if not changes_dir.is_dir(): + return changes + for path in sorted(changes_dir.glob("*.md")): + if path.name == "README.md" or path.name.startswith(("_", ".")): continue - fields = _named(parse_frontmatter(spec.read_text(encoding="utf-8")), bundle.name, BUNDLE_RE) - fields["path"] = f"changes/{bundle.name}/{spec.name}" - fields["name"] = bundle.name - bundles.append(fields) - return bundles + fields = _named(parse_frontmatter(path.read_text(encoding="utf-8")), path.stem, CHANGE_RE) + fields["path"] = f"changes/{path.name}" + fields["name"] = path.stem + changes.append(fields) + return changes -def load_decisions() -> list[dict[str, str]]: +def load_decisions(root: pathlib.Path) -> list[dict[str, str]]: """Read each decision's frontmatter; derive date/slug from the file name.""" + decisions_dir = root / "decisions" decisions: list[dict[str, str]] = [] - if not DECISIONS_DIR.is_dir(): + if not decisions_dir.is_dir(): return decisions - for path in sorted(DECISIONS_DIR.glob("*.md")): + for path in sorted(decisions_dir.glob("*.md")): if path.name == "README.md" or path.name.startswith("_"): continue fields = _named(parse_frontmatter(path.read_text(encoding="utf-8")), path.stem, DECISION_RE) @@ -88,24 +83,24 @@ def load_decisions() -> list[dict[str, str]]: return decisions -def format_row(bundle: dict[str, str]) -> str: - """Render one bundle as a Markdown list item.""" - slug = bundle.get("slug", "?") - path = bundle.get("path", "") - date = bundle.get("date", "") - summary = bundle.get("summary") or "(no summary)" +def format_row(row: dict[str, str]) -> str: + """Render one change or decision as a Markdown list item.""" + slug = row.get("slug", "?") + path = row.get("path", "") + date = row.get("date", "") + summary = row.get("summary") or "(no summary)" line = f"- **[{slug}]({path})** ({date}) — {summary}" - if bundle.get("supersedes"): - line += f" _(supersedes {bundle['supersedes']})_" - if bundle.get("superseded_by"): - line += f" _(superseded by {bundle['superseded_by']})_" + if row.get("supersedes"): + line += f" _(supersedes {row['supersedes']})_" + if row.get("superseded_by"): + line += f" _(superseded by {row['superseded_by']})_" return line -def render(bundles: list[dict[str, str]], decisions: list[dict[str, str]]) -> str: +def render(changes: list[dict[str, str]], decisions: list[dict[str, str]]) -> str: """Render the full Markdown listing: changes then decisions, newest-first.""" out = ["# Planning index", "", "_Generated by `just index` — do not edit._", "", "## Changes", ""] - change_rows = sorted(bundles, key=lambda b: b.get("name", ""), reverse=True) + change_rows = sorted(changes, key=lambda b: b.get("name", ""), reverse=True) out += [format_row(b) for b in change_rows] if change_rows else ["_None._"] out += ["", "## Decisions", ""] decision_rows = sorted(decisions, key=lambda d: d.get("name", ""), reverse=True) @@ -119,32 +114,15 @@ def _require(fields: dict[str, str], keys: tuple[str, ...], rel: str, violations violations.extend(f"{rel}: missing or empty frontmatter key '{key}'" for key in keys if not fields.get(key)) -def _check_spec_file(path: pathlib.Path, rel: str, violations: list[str]) -> None: - """Validate a design.md / change.md spec file (requires `summary`).""" +def _check_change(path: pathlib.Path, violations: list[str]) -> None: + """Validate one change file (requires `summary`).""" + rel = f"changes/{path.name}" + if CHANGE_RE.match(path.stem) is None: + violations.append(f"{rel}: file name is not 'YYYY-MM-DD.NN-slug.md'") fields = parse_frontmatter(path.read_text(encoding="utf-8")) _require(fields, SPEC_REQUIRED, rel, violations) -def _check_bundle(bundle: pathlib.Path, violations: list[str]) -> None: - """Validate one change bundle directory.""" - rel = f"changes/{bundle.name}" - if BUNDLE_RE.match(bundle.name) is None: - violations.append(f"{rel}: directory name is not 'YYYY-MM-DD.NN-slug'") - violations.extend( - f"{rel}/{child.name}: unexpected file in bundle (allowed: {', '.join(sorted(ALLOWED_BUNDLE_FILES))})" - for child in sorted(bundle.iterdir()) - if child.name not in ALLOWED_BUNDLE_FILES - ) - design = bundle / "design.md" - change = bundle / "change.md" - if not design.exists() and not change.exists(): - violations.append(f"{rel}: bundle has neither design.md nor change.md") - for spec_file in (design, change): - if spec_file.exists(): - _check_spec_file(spec_file, f"{rel}/{spec_file.name}", violations) - # plan.md carries no frontmatter — its identity comes from the bundle dir. - - def _check_decision(path: pathlib.Path, violations: list[str]) -> None: """Validate one decision file (requires `status` + `summary`).""" rel = f"decisions/{path.name}" @@ -157,24 +135,39 @@ def _check_decision(path: pathlib.Path, violations: list[str]) -> None: violations.append(f"{rel}: invalid status '{status}' (allowed: {', '.join(sorted(VALID_DECISION_STATUS))})") -def check() -> list[str]: - """Validate every bundle and decision; return the list of violation strings.""" +def check(root: pathlib.Path) -> list[str]: + """Validate every change file and decision; return the list of violation strings.""" violations: list[str] = [] - for bundle in sorted(CHANGES_DIR.iterdir()): - if bundle.is_dir(): - _check_bundle(bundle, violations) - if DECISIONS_DIR.is_dir(): - for path in sorted(DECISIONS_DIR.glob("*.md")): + changes_dir = root / "changes" + decisions_dir = root / "decisions" + if changes_dir.is_dir(): + for path in sorted(changes_dir.iterdir()): + if path.is_dir(): + violations.append( + f"changes/{path.name}: directory found — convention 2.0.0 uses flat change files " + f"(changes/YYYY-MM-DD.NN-slug.md; see CHANGELOG 2.0.0 for the migration)" + ) + continue + if path.name == "README.md" or path.name.startswith(("_", ".")): + continue + if path.suffix != ".md": + violations.append(f"changes/{path.name}: unexpected non-md file in changes/") + else: + _check_change(path, violations) + if decisions_dir.is_dir(): + for path in sorted(decisions_dir.glob("*.md")): if path.name == "README.md" or path.name.startswith("_"): continue _check_decision(path, violations) return violations -def main() -> int: - """Print the listing to stdout, or validate bundles with --check.""" - if "--check" in sys.argv[1:]: - violations = check() +def main(argv: list[str] | None = None, root: pathlib.Path | None = None) -> int: + """Print the listing to stdout, or validate change files and decisions with --check.""" + argv = sys.argv[1:] if argv is None else argv + root = ROOT if root is None else root + if "--check" in argv: + violations = check(root) if violations: sys.stderr.write(f"planning: {len(violations)} violation(s)\n") for violation in violations: @@ -182,7 +175,7 @@ def main() -> int: return 1 sys.stdout.write("planning: OK\n") return 0 - sys.stdout.write(render(load_bundles(), load_decisions())) + sys.stdout.write(render(load_changes(root), load_decisions(root))) return 0 From d4493af0474b27f635591be71251e19af699c065 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Wed, 8 Jul 2026 10:38:28 +0300 Subject: [PATCH 2/5] refactor(planning): flatten change folders to flat files Convention 2.0.0 drops the per-change folder in favor of a single changes/YYYY-MM-DD.NN-.md file. Flatten all 8 existing folders (design.md or change.md -> .md) and drop the 6 committed plan.md files (plans are ephemeral under 2.0.0, never version-controlled). Bodies are unmodified -- only their location changes. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...2026-06-03.01-faststream-0.7-migration.md} | 0 ....01-faststream-0.7.1-testbroker-typing.md} | 0 .../plan.md | 331 ------ ...26-06-13.01-robustness-docs-test-audit.md} | 0 .../plan.md | 966 ------------------ ... => 2026-06-13.02-codify-release-notes.md} | 0 .../plan.md | 176 ---- ...-06-13.03-portable-planning-convention.md} | 0 .../plan.md | 709 ------------- ...6-06-23.01-pending-commits-deep-module.md} | 0 .../plan.md | 646 ------------ ...-06-24.01-commit-scheduler-deep-module.md} | 0 .../plan.md | 576 ----------- ...026-06-24.02-route-classify-extraction.md} | 0 14 files changed, 3404 deletions(-) rename planning/changes/{2026-06-03.01-faststream-0.7-migration/design.md => 2026-06-03.01-faststream-0.7-migration.md} (100%) rename planning/changes/{2026-06-04.01-faststream-0.7.1-testbroker-typing/design.md => 2026-06-04.01-faststream-0.7.1-testbroker-typing.md} (100%) delete mode 100644 planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md rename planning/changes/{2026-06-13.01-robustness-docs-test-audit/design.md => 2026-06-13.01-robustness-docs-test-audit.md} (100%) delete mode 100644 planning/changes/2026-06-13.01-robustness-docs-test-audit/plan.md rename planning/changes/{2026-06-13.02-codify-release-notes/design.md => 2026-06-13.02-codify-release-notes.md} (100%) delete mode 100644 planning/changes/2026-06-13.02-codify-release-notes/plan.md rename planning/changes/{2026-06-13.03-portable-planning-convention/design.md => 2026-06-13.03-portable-planning-convention.md} (100%) delete mode 100644 planning/changes/2026-06-13.03-portable-planning-convention/plan.md rename planning/changes/{2026-06-23.01-pending-commits-deep-module/design.md => 2026-06-23.01-pending-commits-deep-module.md} (100%) delete mode 100644 planning/changes/2026-06-23.01-pending-commits-deep-module/plan.md rename planning/changes/{2026-06-24.01-commit-scheduler-deep-module/design.md => 2026-06-24.01-commit-scheduler-deep-module.md} (100%) delete mode 100644 planning/changes/2026-06-24.01-commit-scheduler-deep-module/plan.md rename planning/changes/{2026-06-24.02-route-classify-extraction/change.md => 2026-06-24.02-route-classify-extraction.md} (100%) diff --git a/planning/changes/2026-06-03.01-faststream-0.7-migration/design.md b/planning/changes/2026-06-03.01-faststream-0.7-migration.md similarity index 100% rename from planning/changes/2026-06-03.01-faststream-0.7-migration/design.md rename to planning/changes/2026-06-03.01-faststream-0.7-migration.md diff --git a/planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/design.md b/planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing.md similarity index 100% rename from planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/design.md rename to planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing.md diff --git a/planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md b/planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md deleted file mode 100644 index 25e9827..0000000 --- a/planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md +++ /dev/null @@ -1,331 +0,0 @@ -# FastStream 0.7.1 TestBroker typing alignment — implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Bump the FastStream floor to 0.7.1 and delete the `fake_test_broker` workaround helper now that upstream PR ag2ai/faststream#2903 makes its `isinstance` narrowing unnecessary. - -**Architecture:** Test-infrastructure refactor only — no library code changes. Replace 24 `async with fake_test_broker(...)` callsites with `async with TestKafkaBroker(...)` directly, then delete the helper. Commit incrementally per file so any breakage is bisected to a single change. - -**Tech Stack:** Python 3.11+, FastStream 0.7.1, aiokafka, pytest, uv, ty. - -**Spec:** [`design.md`](./design.md) - -**Branch:** `chore/faststream-0.7.1-testbroker-typing` (already created). - ---- - -## File map - -- **Modify:** `pyproject.toml` — bump faststream pin floor. -- **Modify:** `uv.lock` — regenerated by `uv lock --upgrade-package faststream`. -- **Modify:** `tests/test_middleware.py` — swap import and 20 callsites. -- **Modify:** `tests/test_healthcheck.py` — swap import and 4 callsites. -- **Modify:** `tests/mocks.py` — delete `fake_test_broker` and its now-unused imports. - -No new files. No changes under `faststream_concurrent_aiokafka/`. - ---- - -## Task 1: Bump the FastStream floor to 0.7.1 - -**Files:** -- Modify: `pyproject.toml:28` -- Modify: `uv.lock` (regenerated) - -- [ ] **Step 1.1: Edit `pyproject.toml:28`** - -Before: -```toml - "faststream[kafka]>=0.7,<0.8", -``` - -After: -```toml - "faststream[kafka]>=0.7.1,<0.8", -``` - -- [ ] **Step 1.2: Regenerate the lockfile (scoped to faststream)** - -Run: `uv lock --upgrade-package faststream` -Expected: `uv.lock` updates the faststream entry to a 0.7.1+ version. Other packages should not move (the scoped upgrade keeps the diff minimal). If the resolver reports the new floor cannot be satisfied, stop and surface the conflict. - -- [ ] **Step 1.3: Sync the environment** - -Run: `uv sync --all-extras --frozen --group lint` -Expected: success; `faststream==0.7.1` (or newer 0.7.x) is now installed. - -- [ ] **Step 1.4: Verify the existing helper-based tests still pass on 0.7.1** - -Run: `uv run --no-sync pytest tests/test_middleware.py tests/test_healthcheck.py -q` -Expected: all tests pass. The helper's `isinstance` assert still holds (single-broker call), so 0.7.1 is a drop-in. If anything fails here, the issue is the version bump itself, not the upcoming refactor. - -- [ ] **Step 1.5: Commit** - -```bash -git add pyproject.toml uv.lock -git commit -m "$(cat <<'EOF' -chore: bump faststream pin to >=0.7.1 - -Adopts upstream TestBroker[Broker, EnterType] typing fix -(ag2ai/faststream#2903). Library code unchanged; the helper -cleanup follows in later commits. -EOF -)" -``` - ---- - -## Task 2: Inline `TestKafkaBroker` in `tests/test_middleware.py` - -**Files:** -- Modify: `tests/test_middleware.py` (imports + 20 callsites) - -- [ ] **Step 2.1: Swap the faststream import on line 10** - -Before: -```python -from faststream.kafka import KafkaBroker -``` - -After: -```python -from faststream.kafka import KafkaBroker, TestKafkaBroker -``` - -- [ ] **Step 2.2: Drop `fake_test_broker` from the mocks import on line 19** - -Before: -```python -from tests.mocks import fake_test_broker, patched_message -``` - -After: -```python -from tests.mocks import patched_message -``` - -- [ ] **Step 2.3: Replace every callsite (20 occurrences)** - -Replace every line matching `async with fake_test_broker(setup_broker) as test_broker:` with `async with TestKafkaBroker(setup_broker) as test_broker:`. - -Use a single `Edit` with `replace_all=true`: -- `old_string`: `async with fake_test_broker(setup_broker) as test_broker:` -- `new_string`: `async with TestKafkaBroker(setup_broker) as test_broker:` - -Confirm afterward with: `grep -c "fake_test_broker" tests/test_middleware.py` -Expected: `0`. - -Confirm replacement count with: `grep -c "async with TestKafkaBroker(setup_broker) as test_broker:" tests/test_middleware.py` -Expected: `20`. - -- [ ] **Step 2.4: Run the test module** - -Run: `uv run --no-sync pytest tests/test_middleware.py -q` -Expected: all tests pass (same set as before, exercising the same FakeConsumer path through the standard 0.7.1 entry). - -- [ ] **Step 2.5: Type-check the module** - -Run: `uv run ty check tests/test_middleware.py` -Expected: clean. No `union-attr` errors on `test_broker` attribute access — confirms the 0.7.1 `EnterType` overload binds `KafkaBroker` directly. - -- [ ] **Step 2.6: Commit** - -```bash -git add tests/test_middleware.py -git commit -m "$(cat <<'EOF' -refactor(tests): drop fake_test_broker helper in test_middleware - -Use TestKafkaBroker directly; faststream 0.7.1's EnterType overload -binds the single-broker call to KafkaBroker, so the helper's -isinstance narrowing is no longer needed. -EOF -)" -``` - ---- - -## Task 3: Inline `TestKafkaBroker` in `tests/test_healthcheck.py` - -**Files:** -- Modify: `tests/test_healthcheck.py` (imports + 4 callsites) - -- [ ] **Step 3.1: Swap the faststream import on line 4** - -Before: -```python -from faststream.kafka import KafkaBroker -``` - -After: -```python -from faststream.kafka import KafkaBroker, TestKafkaBroker -``` - -- [ ] **Step 3.2: Delete the mocks import on line 11** - -Before: -```python -from tests.mocks import fake_test_broker -``` - -Delete the entire line (it is the only `tests.mocks` import in this file). Preserve the surrounding blank lines per the file's existing import-block layout. - -- [ ] **Step 3.3: Replace every callsite (4 occurrences)** - -Use a single `Edit` with `replace_all=true`: -- `old_string`: `async with fake_test_broker(broker) as test_broker:` -- `new_string`: `async with TestKafkaBroker(broker) as test_broker:` - -Confirm afterward with: `grep -c "fake_test_broker" tests/test_healthcheck.py` -Expected: `0`. - -Confirm replacement count with: `grep -c "async with TestKafkaBroker(broker) as test_broker:" tests/test_healthcheck.py` -Expected: `4`. - -- [ ] **Step 3.4: Run the test module** - -Run: `uv run --no-sync pytest tests/test_healthcheck.py -q` -Expected: all 4 tests pass. - -- [ ] **Step 3.5: Type-check the module** - -Run: `uv run ty check tests/test_healthcheck.py` -Expected: clean. - -- [ ] **Step 3.6: Commit** - -```bash -git add tests/test_healthcheck.py -git commit -m "$(cat <<'EOF' -refactor(tests): drop fake_test_broker helper in test_healthcheck - -Use TestKafkaBroker directly; matches the test_middleware cleanup -under faststream 0.7.1's typed __aenter__. -EOF -)" -``` - ---- - -## Task 4: Delete the helper and its unused imports - -**Files:** -- Modify: `tests/mocks.py:1-7, 97-107` - -- [ ] **Step 4.1: Verify no remaining callers** - -Run: `grep -rn "fake_test_broker" tests/ faststream_concurrent_aiokafka/` -Expected: zero matches. If any callsite remains, stop and fix it (re-run Task 2 or Task 3 as appropriate) before proceeding. - -- [ ] **Step 4.2: Delete the helper function and its decorator** - -Edit `tests/mocks.py`. Delete the entire `fake_test_broker` definition — lines 97 through 107 in the current file: - -```python -@contextlib.asynccontextmanager -async def fake_test_broker(broker: KafkaBroker, *, connect_only: bool = False) -> typing.AsyncIterator[KafkaBroker]: - """TestKafkaBroker for a single broker, narrowed to KafkaBroker. - - faststream 0.7's TestKafkaBroker.__aenter__ returns Broker | list[Broker] - (variadic constructor). Every call site here passes one broker; this helper - keeps that invariant in one place so test bodies don't repeat the narrowing. - """ - async with TestKafkaBroker(broker, connect_only=connect_only) as test_broker: - assert isinstance(test_broker, KafkaBroker) - yield test_broker -``` - -Also delete the trailing blank line that separated this function from prior code, if it leaves a double blank at end of file. - -- [ ] **Step 4.3: Drop the now-unused imports** - -Edit `tests/mocks.py` lines 1–7. - -Before: -```python -"""Shared mock classes used across multiple test modules.""" - -import contextlib -import typing -from unittest.mock import AsyncMock, Mock - -from faststream.kafka import KafkaBroker, TestKafkaBroker -``` - -After: -```python -"""Shared mock classes used across multiple test modules.""" - -import typing -from unittest.mock import AsyncMock, Mock -``` - -`contextlib`, `KafkaBroker`, and `TestKafkaBroker` are only referenced by the deleted helper. `typing` and the `unittest.mock` imports are still used by `patched_message` and the `Mock*` classes. - -- [ ] **Step 4.4: Sanity-check the file** - -Run: `grep -n "contextlib\|TestKafkaBroker\|KafkaBroker\|fake_test_broker" tests/mocks.py` -Expected: zero matches. - -Run: `grep -c "patched_message\|MockKafkaMessage\|MockKafkaBatchCommitter\|MockAIOKafkaConsumer\|MockConsumerRecord\|MockAsyncioTask" tests/mocks.py` -Expected: a non-zero count (the survivors are still defined). - -- [ ] **Step 4.5: Run lint** - -Run: `just lint` -Expected: clean. Catches any stale formatting from the deletions and any unused-import lint that we may have missed. - -- [ ] **Step 4.6: Run the full local test suite (FakeConsumer paths)** - -Run: `uv run --no-sync pytest tests/test_middleware.py tests/test_healthcheck.py tests/test_kafka_committer.py tests/test_concurrent_processing.py tests/test_rebalance.py -q` -Expected: all pass. (We skip `tests/test_integration.py` here because it requires Docker/Redpanda — that runs in the final gate via `just test`.) - -- [ ] **Step 4.7: Commit** - -```bash -git add tests/mocks.py -git commit -m "$(cat <<'EOF' -refactor(tests): delete fake_test_broker helper - -No remaining callers after the test_middleware and test_healthcheck -inlining. faststream 0.7.1's overloaded TestKafkaBroker.__init__ -binds EnterType to KafkaBroker on single-broker calls, so the -isinstance narrowing the helper provided is dead weight. -EOF -)" -``` - ---- - -## Task 5: Final integration gate - -**Files:** none modified. - -- [ ] **Step 5.1: Run the full Docker-backed test suite** - -Run: `just test` -Expected: full suite passes — including `tests/test_integration.py`, which exercises a real Redpanda container. This confirms that nothing in the helper deletion regressed the integration path. - -- [ ] **Step 5.2: Confirm branch state** - -Run: `git log --oneline main..HEAD` -Expected: 5 commits in this order (top-down most recent first): -``` -refactor(tests): delete fake_test_broker helper -refactor(tests): drop fake_test_broker helper in test_healthcheck -refactor(tests): drop fake_test_broker helper in test_middleware -chore: bump faststream pin to >=0.7.1 -docs(spec): design for faststream 0.7.1 TestBroker typing alignment -``` - -Run: `git diff --stat main..HEAD` -Expected: changes confined to `pyproject.toml`, `uv.lock`, `tests/mocks.py`, `tests/test_middleware.py`, `tests/test_healthcheck.py`, and the two `planning/` documents. Nothing under `faststream_concurrent_aiokafka/`. - ---- - -## Notes for the executor - -- Do not skip Task 1's verify step. The helper's `isinstance` assert continues to hold on 0.7.1, so all 24 sites pass on the bumped pin before any test-file edits. If they don't, the regression is in the upstream bump, not in the refactor. -- Tasks 2 and 3 are independent of each other (touch different files), but Task 4 must run last — it deletes the helper. -- If `uv lock --upgrade-package faststream` produces no diff in `uv.lock`, it means the existing lock already pinned `faststream` to 0.7.1+. That's fine; commit the `pyproject.toml`-only change. -- The `replace_all=true` strategy in Tasks 2.3 and 3.3 is safe because the `async with fake_test_broker(setup_broker)` / `async with fake_test_broker(broker)` patterns are unique to those files (the helper is not used anywhere else, and the two patterns differ by the variable name). Confirm the `grep -c` counts to catch any partial replacement. diff --git a/planning/changes/2026-06-13.01-robustness-docs-test-audit/design.md b/planning/changes/2026-06-13.01-robustness-docs-test-audit.md similarity index 100% rename from planning/changes/2026-06-13.01-robustness-docs-test-audit/design.md rename to planning/changes/2026-06-13.01-robustness-docs-test-audit.md diff --git a/planning/changes/2026-06-13.01-robustness-docs-test-audit/plan.md b/planning/changes/2026-06-13.01-robustness-docs-test-audit/plan.md deleted file mode 100644 index 4e149a2..0000000 --- a/planning/changes/2026-06-13.01-robustness-docs-test-audit/plan.md +++ /dev/null @@ -1,966 +0,0 @@ -# Robustness, Docs, and Test Audit — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Fix three production-impacting behavioral risks (rebalance-flush hang, batch-subscriber crash, unbounded memory growth) plus documentation drift, a misleading test, and code cleanup surfaced by a full audit of `faststream-concurrent-aiokafka`. - -**Architecture:** Small, mostly-independent changes to an asyncio Kafka-offset-committing middleware. Behavioral fixes add a bounded timeout to the rebalance flush, an explicit batch-subscriber guard, and an opt-in backpressure ceiling on uncommitted tasks. A final pure-refactor extracts the committer's pending/watermark state into its own unit. The at-least-once contract is preserved throughout. - -**Tech Stack:** Python ≥3.11, asyncio, FastStream, aiokafka, pytest + pytest-asyncio (`asyncio_mode=auto`), pytest-cov (**100% coverage enforced**), pytest-xdist (`-n auto`), ruff + ty. - ---- - -## Conventions for every task - -- **Imports at module level** (project rule — no local imports inside functions). -- **No `from __future__ import annotations`**; use `typing.Self`/`typing.Never` directly. -- **Type suppression** uses `# ty: ignore[rule-name]`, never `# type: ignore`. -- **Run a single test during TDD** (coverage gate disabled so the file-scoped run doesn't fail on <100%): - `uv run --no-sync pytest tests/test_X.py::test_name -p no:cacheprovider --no-cov -v` -- **Validate the full unit suite + coverage** before each commit that touches `faststream_concurrent_aiokafka/`: - `uv run --no-sync pytest tests/ --ignore=tests/test_integration.py` - (Integration tests need a real broker; run them via `just test` only where a task says so.) -- **Lint before each commit:** `just lint` -- **Commit messages** end with the project's `Co-Authored-By` trailer (see existing history). - ---- - -## File Structure - -| File | Responsibility | Tasks | -|---|---|---| -| `faststream_concurrent_aiokafka/consts.py` | Default constants | 3, 5 | -| `faststream_concurrent_aiokafka/batch_committer.py` | Committer loop, flush timeout, backpressure | 3, 5 | -| `faststream_concurrent_aiokafka/rebalance.py` | Rebalance listener; forwards flush timeout | 3 | -| `faststream_concurrent_aiokafka/processing.py` | Handler; forwards flush timeout; import cleanup | 3, 8 | -| `faststream_concurrent_aiokafka/middleware.py` | Batch guard; threads backpressure config | 2, 5 | -| `LICENSE` (new) | MIT license text | 1 | -| `README.md` | Docs corrections | 6 | -| `tests/test_middleware.py` | Batch-guard + backpressure-wiring tests | 2, 5 | -| `tests/test_kafka_committer.py` | Flush-timeout + backpressure-unit tests | 3, 5 | -| `tests/test_rebalance.py` | Flush-timeout forwarding test | 3 | -| `tests/test_integration.py` | Rewritten shutdown-cancels test | 7 | -| `faststream_concurrent_aiokafka/_pending_state.py` (new) | Extracted pending/watermark state | 9 | - ---- - -## Task 1: Trivial cleanups (LICENSE, wording, orphaned pyc, imports) - -**Files:** -- Create: `LICENSE` -- Modify: `README.md` (license link already present at line 24; API-reference wording near line 123) -- Modify: `faststream_concurrent_aiokafka/processing.py:8-9` -- Delete (local artifact): `faststream_concurrent_aiokafka/__pycache__/dead_letter_queue.cpython-314.pyc` - -These have no runtime behavior and no new test code. - -- [ ] **Step 1: Add the MIT LICENSE file (#5)** - -Create `LICENSE` with the standard MIT text: - -```text -MIT License - -Copyright (c) 2026 Artur Shiriev - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. -``` - -- [ ] **Step 2: Fix the "observer task" wording (#7)** - -In `README.md`, the `is_kafka_handler_healthy` entry (around line 123) reads: -`False otherwise (not initialized, stopped, or observer task dead)`. -Replace `observer task dead` with `committer task dead`: - -```markdown -### `is_kafka_handler_healthy(context)` - -Returns `True` if the `KafkaConcurrentHandler` stored in `context` is running and healthy, `False` otherwise (not initialized, stopped, or committer task dead). Useful for readiness/liveness probes. -``` - -- [ ] **Step 3: Unify imports in `processing.py` (#12)** - -`processing.py:8-9` currently is: - -```python -from faststream_concurrent_aiokafka import batch_committer, consts -from faststream_concurrent_aiokafka.batch_committer import KafkaBatchCommitter -``` - -`KafkaBatchCommitter` is used as a bare name in the type annotation/`__init__`; `batch_committer.KafkaCommitTask` and `batch_committer.CommitterIsDeadError` are used module-qualified. Keep both forms but order them per the project's isort config (already correct). No change is strictly required here unless ruff flags it — run `just lint` and accept its auto-fix. If ruff leaves it unchanged, leave lines 8-9 as-is (they are already consistent: one module import for qualified use, one symbol import for the annotated constructor arg). **This step is satisfied once `just lint` is clean.** - -- [ ] **Step 4: Delete the orphaned bytecode (#11)** - -```bash -rm -f faststream_concurrent_aiokafka/__pycache__/dead_letter_queue.cpython-314.pyc -grep -rn "dead_letter_queue" faststream_concurrent_aiokafka/ tests/ README.md CLAUDE.md --include='*.py' --include='*.md' -``` -Expected: the `grep` prints nothing (no dangling source references). `__pycache__` is already git-ignored (`.gitignore` has `__pycache__/*` and `*.pyc`), so nothing is tracked — this is a local-filesystem cleanup only. - -- [ ] **Step 5: Lint and commit** - -```bash -just lint -git add LICENSE README.md faststream_concurrent_aiokafka/processing.py -git commit -``` -Commit message: `docs: add LICENSE, fix healthcheck wording, tidy imports (#5,#7,#11,#12)` - ---- - -## Task 2: Reject batch subscribers with a clear error (#2) - -**Files:** -- Modify: `faststream_concurrent_aiokafka/middleware.py` (`consume_scope`, after the non-MANUAL pass-through near line 75) -- Test: `tests/test_middleware.py` - -A batch subscriber delivers `self.msg` as a `tuple` of `ConsumerRecord`s; the current `typing.cast("ConsumerRecord", self.msg)` then crashes with a bare `AttributeError` on `record.offset`. Reject early with a clear message. - -- [ ] **Step 1: Write the failing test** - -Add to `tests/test_middleware.py` (uses the existing `setup_broker` fixture and `patched_message` helper): - -```python -async def test_middleware_batch_subscriber_rejected(setup_broker: KafkaBroker) -> None: - """A batch subscriber (self.msg is a tuple/list of records) is rejected with a clear error.""" - - @setup_broker.subscriber("batch-reject-topic", group_id="batch-reject-group") - async def handler(msg: typing.Any) -> None: ... - - async with TestKafkaBroker(setup_broker) as test_broker: - await initialize_concurrent_processing( - context=test_broker.context, commit_batch_size=10, commit_batch_timeout_sec=5 - ) - - # MANUAL-ack mock so we reach the batch check; raw message is a tuple of records. - mock_msg: typing.Final = MagicMock() - mock_msg.committed = None # MANUAL ack path - mock_msg.consumer._enable_auto_commit = False - - # Force self.msg (the raw middleware message) to look like a batch. - with ( - patched_message(test_broker, mock_msg), - patch.object(KafkaConcurrentProcessingMiddleware, "msg", ({"a": 1}, {"b": 2}), create=True), - pytest.raises(RuntimeError, match="does not support batch subscribers"), - ): - await test_broker.publish({"id": 1}, topic="batch-reject-topic") - - await asyncio.sleep(0) - await stop_concurrent_processing(test_broker.context) -``` - -> Note: `BaseMiddleware` stores the raw message as `self.msg`. The `patch.object(..., "msg", ...)` sets a class-level tuple so `consume_scope` sees a batch-shaped raw message regardless of what FastStream passed. If `self.msg` is set per-instance and the class patch does not take effect during `publish`, fall back to asserting via a direct unit call: construct the middleware, set `middleware.msg = (record_a, record_b)`, and `await middleware.consume_scope(call_next, mock_msg)` — but try the class-patch form first. - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `uv run --no-sync pytest tests/test_middleware.py::test_middleware_batch_subscriber_rejected -p no:cacheprovider --no-cov -v` -Expected: FAIL — currently raises `AttributeError` (not `RuntimeError`), or the message does not match `does not support batch subscribers`. - -- [ ] **Step 3: Add the guard in `consume_scope`** - -In `faststream_concurrent_aiokafka/middleware.py`, insert the batch check **immediately after** the non-MANUAL pass-through block (after `if kafka_message.committed is not None: return await call_next(msg)`, around line 75) and **before** the `if not concurrent_processing:` check: - -```python - if isinstance(self.msg, (list, tuple)): - err = ( - "KafkaConcurrentProcessingMiddleware does not support batch subscribers (batch=True). " - "Use a non-batch subscriber, or remove the middleware from this subscriber." - ) - raise RuntimeError(err) -``` - -Rationale for placement: the FakeConsumer short-circuit and non-MANUAL pass-through run first (so `TestKafkaBroker` and auto-ack subscribers are unaffected); the guard fires only on the MANUAL-ack concurrent path. - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `uv run --no-sync pytest tests/test_middleware.py::test_middleware_batch_subscriber_rejected -p no:cacheprovider --no-cov -v` -Expected: PASS - -- [ ] **Step 5: Run the full unit suite + lint** - -Run: `uv run --no-sync pytest tests/ --ignore=tests/test_integration.py` then `just lint` -Expected: all pass, coverage 100%. - -- [ ] **Step 6: Commit** - -```bash -git add faststream_concurrent_aiokafka/middleware.py tests/test_middleware.py -git commit -``` -Commit message: `fix: reject batch subscribers with a clear error (#2)` - ---- - -## Task 3: Bounded rebalance flush timeout (#1) - -**Files:** -- Modify: `faststream_concurrent_aiokafka/consts.py` (new constant) -- Modify: `faststream_concurrent_aiokafka/batch_committer.py` (`commit_all`) -- Modify: `faststream_concurrent_aiokafka/rebalance.py` (`__init__`, `on_partitions_revoked`) -- Modify: `faststream_concurrent_aiokafka/processing.py` (`create_rebalance_listener`) -- Test: `tests/test_kafka_committer.py`, `tests/test_rebalance.py` - -- [ ] **Step 1: Add the constant** - -In `faststream_concurrent_aiokafka/consts.py`, add (keeping the `typing.Final` style): - -```python -DEFAULT_REBALANCE_FLUSH_TIMEOUT_SEC: typing.Final = 10.0 -``` - -- [ ] **Step 2: Write the failing committer test (timeout path)** - -Add to `tests/test_kafka_committer.py`: - -```python -async def test_commit_all_times_out_on_hung_handler(caplog: pytest.LogCaptureFixture) -> None: - """commit_all returns within flush_timeout_sec even if an in-flight task never completes.""" - caplog.set_level(logging.WARNING) - consumer: typing.Final = MockAIOKafkaConsumer() - committer: typing.Final = KafkaBatchCommitter(commit_batch_timeout_sec=10.0, commit_batch_size=100) - committer.spawn() - - async def hangs() -> None: - await asyncio.sleep(30) - - hung_task: typing.Final = asyncio.create_task(hangs()) - await committer.send_task( - KafkaCommitTask( - asyncio_task=hung_task, - offset=1, - consumer=consumer, - topic_partition=TopicPartition(topic="t", partition=0), - ) - ) - - loop: typing.Final = asyncio.get_running_loop() - started: typing.Final = loop.time() - await committer.commit_all(flush_timeout_sec=0.1) - elapsed: typing.Final = loop.time() - started - - assert elapsed < 1.0, f"commit_all blocked on the hung task ({elapsed:.2f}s)" - assert "flush timed out" in caplog.text - assert committer.is_healthy # loop still running after a timed-out flush - - hung_task.cancel() - with contextlib.suppress(asyncio.CancelledError): - await hung_task - await committer.close() -``` - -- [ ] **Step 3: Run it to verify it fails** - -Run: `uv run --no-sync pytest tests/test_kafka_committer.py::test_commit_all_times_out_on_hung_handler -p no:cacheprovider --no-cov -v` -Expected: FAIL — `commit_all()` takes no `flush_timeout_sec` argument (TypeError), or blocks ~30 s. - -- [ ] **Step 4: Implement the timeout in `commit_all`** - -In `faststream_concurrent_aiokafka/batch_committer.py`, replace `commit_all`: - -```python - async def commit_all( - self, flush_timeout_sec: float = consts.DEFAULT_REBALANCE_FLUSH_TIMEOUT_SEC - ) -> None: - """Flush and commit pending tasks without stopping the committer loop. - - Bounded by ``flush_timeout_sec``: on timeout, already-completed offsets are - committed and any still-in-flight tasks stay uncommitted (redelivered after - reassignment — at-least-once). Safe to call during rebalance - (on_partitions_revoked); the committer keeps running after this returns. - """ - self._flush_batch_event.set() - try: - await asyncio.wait_for(self._messages_queue.join(), timeout=flush_timeout_sec) - except TimeoutError: - logger.warning( - "Kafka middleware. commit_all flush timed out after %.1fs; " - "in-flight offsets will be redelivered on restart/reassignment", - flush_timeout_sec, - ) -``` - -(`consts` is already imported in `batch_committer.py`.) - -- [ ] **Step 5: Run the committer test to verify it passes** - -Run: `uv run --no-sync pytest tests/test_kafka_committer.py::test_commit_all_times_out_on_hung_handler -p no:cacheprovider --no-cov -v` -Expected: PASS - -- [ ] **Step 6: Write the failing rebalance-forwarding test** - -Add to `tests/test_rebalance.py`: - -```python -async def test_rebalance_forwards_flush_timeout(committer: MockKafkaBatchCommitter) -> None: - """The listener forwards its configured flush timeout to commit_all.""" - listener: typing.Final = ConsumerRebalanceListener(committer, flush_timeout_sec=2.5) # ty: ignore[invalid-argument-type] - await listener.on_partitions_revoked(set()) - committer.commit_all.assert_called_once_with(2.5) -``` - -> The existing `test_rebalance_on_partitions_revoked_calls_commit_all` asserts `commit_all` was called once with no specific args; `assert_called_once()` still passes when an argument is forwarded, so it needs no change. - -- [ ] **Step 7: Run it to verify it fails** - -Run: `uv run --no-sync pytest tests/test_rebalance.py::test_rebalance_forwards_flush_timeout -p no:cacheprovider --no-cov -v` -Expected: FAIL — `ConsumerRebalanceListener.__init__` takes no `flush_timeout_sec`. - -- [ ] **Step 8: Wire the timeout through the listener** - -In `faststream_concurrent_aiokafka/rebalance.py`, add the import and update `__init__` / `on_partitions_revoked`: - -```python -from faststream_concurrent_aiokafka import consts -from faststream_concurrent_aiokafka.batch_committer import KafkaBatchCommitter -``` - -```python - def __init__( - self, - committer: KafkaBatchCommitter, - flush_timeout_sec: float = consts.DEFAULT_REBALANCE_FLUSH_TIMEOUT_SEC, - ) -> None: - self._committer = committer - self._flush_timeout_sec = flush_timeout_sec -``` - -```python - async def on_partitions_revoked(self, revoked: object) -> None: - await self._committer.commit_all(self._flush_timeout_sec) - self._committer.clear_cancellation_watermarks( - typing.cast("typing.Iterable[TopicPartition]", revoked) - ) -``` - -- [ ] **Step 9: Forward the timeout from the handler** - -In `faststream_concurrent_aiokafka/processing.py`, update `create_rebalance_listener`: - -```python - def create_rebalance_listener( - self, flush_timeout_sec: float = consts.DEFAULT_REBALANCE_FLUSH_TIMEOUT_SEC - ) -> ConsumerRebalanceListener: - """Return a ConsumerRebalanceListener that flushes pending commits on partition revocation. - - ``flush_timeout_sec`` bounds how long the revoke callback waits for in-flight - handlers to finish before returning (keeps a slow handler from stalling the - rebalance past max.poll.interval.ms). Pass the returned listener to - ``@broker.subscriber(..., listener=listener)``. - """ - return ConsumerRebalanceListener(self._committer, flush_timeout_sec) -``` - -- [ ] **Step 10: Run rebalance + committer tests to verify pass** - -Run: `uv run --no-sync pytest tests/test_rebalance.py tests/test_kafka_committer.py -p no:cacheprovider --no-cov -v` -Expected: PASS (including the unchanged `test_rebalance_on_partitions_revoked_calls_commit_all`). - -- [ ] **Step 11: Full unit suite + lint, then commit** - -Run: `uv run --no-sync pytest tests/ --ignore=tests/test_integration.py` then `just lint` -Expected: all pass, 100% coverage. - -```bash -git add faststream_concurrent_aiokafka/consts.py faststream_concurrent_aiokafka/batch_committer.py \ - faststream_concurrent_aiokafka/rebalance.py faststream_concurrent_aiokafka/processing.py \ - tests/test_kafka_committer.py tests/test_rebalance.py -git commit -``` -Commit message: `fix: bound rebalance flush with a timeout (#1)` - ---- - -## Task 4: Backpressure ceiling on uncommitted tasks (#3) — committer core - -**Files:** -- Modify: `faststream_concurrent_aiokafka/consts.py` (new constant) -- Modify: `faststream_concurrent_aiokafka/batch_committer.py` (`__init__`, `send_task`, `_call_committer`, `_commit_partitions`, `_run_commit_process` finally) -- Test: `tests/test_kafka_committer.py` - -**Accounting model.** `_uncommitted_count` counts tasks admitted via `send_task` but not yet finally committed or dropped. Increment on admission (`send_task`) and on re-queue (`_call_committer` transient-error branch, a re-admission). Decrement by `len(flat)` once per commit round (`_commit_partitions`). Net effect: a re-queued task is `+1` (re-queue) then `-1` (round) → stays counted; a committed/dropped task is `-1` → leaves. This mirrors the existing `queue.put`/`task_done` balance. - -- [ ] **Step 1: Add the constant** - -In `faststream_concurrent_aiokafka/consts.py`: - -```python -DEFAULT_MAX_UNCOMMITTED_TASKS: typing.Final = 10_000 -``` - -- [ ] **Step 2: Write the failing backpressure test** - -Add to `tests/test_kafka_committer.py`: - -```python -async def test_send_task_blocks_when_uncommitted_ceiling_reached() -> None: - """send_task blocks once the uncommitted ceiling is hit, then unblocks as commits drain it.""" - consumer: typing.Final = MockAIOKafkaConsumer() - # Ceiling of 2; commits stalled by a gate so nothing drains until we open it. - committer: typing.Final = KafkaBatchCommitter( - commit_batch_timeout_sec=10.0, commit_batch_size=100, max_uncommitted_tasks=2 - ) - commit_gate: typing.Final = asyncio.Event() - - async def gated_commit(_offsets: object) -> None: - await commit_gate.wait() - - consumer.commit.side_effect = gated_commit - committer.spawn() - - async def done() -> None: - return None - - tp: typing.Final = TopicPartition(topic="t", partition=0) - - async def send(offset: int) -> None: - await committer.send_task( - KafkaCommitTask( - asyncio_task=asyncio.create_task(done()), - offset=offset, - consumer=consumer, - topic_partition=tp, - ) - ) - - await send(1) - await send(2) # count now at ceiling (2) - - third: typing.Final = asyncio.create_task(send(3)) - await asyncio.sleep(0.05) - assert not third.done(), "send_task should block at the uncommitted ceiling" - - # Let the stalled commit complete → count drops → blocked send_task proceeds. - commit_gate.set() - await asyncio.wait_for(third, timeout=1.0) - assert third.done() - - await committer.close() - - -async def test_send_task_unbounded_when_ceiling_is_none() -> None: - """max_uncommitted_tasks=None preserves unbounded admission (no blocking).""" - consumer: typing.Final = MockAIOKafkaConsumer() - committer: typing.Final = KafkaBatchCommitter( - commit_batch_timeout_sec=10.0, commit_batch_size=100, max_uncommitted_tasks=None - ) - consumer.commit.side_effect = lambda _o: asyncio.sleep(30) # never drains - committer.spawn() - - async def done() -> None: - return None - - tp: typing.Final = TopicPartition(topic="t", partition=0) - for offset in range(5): - await asyncio.wait_for( - committer.send_task( - KafkaCommitTask( - asyncio_task=asyncio.create_task(done()), - offset=offset, - consumer=consumer, - topic_partition=tp, - ) - ), - timeout=1.0, - ) # never blocks - - committer._commit_task.cancel() # noqa: SLF001 - with contextlib.suppress(asyncio.CancelledError): - await committer._commit_task # noqa: SLF001 - - -async def test_send_task_unblocks_with_dead_committer_error() -> None: - """A send_task blocked on the ceiling raises CommitterIsDeadError if the loop dies.""" - consumer: typing.Final = MockAIOKafkaConsumer() - committer: typing.Final = KafkaBatchCommitter( - commit_batch_timeout_sec=10.0, commit_batch_size=100, max_uncommitted_tasks=1 - ) - consumer.commit.side_effect = lambda _o: asyncio.sleep(30) # never drains - committer.spawn() - - async def done() -> None: - return None - - tp: typing.Final = TopicPartition(topic="t", partition=0) - await committer.send_task( - KafkaCommitTask( - asyncio_task=asyncio.create_task(done()), - offset=1, - consumer=consumer, - topic_partition=tp, - ) - ) # count now at ceiling (1) - - blocked: typing.Final = asyncio.create_task( - committer.send_task( - KafkaCommitTask( - asyncio_task=asyncio.create_task(done()), - offset=2, - consumer=consumer, - topic_partition=tp, - ) - ) - ) - await asyncio.sleep(0.05) - assert not blocked.done() - - committer._commit_task.cancel() # noqa: SLF001 - with contextlib.suppress(asyncio.CancelledError): - await committer._commit_task # noqa: SLF001 - - with pytest.raises(CommitterIsDeadError): - await asyncio.wait_for(blocked, timeout=1.0) -``` - -- [ ] **Step 3: Run the new tests to verify they fail** - -Run: `uv run --no-sync pytest tests/test_kafka_committer.py -k uncommitted -p no:cacheprovider --no-cov -v` -Expected: FAIL — `KafkaBatchCommitter.__init__` takes no `max_uncommitted_tasks`. - -- [ ] **Step 4: Add fields in `__init__`** - -In `faststream_concurrent_aiokafka/batch_committer.py`, extend the constructor signature and body: - -```python - def __init__( - self, - commit_batch_timeout_sec: float = consts.DEFAULT_COMMIT_BATCH_TIMEOUT_SEC, - commit_batch_size: int = consts.DEFAULT_COMMIT_BATCH_SIZE, - shutdown_timeout_sec: float = consts.DEFAULT_SHUTDOWN_TIMEOUT_SEC, - max_uncommitted_tasks: int | None = consts.DEFAULT_MAX_UNCOMMITTED_TASKS, - ) -> None: - ... # existing body unchanged - # Backpressure: count of tasks admitted via send_task but not yet finally - # committed/dropped (in _messages_queue + pending). When it reaches the - # ceiling, send_task blocks so the consume path stalls and the consumer stops - # fetching until commits catch up. None disables the bound. - self._max_uncommitted_tasks = max_uncommitted_tasks - self._uncommitted_count: int = 0 - # Set whenever the count drops (a commit round) or the loop exits; wakes a - # send_task blocked at the ceiling so it re-checks. - self._uncommitted_drained = asyncio.Event() - self._uncommitted_drained.set() -``` - -Add these two lines at the end of the constructor body (after the existing `_partition_owner` assignment). - -- [ ] **Step 5: Add the blocking + admission in `send_task`** - -Replace `send_task`: - -```python - async def send_task(self, new_task: KafkaCommitTask) -> None: - self._check_is_commit_task_running() - while ( - self._max_uncommitted_tasks is not None - and self._uncommitted_count >= self._max_uncommitted_tasks - ): - self._uncommitted_drained.clear() - # Re-check liveness before parking: if the loop died we must raise, not hang. - self._check_is_commit_task_running() - await self._uncommitted_drained.wait() - self._uncommitted_count += 1 - await self._messages_queue.put(new_task) -``` - -- [ ] **Step 6: Count re-queues in `_call_committer`** - -In the transient-`KafkaError` branch of `_call_committer`, count each re-admission: - -```python - except KafkaError: - # Transient error — re-queue batch for retry on next cycle - logger.exception("Error during commit to kafka, re-queuing batch") - for task in tasks_batch: - self._uncommitted_count += 1 - await self._messages_queue.put(task) - return False -``` - -- [ ] **Step 7: Decrement once per commit round in `_commit_partitions`** - -In `_commit_partitions`, after the `for _ in flat: self._messages_queue.task_done()` loop and before `return all(results)`, add: - -```python - self._uncommitted_count -= len(flat) - self._uncommitted_drained.set() -``` - -- [ ] **Step 8: Wake blocked producers when the loop exits** - -In `_run_commit_process`, the `finally` currently calls `state.cancel_outstanding()`. Add the drain-set so a blocked `send_task` wakes and hits the dead-committer check: - -```python - finally: - state.cancel_outstanding() - self._uncommitted_drained.set() -``` - -- [ ] **Step 9: Run the backpressure tests to verify they pass** - -Run: `uv run --no-sync pytest tests/test_kafka_committer.py -k uncommitted -p no:cacheprovider --no-cov -v` -Expected: PASS (all three). - -- [ ] **Step 10: Full unit suite + lint, then commit** - -Run: `uv run --no-sync pytest tests/ --ignore=tests/test_integration.py` then `just lint` -Expected: all pass, 100% coverage. (Existing committer tests still pass — the default ceiling of 10_000 never trips in those small tests.) - -```bash -git add faststream_concurrent_aiokafka/consts.py faststream_concurrent_aiokafka/batch_committer.py \ - tests/test_kafka_committer.py -git commit -``` -Commit message: `fix: add opt-in backpressure ceiling on uncommitted tasks (#3)` - ---- - -## Task 5: Thread `max_uncommitted_tasks` through initialization (#3 wiring) - -**Files:** -- Modify: `faststream_concurrent_aiokafka/middleware.py` (`initialize_concurrent_processing`) -- Test: `tests/test_middleware.py` - -- [ ] **Step 1: Write the failing wiring test** - -Add to `tests/test_middleware.py`: - -```python -async def test_middleware_initialize_passes_max_uncommitted_tasks(setup_broker: KafkaBroker) -> None: - """initialize_concurrent_processing forwards max_uncommitted_tasks to the committer.""" - async with TestKafkaBroker(setup_broker) as test_broker: - handler: typing.Final = await initialize_concurrent_processing( - context=test_broker.context, max_uncommitted_tasks=500 - ) - try: - assert handler._committer._max_uncommitted_tasks == 500 - finally: - await stop_concurrent_processing(test_broker.context) -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `uv run --no-sync pytest tests/test_middleware.py::test_middleware_initialize_passes_max_uncommitted_tasks -p no:cacheprovider --no-cov -v` -Expected: FAIL — `initialize_concurrent_processing` takes no `max_uncommitted_tasks`. - -- [ ] **Step 3: Add the parameter** - -In `faststream_concurrent_aiokafka/middleware.py`, extend `initialize_concurrent_processing`: - -```python -async def initialize_concurrent_processing( - context: ContextRepo, - concurrency_limit: int = consts.DEFAULT_CONCURRENCY_LIMIT, - commit_batch_size: int = consts.DEFAULT_COMMIT_BATCH_SIZE, - commit_batch_timeout_sec: float = consts.DEFAULT_COMMIT_BATCH_TIMEOUT_SEC, - shutdown_timeout_sec: float = consts.DEFAULT_SHUTDOWN_TIMEOUT_SEC, - max_uncommitted_tasks: int | None = consts.DEFAULT_MAX_UNCOMMITTED_TASKS, -) -> KafkaConcurrentHandler: -``` - -and pass it into the `KafkaBatchCommitter(...)` construction: - -```python - committer=KafkaBatchCommitter( - commit_batch_timeout_sec=commit_batch_timeout_sec, - commit_batch_size=commit_batch_size, - shutdown_timeout_sec=shutdown_timeout_sec, - max_uncommitted_tasks=max_uncommitted_tasks, - ), -``` - -- [ ] **Step 4: Run it to verify it passes** - -Run: `uv run --no-sync pytest tests/test_middleware.py::test_middleware_initialize_passes_max_uncommitted_tasks -p no:cacheprovider --no-cov -v` -Expected: PASS - -- [ ] **Step 5: Full unit suite + lint, then commit** - -Run: `uv run --no-sync pytest tests/ --ignore=tests/test_integration.py` then `just lint` - -```bash -git add faststream_concurrent_aiokafka/middleware.py tests/test_middleware.py -git commit -``` -Commit message: `feat: expose max_uncommitted_tasks via initialize_concurrent_processing (#3)` - ---- - -## Task 6: Update README for shipped behavior (#4, #6) - -**Files:** -- Modify: `README.md` - -No tests (docs only). Make the docs match the code shipped in Tasks 2–5. - -- [ ] **Step 1: Rewrite §Core Concepts → KafkaConcurrentHandler (#4)** - -In `README.md`, replace the `### KafkaConcurrentHandler` bullet list (currently claiming "counter + `asyncio.Event`" and "Signal handlers for graceful shutdown") with: - -```markdown -### KafkaConcurrentHandler - -The processing engine. Manages: -- An `asyncio.Semaphore` to enforce `concurrency_limit` -- In-flight task tracking via a `set[asyncio.Task]`; each task's done-callback releases the semaphore, removes the task from the set, and logs any non-cancellation exception -- A `KafkaBatchCommitter` for offset commits -- An optional `ConsumerRebalanceListener` (via `handler.create_rebalance_listener()`) that flushes pending commits when partitions are revoked - -This library does **not** install signal handlers — shutdown is driven by your lifespan / process manager calling `stop_concurrent_processing`. -``` - -- [ ] **Step 2: Document the rebalance flush cost + timeout (#6)** - -In `README.md` "How It Works", update the rebalance step (4) to note the bounded wait: - -```markdown -4. **Rebalance handling**: When Kafka revokes a partition, the `ConsumerRebalanceListener` (returned by `handler.create_rebalance_listener(flush_timeout_sec=...)`) calls `committer.commit_all()` to flush pending offsets before the partition is reassigned. The flush waits for in-flight handlers up to `flush_timeout_sec` (default 10 s) so a slow handler cannot stall the rebalance past `max.poll.interval.ms`; on timeout, the remaining in-flight messages are redelivered after reassignment (at-least-once). A future optimization may scope the wait to only the revoked partitions. -``` - -Also add `max_uncommitted_tasks` to the `initialize_concurrent_processing` parameter table: - -```markdown -| `max_uncommitted_tasks` | `10000` | Max tasks accepted but not yet committed before the consume path blocks (backpressure). `None` disables the bound. | -``` - -- [ ] **Step 3: Commit** - -```bash -git add README.md -git commit -``` -Commit message: `docs: align README with shipped flush-timeout and backpressure behavior (#4,#6)` - ---- - -## Task 7: Rewrite the misleading shutdown integration test (#8) - -**Files:** -- Modify: `tests/test_integration.py` (`test_real_kafka_graceful_shutdown_waits_for_tasks`, lines ~266-289) - -This test asserts shutdown "waits for in-flight tasks," but `stop()` **cancels** them. Rewrite it to assert cancellation + at-least-once redelivery. **Requires Docker** — validate via `just test`. - -- [ ] **Step 1: Replace the test** - -In `tests/test_integration.py`, replace `test_real_kafka_graceful_shutdown_waits_for_tasks` with a two-phase cancel/replay test (mirrors the structure of `test_real_kafka_multi_subscriber_commits_all_offsets`): - -```python -async def test_real_kafka_shutdown_cancels_in_flight_tasks(kafka_bootstrap_servers: str) -> None: - """stop_concurrent_processing cancels in-flight handlers; their offsets are redelivered (at-least-once).""" - topic: typing.Final = _topic("shutdown-cancel") - group: typing.Final = f"shutdown-cancel-group-{uuid.uuid4().hex[:6]}" - await _create_topic(kafka_bootstrap_servers, topic) - - # Phase 1: dispatch a handler that blocks; stop() while it is genuinely in-flight. - started: typing.Final = asyncio.Event() - cancelled_seen: typing.Final[list[bool]] = [] - completed_phase1: typing.Final[list[int]] = [] - - broker1: typing.Final = _broker(kafka_bootstrap_servers) - - @broker1.subscriber(topic, group_id=group, auto_offset_reset="earliest", ack_policy=AckPolicy.MANUAL) - async def handler1(msg: dict[str, int]) -> None: - started.set() - try: - await asyncio.sleep(30) # far longer than the test window - completed_phase1.append(msg["id"]) - except asyncio.CancelledError: - cancelled_seen.append(True) - raise - - async with broker1: - await broker1.start() - await initialize_concurrent_processing( - context=broker1.context, commit_batch_size=10, commit_batch_timeout_sec=5, concurrency_limit=5 - ) - await asyncio.sleep(CONSUMER_READY_SLEEP) - await broker1.publish({"id": 1}, topic=topic) - await asyncio.wait_for(started.wait(), timeout=POLL_SLEEP) - # Stop while the handler is still sleeping → it must be cancelled, not awaited. - await stop_concurrent_processing(broker1.context) - - assert cancelled_seen == [True], "in-flight handler was not cancelled on stop" - assert completed_phase1 == [], "handler completed despite shutdown cancellation" - - # Phase 2: restart with the same group id → the uncommitted message is redelivered. - replayed: typing.Final[list[int]] = [] - broker2: typing.Final = _broker(kafka_bootstrap_servers) - - @broker2.subscriber(topic, group_id=group, auto_offset_reset="earliest", ack_policy=AckPolicy.MANUAL) - async def handler2(msg: dict[str, int]) -> None: - replayed.append(msg["id"]) - - async with broker2: - await broker2.start() - await initialize_concurrent_processing( - context=broker2.context, commit_batch_size=10, commit_batch_timeout_sec=2, concurrency_limit=5 - ) - await asyncio.sleep(CONSUMER_READY_SLEEP) - try: - await asyncio.sleep(POLL_SLEEP) - assert replayed == [1], f"cancelled message was not redelivered: {replayed}" - finally: - await stop_concurrent_processing(broker2.context) -``` - -- [ ] **Step 2: Run the integration suite (Docker)** - -Run: `just test tests/test_integration.py::test_real_kafka_shutdown_cancels_in_flight_tasks` -Expected: PASS (Redpanda starts, the handler is cancelled in phase 1, message 1 replays in phase 2). - -> If `just test` runs the whole file, that is fine; ensure the new test passes and no others regress. - -- [ ] **Step 3: Commit** - -```bash -git add tests/test_integration.py -git commit -``` -Commit message: `test: assert shutdown cancels in-flight tasks and redelivers (#8)` - ---- - -## Task 8: Extract pending/watermark state from `batch_committer.py` (#10) - -**Files:** -- Create: `faststream_concurrent_aiokafka/_pending_state.py` -- Modify: `faststream_concurrent_aiokafka/batch_committer.py` -- (No new tests — existing `tests/test_kafka_committer.py` is the safety net.) - -**This task is a pure refactor: no behavior change.** It lands LAST, on top of green tests from Tasks 2–7. The goal is to move the per-partition pending bookkeeping, offset mapping, and cancellation watermarks into a focused unit, leaving `KafkaBatchCommitter` as the loop driver. - -> **Execution note:** Because the existing committer tests call several of these as `KafkaBatchCommitter._map_offsets_per_partition(...)`, `_extract_ready_prefixes(...)`, and `_insert_sorted(...)` directly (see `tests/test_kafka_committer.py`), do the extraction **without breaking those call sites**: keep thin static delegators on `KafkaBatchCommitter` (or update the tests in the same commit). The lowest-risk path that keeps tests untouched is to keep the static methods on `KafkaBatchCommitter` as one-line delegators to the new module. Verify by running the full committer suite unchanged. - -- [ ] **Step 1: Create `_pending_state.py` with the moved pure functions** - -Create `faststream_concurrent_aiokafka/_pending_state.py` and move the bodies of `_insert_sorted` (currently module-level in `batch_committer.py`), `_extract_ready_prefixes`, and `_map_offsets_per_partition` (currently static methods) into it verbatim, plus the `_OFFSET_KEY` helper: - -```python -import bisect -import operator -import typing - -from faststream.kafka import TopicPartition - -from faststream_concurrent_aiokafka.batch_committer import KafkaCommitTask # see Step 3 note - - -_OFFSET_KEY: typing.Final = operator.attrgetter("offset") - - -def insert_sorted(partition_pending: list["KafkaCommitTask"], new_ct: "KafkaCommitTask") -> None: - # (verbatim body from batch_committer._insert_sorted) - ... - - -def extract_ready_prefixes( - pending: dict[TopicPartition, list["KafkaCommitTask"]], -) -> tuple[dict[TopicPartition, list["KafkaCommitTask"]], int]: - # (verbatim body from KafkaBatchCommitter._extract_ready_prefixes) - ... - - -def map_offsets_per_partition( - consumer_id: int, - consumer_tasks: list["KafkaCommitTask"], - watermarks: dict[tuple[int, TopicPartition], int], -) -> dict[TopicPartition, int]: - # (verbatim body from KafkaBatchCommitter._map_offsets_per_partition) - ... -``` - -> **Circular-import note (Step 3):** `KafkaCommitTask` lives in `batch_committer.py`. To avoid a cycle, either (a) move the `KafkaCommitTask` dataclass into `_pending_state.py` and import it back into `batch_committer.py`, or (b) keep `KafkaCommitTask` in `batch_committer.py` and use a string/`TYPE_CHECKING` import in `_pending_state.py` (the functions only read `.offset`, `.topic_partition`, `.asyncio_task`, so no runtime import of the class is needed). Prefer (a) — `KafkaCommitTask` is the state's natural home — but (b) is acceptable if it keeps the diff smaller. - -- [ ] **Step 2: Replace the originals in `batch_committer.py` with delegators** - -In `batch_committer.py`, replace the three moved definitions with thin delegators so existing test call sites keep working: - -```python -from faststream_concurrent_aiokafka import _pending_state - - -def _insert_sorted(partition_pending: list[KafkaCommitTask], new_ct: KafkaCommitTask) -> None: - _pending_state.insert_sorted(partition_pending, new_ct) - - -class KafkaBatchCommitter: - @staticmethod - def _extract_ready_prefixes( - pending: dict[TopicPartition, list[KafkaCommitTask]], - ) -> tuple[dict[TopicPartition, list[KafkaCommitTask]], int]: - return _pending_state.extract_ready_prefixes(pending) - - @staticmethod - def _map_offsets_per_partition( - consumer_id: int, - consumer_tasks: list[KafkaCommitTask], - watermarks: dict[tuple[int, TopicPartition], int], - ) -> dict[TopicPartition, int]: - return _pending_state.map_offsets_per_partition(consumer_id, consumer_tasks, watermarks) -``` - -Update the loop's internal calls (`_insert_sorted(...)`, `self._extract_ready_prefixes(...)`, `self._map_offsets_per_partition(...)`) to keep referencing these names — no change needed at call sites since the delegators preserve the signatures. - -- [ ] **Step 3: Run the FULL committer suite unchanged to prove no behavior change** - -Run: `uv run --no-sync pytest tests/test_kafka_committer.py -p no:cacheprovider --no-cov -v` -Expected: PASS — every existing test passes without modification (the delegators preserve the public-to-tests surface). - -- [ ] **Step 4: Full unit suite + lint** - -Run: `uv run --no-sync pytest tests/ --ignore=tests/test_integration.py` then `just lint` -Expected: all pass, 100% coverage. (`ty check` must be clean — watch the `KafkaCommitTask` import direction from Step 1's note.) - -- [ ] **Step 5: Commit** - -```bash -git add faststream_concurrent_aiokafka/_pending_state.py faststream_concurrent_aiokafka/batch_committer.py -git commit -``` -Commit message: `refactor: extract pending/watermark helpers into _pending_state (#10)` - ---- - -## Task 9: Final verification - -- [ ] **Step 1: Run the complete suite in Docker (unit + integration + coverage)** - -Run: `just test` -Expected: all tests pass; coverage 100%; no lint failures in `lint-ci` if invoked. - -- [ ] **Step 2: Lint-CI parity check** - -Run: `just lint-ci` -Expected: no changes needed (formatting, ruff, ty all clean). - -- [ ] **Step 3: Confirm the branch is ready** - -```bash -git log --oneline main..HEAD -git status -``` -Expected: one commit per task, clean working tree. Hand off to `superpowers:finishing-a-development-branch`. - ---- - -## Self-Review notes (already applied) - -- **Spec coverage:** #1 → Task 3; #2 → Task 2; #3 → Tasks 4–5; #4,#6 → Task 6; #5,#7 → Task 1; #8 → Task 7; #9(a) → Task 3; #9(b) → Task 2; #9(c) → Task 4; #10 → Task 8; #11,#12 → Task 1. All twelve findings mapped. -- **Defaults:** `DEFAULT_REBALANCE_FLUSH_TIMEOUT_SEC = 10.0`, `DEFAULT_MAX_UNCOMMITTED_TASKS = 10_000` (matching the spec table). -- **Type consistency:** `flush_timeout_sec` and `max_uncommitted_tasks` names used identically across committer, listener, handler, and middleware. The backpressure accounting (`_uncommitted_count`, `_uncommitted_drained`) increments/decrements are symmetric with the existing `queue.put`/`task_done` balance. -- **Coverage gate:** every code-adding task pairs the change with a test in the same commit; the timeout branch, the three backpressure branches, and the batch-guard branch each have an explicit covering test. diff --git a/planning/changes/2026-06-13.02-codify-release-notes/design.md b/planning/changes/2026-06-13.02-codify-release-notes.md similarity index 100% rename from planning/changes/2026-06-13.02-codify-release-notes/design.md rename to planning/changes/2026-06-13.02-codify-release-notes.md diff --git a/planning/changes/2026-06-13.02-codify-release-notes/plan.md b/planning/changes/2026-06-13.02-codify-release-notes/plan.md deleted file mode 100644 index 5d4f2a4..0000000 --- a/planning/changes/2026-06-13.02-codify-release-notes/plan.md +++ /dev/null @@ -1,176 +0,0 @@ -# Codify `planning/releases/` as a Workflow Step — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the release-notes pattern shipped in `planning/releases/0.6.0.md` a documented, repeatable part of the project workflow. - -**Architecture:** Two docs-only changes — add a `planning/releases/TEMPLATE.md` skeleton that is descriptive of `0.6.0.md`'s existing structure, and extend the CLAUDE.md Workflow chain with a trailing release-notes step. No code, no automated tests; verification is by inspection. - -**Tech Stack:** Markdown only. - -Spec: [`design.md`](./design.md). - ---- - -### Task 1: Create the release-notes template - -**Files:** -- Create: `planning/releases/TEMPLATE.md` -- Reference (do not modify): `planning/releases/0.6.0.md` - -- [ ] **Step 1: Write `planning/releases/TEMPLATE.md`** - -Create the file with exactly this content: - -````markdown - - -# faststream-concurrent-aiokafka — - -** release. .** - -Spec: [`planning/specs/--design.md`](../specs/--design.md). - -## 1. (``) - -**** - -### The gap - - - -### The fix - - - - - -## Packaging - -- - -## Docs - -- - -## Tests - -- - -## Internal - -- - -## Upgrade notes - - - -## Release process - -Tag-and-release: publishing runs on a GitHub **release: published** event -(`.github/workflows/publish.yml` → `just publish`, which sets the version from -the tag name). Create a GitHub release with tag **``** (bare, no `v` -prefix) targeting `main`; the workflow builds and publishes to PyPI. -```` - -- [ ] **Step 2: Verify the template is a superset of `0.6.0.md`** - -Run: `grep -E '^(#|##|###) ' planning/releases/0.6.0.md` -Then: `grep -E '^(#|##|###) ' planning/releases/TEMPLATE.md` - -Expected: every heading shape present in `0.6.0.md` (title, `## N.` numbered -changes with `### The gap` / `### The fix`, `## Packaging`, `## Docs`, -`## Tests`, `## Internal`, `## Upgrade notes`, `## Release process`) has a -corresponding placeholder in the template. If any section in `0.6.0.md` is -missing from the template, add it before continuing. - -- [ ] **Step 3: Commit** - -```bash -git add planning/releases/TEMPLATE.md -git commit -m "docs(releases): add release-notes template" -``` - ---- - -### Task 2: Document the release-notes step in CLAUDE.md - -**Files:** -- Modify: `CLAUDE.md` (the `## Workflow` section) - -- [ ] **Step 1: Read the current Workflow section** - -Run: `grep -n -A 12 '^## Workflow' CLAUDE.md` -Expected: the per-feature chain ending in -`requesting-code-review → finishing-a-development-branch.` followed by the -"Topic slugs" paragraph. - -- [ ] **Step 2: Extend the workflow chain** - -In `CLAUDE.md`, replace this exact text: - -``` -executing-plans / subagent-driven-development → -requesting-code-review → finishing-a-development-branch. -``` - -with: - -``` -executing-plans / subagent-driven-development → -requesting-code-review → finishing-a-development-branch → -release notes in `planning/releases/.md`. -``` - -- [ ] **Step 3: Add the release-notes clarifying sentence** - -In `CLAUDE.md`, immediately after the "Topic slugs are kebab-case..." paragraph -in the `## Workflow` section, add this new paragraph: - -``` -Release notes are written when cutting a release (not per-feature): -copy `planning/releases/TEMPLATE.md` to `planning/releases/.md` -(bare version, no `v` prefix) and link back to the driving spec in -`planning/specs/`. -``` - -- [ ] **Step 4: Verify the edit reads cleanly** - -Run: `grep -n -A 16 '^## Workflow' CLAUDE.md` -Expected: the chain now ends with the `planning/releases/.md` step, -and the new clarifying paragraph appears below the Topic-slugs paragraph. No -duplicated or dangling arrows. - -- [ ] **Step 5: Commit** - -```bash -git add CLAUDE.md -git commit -m "docs: add release-notes step to the workflow" -``` - ---- - -## Self-Review - -- **Spec coverage:** - - Deliverable 1 (`planning/releases/TEMPLATE.md`) → Task 1. ✓ - - Deliverable 2 (CLAUDE.md Workflow edit + clarifying sentence) → Task 2. ✓ - - Non-goals (no backfill, no CI change, no specs/plans changes) → respected; - no task touches them. ✓ - - Verification (template superset of `0.6.0.md`; CLAUDE.md reads cleanly) → - Task 1 Step 2, Task 2 Step 4. ✓ -- **Placeholders:** The `<...>` markers in the template are intentional - authoring placeholders for future releases, not plan gaps; all plan steps - contain concrete content and exact commands. -- **Naming consistency:** `planning/releases/TEMPLATE.md` and - `planning/releases/.md` are referred to identically across both - tasks and the spec. diff --git a/planning/changes/2026-06-13.03-portable-planning-convention/design.md b/planning/changes/2026-06-13.03-portable-planning-convention.md similarity index 100% rename from planning/changes/2026-06-13.03-portable-planning-convention/design.md rename to planning/changes/2026-06-13.03-portable-planning-convention.md diff --git a/planning/changes/2026-06-13.03-portable-planning-convention/plan.md b/planning/changes/2026-06-13.03-portable-planning-convention/plan.md deleted file mode 100644 index 4a5545b..0000000 --- a/planning/changes/2026-06-13.03-portable-planning-convention/plan.md +++ /dev/null @@ -1,709 +0,0 @@ -# Portable planning convention — implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Migrate `faststream-concurrent-aiokafka`'s `planning/` to the portable -two-axis convention from `faststream-outbox` — `architecture/` truth home + -`planning/changes/` bundles — and seed the truth home. - -**Spec:** [`design.md`](./design.md) - -**Architecture:** Pure file moves + new docs. Create `architecture/` (4 seed -files), restructure `planning/` into `changes/{active,archive}/` bundles + -empty `audits/` + `retros/`, copy the portable README Conventions + templates -byte-identical from `faststream-outbox`, author a fresh Index, rewrite -`CLAUDE.md`'s `## Workflow` (preserving the #33 release-notes step), and remove -the `.gitignore` `plan.md` trap. No runtime/test/API code is touched. This repo -has **no mkdocs site**, so there is no `docs-build` step. - -**Tech Stack:** Markdown, `git mv`, `just lint-ci`. - -**Branch:** `docs/portable-planning-convention` (already created; `design.md` is -already committed there). - -**Source repo (copy-from):** `/Users/kevinsmith/src/pypi/faststream-outbox`. - -**Commit strategy:** Per-task commits. - -**`.NN` assignment (final, by merge order):** - -| Bundle id | design.md source | plan.md source | PR | -|-----------|------------------|----------------|----| -| `2026-06-03.01-faststream-0.7-migration` | `specs/2026-06-03-faststream-0.7-migration-design.md` | — (design-only) | #28 | -| `2026-06-04.01-faststream-0.7.1-testbroker-typing` | `specs/2026-06-04-faststream-0.7.1-testbroker-typing-design.md` | `plans/2026-06-04-faststream-0.7.1-testbroker-typing-plan.md` | #29 | -| `2026-06-13.01-robustness-docs-test-audit` | `specs/2026-06-13-robustness-docs-test-audit-design.md` | `plans/2026-06-13-robustness-docs-test-audit-plan.md` | #32 | -| `2026-06-13.02-codify-release-notes` | `specs/2026-06-13-codify-release-notes-design.md` | `plans/2026-06-13-codify-release-notes-plan.md` | #33 | - -All commands assume CWD `/Users/kevinsmith/src/pypi/faststream-concurrent-aiokafka`. - ---- - -### Task 1: Remove the `plan.md` ignore trap + scaffold the skeleton + copy templates - -**Files:** -- Modify: `.gitignore` (delete line 22, `plan.md`) -- Create: `planning/changes/active/.gitkeep`, `planning/audits/.gitkeep`, `planning/retros/.gitkeep` -- Create: `planning/_templates/{design,plan,change}.md` (copied byte-identical) -- Create: `planning/deferred.md` - -- [ ] **Step 1: Remove the stray `plan.md` ignore rule** - - `.gitignore:22` ignores the literal filename `plan.md` — it collides with - every change bundle's `plan.md` under the new convention. Delete the line: - - ```bash - sed -i '' '/^plan\.md$/d' .gitignore - git check-ignore planning/changes/active/2026-06-13.03-portable-planning-convention/plan.md && echo "STILL IGNORED (bad)" || echo "plan.md no longer ignored" - ``` - Expected: `plan.md no longer ignored`. - -- [ ] **Step 2: Create the directory skeleton** - - ```bash - mkdir -p planning/changes/active planning/audits planning/retros planning/_templates - touch planning/changes/active/.gitkeep planning/audits/.gitkeep planning/retros/.gitkeep - ``` - -- [ ] **Step 3: Copy the three templates byte-identical** - - ```bash - cp /Users/kevinsmith/src/pypi/faststream-outbox/planning/_templates/design.md planning/_templates/design.md - cp /Users/kevinsmith/src/pypi/faststream-outbox/planning/_templates/plan.md planning/_templates/plan.md - cp /Users/kevinsmith/src/pypi/faststream-outbox/planning/_templates/change.md planning/_templates/change.md - ``` - -- [ ] **Step 4: Verify templates are byte-identical** - - ```bash - diff /Users/kevinsmith/src/pypi/faststream-outbox/planning/_templates/design.md planning/_templates/design.md \ - && diff /Users/kevinsmith/src/pypi/faststream-outbox/planning/_templates/plan.md planning/_templates/plan.md \ - && diff /Users/kevinsmith/src/pypi/faststream-outbox/planning/_templates/change.md planning/_templates/change.md \ - && echo "TEMPLATES IDENTICAL" - ``` - Expected: `TEMPLATES IDENTICAL` (no diff output). - -- [ ] **Step 5: Create `planning/deferred.md`** - - Write `planning/deferred.md` with exactly this content: - - ```markdown - # Deferred Work - - Items raised in reviews or audits that are real but not actionable now. - Each is parked here with the reason it's deferred and the concrete trigger - that should bring it back. This is the long-tail register — not a backlog - of planned work. When an item is picked up it graduates to a change bundle - in [`changes/active/`](changes/active/); see [CLAUDE.md](../CLAUDE.md#workflow). - - ## Open - - _None._ - ``` - -- [ ] **Step 6: Commit** - - ```bash - git add .gitignore planning/changes planning/audits planning/retros planning/_templates planning/deferred.md - git commit -m "docs(planning): scaffold changes/audits/retros/_templates; drop plan.md ignore trap - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 2: Seed `architecture/` truth home (4 capability files) - -**Files:** -- Create: `architecture/concurrent-handler.md` -- Create: `architecture/batch-committer.md` -- Create: `architecture/middleware-lifecycle.md` -- Create: `architecture/rebalance.md` - -**Source to synthesize from:** `CLAUDE.md` `## Architecture` section (already in -this repo — read it before writing). Living prose, **no frontmatter** (dated by -git). Each file describes what the capability does *now* — invariants, not -change history. Aim for ~2–5 KB each, matching the granularity of -`faststream-outbox/architecture/*.md`. Use `file.py:line` references where -`CLAUDE.md` does. - -- [ ] **Step 1: Write `architecture/concurrent-handler.md`** - - Title `# Concurrent handler`. Cover (from `CLAUDE.md` → `processing.py — - KafkaConcurrentHandler`): - - One handler per `initialize_concurrent_processing`, stored in FastStream's - `ContextRepo` under key `"concurrent_processing"`; **not** a singleton — - `stop_concurrent_processing` clears the entry so a fresh handler can start. - - State it owns: `asyncio.Semaphore` (concurrency limit, minimum 1); the - `set[asyncio.Task]` `_tracked_tasks`; the per-task done-callback - `_finish_task` (releases the semaphore, removes the task from the set); a - `KafkaBatchCommitter`. - - `handle_task()` fires-and-forgets the user coroutine as an asyncio task and - enqueues a `KafkaCommitTask`; offsets are not committed until the user task - finishes (**at-least-once**). - - `stop()`: cancels every in-flight tracked task, then awaits - `committer.close()`; cancelled tasks are a hard offset boundary (cancelled + - after stay uncommitted → redelivered on restart); shutdown wall-clock bounded - by the committer's `shutdown_timeout_sec` (default 20 s), sub-second normally. - - No signal handlers — shutdown is driven by the FastStream lifespan calling - `stop_concurrent_processing`. - -- [ ] **Step 2: Write `architecture/batch-committer.md`** - - Title `# Batch committer`. Cover (from `CLAUDE.md` → `batch_committer.py — - KafkaBatchCommitter`): - - Runs as a background asyncio task (`spawn()`); streaming loop absorbs - `KafkaCommitTask`s into per-partition pending state. - - Commit triggers: total pending ≥ `commit_batch_size`, - `commit_batch_timeout_sec` fires, or `commit_all`/`close` sets the flush event. - - `_extract_ready_prefixes` sorts by offset (tolerates re-queued tasks landing - out of order), stops at the first not-done task; a cancelled task is a hard - boundary (cancelled + after dropped from pending); - `_map_offsets_per_partition` stops the offset advance at the cancelled task. - - Per consumer-id group: `consumer.commit({TopicPartition: max_offset+1})`. - - Error policy: transient `KafkaError` re-queues the batch; - `CommitFailedError` / `IllegalStateError` (rebalance/revocation) discards it; - `CommitterIsDeadError` is raised to callers when the main task has died → - triggers `handler.stop()`. - - `max_uncommitted_tasks` backpressure (default 10000, `None` disables): - `send_task` admits/blocks; the count releases once a task is committed or - dropped; hitting the ceiling also nudges a flush; keep - `max_uncommitted_tasks >= commit_batch_size`. - -- [ ] **Step 3: Write `architecture/middleware-lifecycle.md`** - - Title `# Middleware & lifecycle`. Cover (from `CLAUDE.md` → `middleware.py` and - `healthcheck.py`): - - `KafkaConcurrentProcessingMiddleware` (FastStream `BaseMiddleware`): - `consume_scope` retrieves the handler from `self.context`. Pass-through - cases: (a) FakeConsumer (`TestKafkaBroker`); (b) any subscriber whose ack - policy is not MANUAL (`kafka_message.committed is not None`). Refuses if - `_enable_auto_commit=True`. Batch raw messages (`isinstance(self.msg, (list, - tuple))`) raise a clear `RuntimeError` (batch subscribers unsupported), after - the pass-throughs. If the handler is stopped, logs a warning and skips the - message (offset stays uncommitted → redelivered on restart). - - `initialize_concurrent_processing(context, ...)`: create + start a handler, - store under `"concurrent_processing"`. - - `stop_concurrent_processing(context)`: gates on `is_running`; calls - `handler.stop()`; clears the context entry; safe when the committer task has - already died. - - `is_kafka_handler_healthy(context)` (`healthcheck.py`): `True` iff the - handler is present and `is_healthy` (`_is_running` AND committer task alive); - intended for readiness/liveness probes. - -- [ ] **Step 4: Write `architecture/rebalance.md`** - - Title `# Rebalance listener`. Cover (from `CLAUDE.md` → `rebalance.py` and the - `flush_timeout_sec` notes): - - `ConsumerRebalanceListener` returned by - `handler.create_rebalance_listener(flush_timeout_sec=...)`. - - On `on_partitions_revoked`, calls `committer.commit_all()` to flush completed - offsets before the partition is reassigned, preventing duplicate processing - after rebalance. - - The flush is bounded by `flush_timeout_sec` (default 10 s, comfortably under - aiokafka's default `max.poll.interval.ms` of 300 s); on timeout it logs a - warning and returns — already-completed offsets commit, in-flight stay - uncommitted (at-least-once; duplicates only on the timeout path). - -- [ ] **Step 5: Verify the four files exist and are non-empty** - - ```bash - wc -l architecture/*.md - ``` - Expected: four files, each with substantive line counts (not empty). - -- [ ] **Step 6: Commit** - - ```bash - git add architecture/ - git commit -m "docs(architecture): seed truth home (handler, committer, middleware, rebalance) - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 3: Migrate the four change bundles into `changes/` - -**Files:** four bundle folders under `planning/changes/`; `git mv` each -design (+ plan where present); prepend YAML frontmatter; fix the two plan `Spec` -links. - -- [ ] **Step 1: `git mv` the designs and plans into bundle folders** - - ```bash - # 2026-06-03.01 faststream-0.7-migration (design-only) - mkdir -p planning/changes/2026-06-03.01-faststream-0.7-migration - git mv planning/specs/2026-06-03-faststream-0.7-migration-design.md planning/changes/2026-06-03.01-faststream-0.7-migration/design.md - - # 2026-06-04.01 faststream-0.7.1-testbroker-typing - mkdir -p planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing - git mv planning/specs/2026-06-04-faststream-0.7.1-testbroker-typing-design.md planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/design.md - git mv planning/plans/2026-06-04-faststream-0.7.1-testbroker-typing-plan.md planning/changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md - - # 2026-06-13.01 robustness-docs-test-audit - mkdir -p planning/changes/2026-06-13.01-robustness-docs-test-audit - git mv planning/specs/2026-06-13-robustness-docs-test-audit-design.md planning/changes/2026-06-13.01-robustness-docs-test-audit/design.md - git mv planning/plans/2026-06-13-robustness-docs-test-audit-plan.md planning/changes/2026-06-13.01-robustness-docs-test-audit/plan.md - - # 2026-06-13.02 codify-release-notes - mkdir -p planning/changes/2026-06-13.02-codify-release-notes - git mv planning/specs/2026-06-13-codify-release-notes-design.md planning/changes/2026-06-13.02-codify-release-notes/design.md - git mv planning/plans/2026-06-13-codify-release-notes-plan.md planning/changes/2026-06-13.02-codify-release-notes/plan.md - ``` - -- [ ] **Step 2: Prepend frontmatter to each `design.md`** - - Each design currently opens with a prose `# Title`. Insert a YAML frontmatter - block at the very top (above the `#`), leaving the body intact. - - `…/2026-06-03.01-faststream-0.7-migration/design.md`: - ```yaml - --- - status: shipped - date: 2026-06-03 - slug: faststream-0.7-migration - supersedes: null - superseded_by: null - pr: "28" - outcome: "merged as #28 (plan removed post-execution; design-only bundle)" - --- - ``` - - `…/2026-06-04.01-faststream-0.7.1-testbroker-typing/design.md`: - ```yaml - --- - status: shipped - date: 2026-06-04 - slug: faststream-0.7.1-testbroker-typing - supersedes: null - superseded_by: null - pr: "29" - outcome: "merged as #29" - --- - ``` - - `…/2026-06-13.01-robustness-docs-test-audit/design.md`: - ```yaml - --- - status: shipped - date: 2026-06-13 - slug: robustness-docs-test-audit - supersedes: null - superseded_by: null - pr: "32" - outcome: "merged as #32; shipped in 0.6.0" - --- - ``` - - `…/2026-06-13.02-codify-release-notes/design.md`: - ```yaml - --- - status: shipped - date: 2026-06-13 - slug: codify-release-notes - supersedes: null - superseded_by: null - pr: "33" - outcome: "merged as #33" - --- - ``` - -- [ ] **Step 3: Prepend frontmatter to each `plan.md` (the three that exist)** - - `…/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md`: - ```yaml - --- - status: shipped - date: 2026-06-04 - slug: faststream-0.7.1-testbroker-typing - spec: faststream-0.7.1-testbroker-typing - pr: "29" - --- - ``` - - `…/2026-06-13.01-robustness-docs-test-audit/plan.md`: - ```yaml - --- - status: shipped - date: 2026-06-13 - slug: robustness-docs-test-audit - spec: robustness-docs-test-audit - pr: "32" - --- - ``` - - `…/2026-06-13.02-codify-release-notes/plan.md`: - ```yaml - --- - status: shipped - date: 2026-06-13 - slug: codify-release-notes - spec: codify-release-notes - pr: "33" - --- - ``` - -- [ ] **Step 4: Repoint the two plan `Spec` links to `./design.md`** - - Both migrated plans link their spec by the old `../specs/...` path. Fix each: - - In `…/2026-06-04.01-faststream-0.7.1-testbroker-typing/plan.md`, replace: - ``` - **Spec:** [`planning/specs/2026-06-04-faststream-0.7.1-testbroker-typing-design.md`](../specs/2026-06-04-faststream-0.7.1-testbroker-typing-design.md) - ``` - with: - ``` - **Spec:** [`design.md`](./design.md) - ``` - - In `…/2026-06-13.02-codify-release-notes/plan.md`, replace: - ``` - Spec: [`planning/specs/2026-06-13-codify-release-notes-design.md`](../specs/2026-06-13-codify-release-notes-design.md). - ``` - with: - ``` - Spec: [`design.md`](./design.md). - ``` - - (The robustness plan references its spec by section name, not a relative link — - no change. The `../specs/--design.md` placeholder on line 40 of the - codify plan is archived prose quoting the release template and is left - verbatim — see Non-goals.) - -- [ ] **Step 5: Verify frontmatter parses on all bundle design/plan files** - - ```bash - for f in planning/changes/*/design.md planning/changes/*/plan.md; do - uv run python -c "import sys,yaml; t=open('$f').read(); assert t.startswith('---'),'$f'; yaml.safe_load(t.split('---')[1]); print('OK','$f')" - done - ``` - Expected: `OK` for all seven files (four designs + three plans). - -- [ ] **Step 6: Commit** - - ```bash - git add planning/changes planning/specs planning/plans - git commit -m "docs(planning): migrate the four shipped changes into archive bundles - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 4: Remove the old dirs + repoint the live cross-links - -**Files:** -- Remove: `planning/plans/.gitkeep`, now-empty `planning/specs/`, `planning/plans/` -- Modify: `planning/releases/0.6.0.md` (audit-spec link), `planning/releases/TEMPLATE.md` (spec-link placeholder) - -- [ ] **Step 1: Remove the leftover `.gitkeep` and empty dirs** - - ```bash - git rm planning/plans/.gitkeep - rmdir planning/specs planning/plans 2>/dev/null || true - ls planning/ - ``` - Expected listing: `_templates audits changes deferred.md releases retros` - (no `specs`, no `plans`; `README.md` is added in Task 5). - -- [ ] **Step 2: Repoint the `0.6.0.md` audit-spec link** - - In `planning/releases/0.6.0.md`, replace: - ``` - Audit spec: [`planning/specs/2026-06-13-robustness-docs-test-audit-design.md`](../specs/2026-06-13-robustness-docs-test-audit-design.md). - ``` - with: - ``` - Audit spec: [`planning/changes/2026-06-13.01-robustness-docs-test-audit/design.md`](../changes/2026-06-13.01-robustness-docs-test-audit/design.md). - ``` - -- [ ] **Step 3: Repoint the `TEMPLATE.md` spec-link placeholder** - - In `planning/releases/TEMPLATE.md`, replace: - ``` - Spec: [`planning/specs/--design.md`](../specs/--design.md). - ``` - with: - ``` - Spec: [`planning/changes//design.md`](../changes//design.md). - ``` - -- [ ] **Step 4: Verify no live link points at the old paths** - - ```bash - grep -rn "(\.\./specs/\|(\.\./plans/" planning/releases && echo "STILL BROKEN (bad)" || echo "release links repointed" - ``` - Expected: `release links repointed`. - -- [ ] **Step 5: Commit** - - ```bash - git add planning/ - git commit -m "docs(planning): drop empty specs/plans dirs; repoint release links to bundles - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 5: Author `planning/README.md` (Conventions byte-identical + fresh Index) - -**Files:** -- Create: `planning/README.md` - -- [ ] **Step 1: Extract the byte-identical Conventions block** - - ```bash - awk '/^## Conventions/{f=1} /^## Index/{f=0} f' \ - /Users/kevinsmith/src/pypi/faststream-outbox/planning/README.md > /tmp/conventions.md - head -1 /tmp/conventions.md # should print: ## Conventions - ``` - -- [ ] **Step 2: Assemble `planning/README.md` from three parts** - - Write `planning/README.md` as: the intro below, then the **exact** contents of - `/tmp/conventions.md` (do not edit a word), then the Index + Other below. - - Intro (top of file): - ```markdown - # Planning - - Specs, plans, and change history for `faststream-concurrent-aiokafka`. The - living truth about *what the system does now* lives in - [`architecture/`](../architecture/) at the repo root; this directory records - *how it got there*. - ``` - - Index + Other (after the Conventions block): - ```markdown - ## Index - - ### Active - - - **[portable-planning-convention](changes/active/2026-06-13.03-portable-planning-convention/design.md)** - (2026-06-13) — Adopt the portable two-axis convention: `architecture/` truth - home + `changes/` bundles, fresh Index. *This change.* - - ### Archived (shipped) - - - **[codify-release-notes](changes/2026-06-13.02-codify-release-notes/design.md)** - (#33, 2026-06-13) — Codify `planning/releases/` as a workflow step; add the - release-notes template. - - **[robustness-docs-test-audit](changes/2026-06-13.01-robustness-docs-test-audit/design.md)** - (#32, 2026-06-13) — Rebalance-flush timeout, batch-subscriber guard, - uncommitted-task backpressure, plus docs/test/refactor. Shipped in 0.6.0. - - **[faststream-0.7.1-testbroker-typing](changes/2026-06-04.01-faststream-0.7.1-testbroker-typing/design.md)** - (#29, 2026-06-04) — Adopt FastStream 0.7.1's `TestBroker` typing fix; drop - `# ty: ignore` directives. - - **[faststream-0.7-migration](changes/2026-06-03.01-faststream-0.7-migration/design.md)** - (#28, 2026-06-03) — Migrate to `faststream>=0.7` (drop 0.6 support). - Design-only bundle (execution plan removed post-merge). - - ## Other - - - **[`architecture/`](../architecture/)** at the repo root — the living - capability truth (concurrent handler, batch committer, middleware & - lifecycle, rebalance listener). The promotion target on every ship. - - **[releases/](releases/)** — per-release user-facing notes, plus - [`releases/TEMPLATE.md`](releases/TEMPLATE.md), a repo-specific release-notes - template (not part of the portable core). - - **[audits/](audits/)** — findings reports from code/docs/bug-hunt sweeps - (none yet). - - **[retros/](retros/)** — what we learned after a body of work (none yet). - - **[deferred.md](deferred.md)** — the long-tail register of real-but- - unscheduled items with revisit triggers. - ``` - -- [ ] **Step 3: Verify the Conventions block is byte-identical** - - ```bash - diff <(awk '/^## Conventions/{f=1} /^## Index/{f=0} f' /Users/kevinsmith/src/pypi/faststream-outbox/planning/README.md) \ - <(awk '/^## Conventions/{f=1} /^## Index/{f=0} f' planning/README.md) \ - && echo "CONVENTIONS IDENTICAL" - ``` - Expected: `CONVENTIONS IDENTICAL` (no diff). - -- [ ] **Step 4: Verify every Index/Other link resolves** - - ```bash - grep -oE '\]\(([^)]+\.md)\)' planning/README.md | sed -E 's/\]\(|\)//g' | while read p; do - case "$p" in - ../*) [ -f "planning/$p" ] || echo "BROKEN: $p" ;; - *) [ -f "planning/$p" ] || echo "BROKEN: $p" ;; - esac - done; echo "done" - ``` - Expected: only `done` (no `BROKEN:` lines). - -- [ ] **Step 5: Commit** - - ```bash - git add planning/README.md - git commit -m "docs(planning): add README — portable Conventions + fresh Index - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 6: Rewrite `CLAUDE.md` `## Workflow` (preserve the release-notes step) - -**Files:** -- Modify: `CLAUDE.md` (`## Workflow` section, currently lines 24–39) - -- [ ] **Step 1: Replace the whole `## Workflow` section** - - Replace everything from the `## Workflow` line up to (but **not** including) the - `## Architecture` line with this exact text: - - ```markdown - ## Workflow - - Per-feature: brainstorming → spec in `planning/changes/active/YYYY-MM-DD.NN-/design.md` → writing-plans → plan in `planning/changes/active/YYYY-MM-DD.NN-/plan.md` → executing-plans / subagent-driven-development → requesting-code-review → finishing-a-development-branch. Each change is a folder bundle; `` is a kebab-case description, not a story ID; `.NN` is a zero-padded intra-day counter that breaks same-date ties so the timeline sorts stably. On merge, the bundle moves to `planning/changes/` with `status: shipped`, `pr:`, and `outcome:` filled, **and the change promotes its conclusions into the affected `architecture/.md`** — that hand-edit is what keeps `architecture/` true. See [`planning/README.md`](planning/README.md) for the conventions + index and [`planning/_templates/`](planning/_templates/) for copy-and-fill starting points. - - **Spec** (`design.md`) captures the *thinking* — why, what the design is, trade-offs, scope. Written before code; rarely revised after merge. **Plan** (`plan.md`) captures the *sequencing* — the ordered checklist an executor walks; references the spec for the "why". **`architecture/`** captures the *invariants* of shipped systems — the living truth, promoted from a change on merge. A plan paragraph that would still read correctly with all task numbers and checkboxes removed is design content and belongs in the spec. - - **Three lanes.** Scale the artifact to the change. **Full** — a `design.md` + `plan.md` bundle — for real design judgment, a new file/module, a public-API change, cross-cutting/multi-file work, or non-trivial test design. **Lightweight** — a single `change.md` — for small-but-real changes (≲30 LOC net, ≤2 files, no new file, no public-API change, a single straightforward test). **Tiny** — no bundle, just a conventional commit — for a typo, dep bump, linter/formatter/CI tweak, a mechanical rename, or a single-line config change. Heavier lane wins on ambiguity; a `change.md` that outgrows its lane splits into `design.md` + `plan.md`. - - Release notes are written when cutting a release (not per-feature): copy `planning/releases/TEMPLATE.md` to `planning/releases/.md` (bare version, no `v` prefix) and link back to the driving change bundle under `planning/changes/`. - - ``` - -- [ ] **Step 2: Verify the section reads cleanly and `## Architecture` still follows** - - ```bash - grep -n -A 9 "^## Workflow" CLAUDE.md - grep -n "^## Architecture" CLAUDE.md - ``` - Expected: the new four-paragraph Workflow section; `## Architecture` still - present immediately after it. No leftover `planning/specs/` or `planning/plans/` - mentions in the Workflow section. - -- [ ] **Step 3: Commit** - - ```bash - git add CLAUDE.md - git commit -m "docs: rewrite CLAUDE.md Workflow for the change-bundle convention - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 7: Full verification sweep - -**Files:** none (verification only). - -- [ ] **Step 1: Lint (markdown/format gate; no code changed)** - - ```bash - just lint-ci - ``` - Expected: passes (eof-fixer --check, ruff format --check, ruff check --no-fix, - ty all green). If eof-fixer flags a new markdown file missing a trailing - newline, run `just lint` to autofix, then re-stage and amend/commit. - -- [ ] **Step 2: Stale-pointer sweep (CLAUDE.md / README / justfile only)** - - ```bash - grep -rn "planning/specs\|planning/plans" CLAUDE.md README.md justfile 2>/dev/null \ - && echo "STALE POINTER (review)" || echo "no stale pointers in CLAUDE.md/README/justfile" - ``` - Expected: `no stale pointers in CLAUDE.md/README/justfile`. (Mentions inside - migrated archive prose under `planning/changes/` are historical and - acceptable.) - -- [ ] **Step 3: Final tree check** - - ```bash - ls -R architecture planning | sed -n '1,60p' - ``` - Expected: `architecture/` has the four `.md` files; `planning/` has - `README.md`, `_templates/`, `audits/`, `changes/{active,archive}`, `deferred.md`, - `releases/`, `retros/`; **no** `specs/`, `plans/`, or `templates/`. - `changes/active/` holds only this convention bundle (+ `.gitkeep`); - `changes/` holds the four migrated bundles. - -- [ ] **Step 4: Frontmatter parse sweep across every bundle file** - - ```bash - for f in $(find planning/changes -name design.md -o -name plan.md); do - uv run python -c "import yaml; t=open('$f').read(); assert t.startswith('---'),'$f'; yaml.safe_load(t.split('---')[1])" && echo "OK $f" || echo "BAD $f" - done - ``` - Expected: `OK` for every `design.md` / `plan.md` (the active convention bundle + - all archived bundles). No `BAD` lines. - -- [ ] **Step 5: Commit any lint autofix** - - ```bash - git add -A && git commit -m "docs(planning): lint/format fixups from convention migration - - Co-Authored-By: Claude Opus 4.8 (1M context) " || echo "nothing to commit" - ``` - ---- - -### Task 8: On merge — self-migrate this bundle from `active/` to `archive/` - -This convention bundle self-migrates (it defines the convention, so **no -`architecture/` promotion applies**). Do it within the shipping PR so merged -`main` lands in its final archived state. - -- [ ] **Step 1: Move the bundle and update its frontmatter** - - ```bash - git mv planning/changes/active/2026-06-13.03-portable-planning-convention \ - planning/changes/2026-06-13.03-portable-planning-convention - ``` - Then in both `design.md` and `plan.md` of the moved bundle set `status: - shipped` and `pr: ""`; in `design.md` also set `outcome: - "ships in #; defines the convention, no architecture/ promotion"`. - -- [ ] **Step 2: Move its README Index line from Active to Archived** - - In `planning/README.md`, move the `portable-planning-convention` entry from - `### Active` to the top of `### Archived (shipped)`, change its path from - `changes/active/...` to `changes/...`, add the PR number, and drop the - *This change.* suffix. Leave `### Active` as `_None._`. - -- [ ] **Step 3: Verify and commit** - - ```bash - grep -rn "changes/active/2026-06-13.03" planning/README.md && echo "STALE ACTIVE LINK (bad)" || echo "index updated" - git add planning/ - git commit -m "docs(planning): archive the portable-planning-convention bundle - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - Expected: `index updated`. - ---- - -## Self-Review - -- **Spec coverage:** - - Design §2 layout → Tasks 1, 3, 4 (skeleton, bundles, dir removal). ✓ - - Design §3 four `architecture/` files → Task 2. ✓ - - Design §4 migration mapping (4 bundles, `.NN`, frontmatter, design-only - 0.7-migration) → Task 3. ✓ - - Design §5 README (byte-identical Conventions + fresh Index/Other) → Task 5. ✓ - - Design §6 `_templates/` byte-identical → Task 1 Steps 3–4. ✓ - - Design §7 CLAUDE.md Workflow + preserved release-notes step → Task 6. ✓ - - Design §8 `.gitignore` trap → Task 1 Step 1. ✓ - - Design §9 `deferred.md` → Task 1 Step 5. ✓ - - Design §10 dogfood self-migration → Task 8. ✓ - - Non-goals (no docs-build, no releases/ move, no bundle backfill for #30/#31, - no archived-prose rewrite) → respected; no task violates them. ✓ - - Testing (lint-ci, grep sweeps, frontmatter parse, byte-identical diff, tree) - → Task 7. ✓ -- **Placeholders:** The `` / `` / `` - markers are intentional convention placeholders, not plan gaps; every concrete - step has exact paths, commands, and content. `architecture/` seed steps give - per-file section outlines bound to named `CLAUDE.md` content and code symbols. -- **Naming consistency:** bundle ids, slugs, and frontmatter `slug`/`spec` values - match across Tasks 3, 5, and 8; `commit_batch_size` / `max_uncommitted_tasks` / - `flush_timeout_sec` match `CLAUDE.md`. diff --git a/planning/changes/2026-06-23.01-pending-commits-deep-module/design.md b/planning/changes/2026-06-23.01-pending-commits-deep-module.md similarity index 100% rename from planning/changes/2026-06-23.01-pending-commits-deep-module/design.md rename to planning/changes/2026-06-23.01-pending-commits-deep-module.md diff --git a/planning/changes/2026-06-23.01-pending-commits-deep-module/plan.md b/planning/changes/2026-06-23.01-pending-commits-deep-module/plan.md deleted file mode 100644 index e9cf381..0000000 --- a/planning/changes/2026-06-23.01-pending-commits-deep-module/plan.md +++ /dev/null @@ -1,646 +0,0 @@ -# pending-commits-deep-module — implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps -> use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Move the committer's pending lists, pending count, cancellation -watermarks, and partition owners behind a synchronous `PendingCommits` module so -the offset invariants have one home and the unit tests stop reaching past the -committer's interface. - -**Spec:** [`design.md`](./design.md) - -**Branch:** `refactor/pending-commits-deep-module` - -**Commit strategy:** Per-task commits. - -## Global Constraints - -- Python ≥ 3.11. No `from __future__ import annotations`; annotations evaluate - eagerly; `typing.Self`/`typing.Never` used directly. -- All imports at module level — never inside function bodies. -- Type suppression is `# ty: ignore[rule-name]`, never `# type: ignore`. -- `PendingCommits` is **synchronous and I/O-free**: it reads `task.done()` / - `task.cancelled()` but never `await`s, never touches `asyncio.Queue`, never - calls Kafka. -- `PendingCommits` is **internal**: not added to `__init__.py`'s `__all__`. -- Public interface unchanged: `initialize_concurrent_processing`, - `KafkaConcurrentProcessingMiddleware`, `ConsumerRebalanceListener`, - `is_kafka_handler_healthy`, and the committer's collaborator surface - (`spawn` / `close` / `send_task` / `commit_all` / - `clear_cancellation_watermarks` / `is_healthy`). -- Strictly behaviour-preserving: same offsets committed, same redelivery on - cancellation, same rebalance flush. `test_integration.py`, `test_rebalance.py`, - and `test_middleware.py` pass **untouched**. -- 100% coverage enforced (pytest-cov). `just lint` (ruff + ty) must pass. -- Unit tests run without Docker: `uv run --no-sync pytest tests/ -v`. - The full gate (integration vs Redpanda) runs via `just test`. - ---- - -### Task 1: `PendingCommits` + `ReadyCommit` in `_pending_state.py` (TDD, new test file) - -**Files:** -- Modify: `faststream_concurrent_aiokafka/_pending_state.py` -- Test: `tests/test_pending_commits.py` (create) - -**Interfaces:** -- Consumes: existing module-level `insert_sorted`, `extract_ready_prefixes`, - `map_offsets_per_partition`, `KafkaCommitTask` from `_pending_state.py`; - `MockAIOKafkaConsumer` and `MockAsyncioTask` from `tests/mocks.py`. -- Produces (relied on by Task 2): - - `tests/mocks.py::make_commit_task(...)` — a shared `KafkaCommitTask` builder - (new; replaces the inline `KafkaCommitTask(asyncio_task=MockAsyncioTask(...))` - pattern duplicated across `test_kafka_committer.py`). - - `class ReadyCommit` — frozen dataclass with `consumer: Any`, - `offsets: dict[TopicPartition, int]`, `tasks: list[KafkaCommitTask]`. - - `class PendingCommits` with: - - `absorb(self, ct: KafkaCommitTask) -> None` - - `take_ready(self) -> list[ReadyCommit]` - - `clear_watermarks(self, partitions: typing.Iterable[TopicPartition] | None = None) -> None` - - `__len__(self) -> int` - -- [ ] **Step 1: Add a shared `make_commit_task` builder to `tests/mocks.py`** - - `tests/mocks.py` already has `MockAIOKafkaConsumer` and `MockAsyncioTask` - (`done()`/`cancelled()` controllable), but committer tests build - `KafkaCommitTask` inline. Add one builder both test files share: - - ```python - from faststream.kafka import TopicPartition - from faststream_concurrent_aiokafka._pending_state import KafkaCommitTask - - - def make_commit_task( - consumer: MockAIOKafkaConsumer, - topic_partition: TopicPartition, - offset: int, - *, - done: bool = False, - cancelled: bool = False, - ) -> KafkaCommitTask: - return KafkaCommitTask( - asyncio_task=MockAsyncioTask(result="ok", done=done, cancelled=cancelled), # ty: ignore[invalid-argument-type] - topic_partition=topic_partition, - offset=offset, - consumer=consumer, - ) - ``` - - (The `# ty: ignore[invalid-argument-type]` matches the existing inline pattern — - `MockAsyncioTask` is not a real `asyncio.Task`.) - -- [ ] **Step 2: Write the failing tests for `PendingCommits`** - - Create `tests/test_pending_commits.py`. The first lines define the shared - `mock_consumer` fixture (the existing one is local to `test_kafka_committer.py`): - - ```python - @pytest.fixture - def mock_consumer() -> MockAIOKafkaConsumer: - return MockAIOKafkaConsumer() - ``` - - Create `tests/test_pending_commits.py`. These tests target the new interface - directly — no committer, no queue, no I/O. - - ```python - import asyncio - import typing - - import pytest - from faststream.kafka import TopicPartition - - from faststream_concurrent_aiokafka import _pending_state - from faststream_concurrent_aiokafka._pending_state import PendingCommits, ReadyCommit - from tests.mocks import MockAIOKafkaConsumer, make_commit_task # reuse existing helpers - - - def _tp(partition: int = 0, topic: str = "t") -> TopicPartition: - return TopicPartition(topic=topic, partition=partition) - - - def test_len_counts_absorbed_tasks(mock_consumer: MockAIOKafkaConsumer) -> None: - pending = PendingCommits() - assert len(pending) == 0 - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=0, done=False)) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=1, done=False)) - assert len(pending) == 2 - - - def test_take_ready_empty_returns_empty_list() -> None: - assert PendingCommits().take_ready() == [] - - - def test_take_ready_commits_contiguous_done_prefix(mock_consumer: MockAIOKafkaConsumer) -> None: - pending = PendingCommits() - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=0, done=True)) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=1, done=True)) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=2, done=False)) - - ready = pending.take_ready() - - assert len(ready) == 1 - rc = ready[0] - assert rc.consumer is mock_consumer - assert rc.offsets == {_tp(): 2} # max processed (1) + 1 - assert len(rc.tasks) == 2 # only the done prefix - assert len(pending) == 1 # offset 2 still pending - - - def test_take_ready_stops_at_first_not_done(mock_consumer: MockAIOKafkaConsumer) -> None: - pending = PendingCommits() - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=0, done=False)) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=1, done=True)) - assert pending.take_ready() == [] # head not done → nothing ready - assert len(pending) == 2 - - - def test_cancelled_task_is_hard_boundary(mock_consumer: MockAIOKafkaConsumer) -> None: - pending = PendingCommits() - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=0, done=True)) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=1, cancelled=True)) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=2, done=True)) - - ready = pending.take_ready() - rc = ready[0] - - assert rc.offsets == {_tp(): 1} # advance stops at the cancelled task - assert len(rc.tasks) == 3 # cancelled + after dropped from pending into ready - assert len(pending) == 0 - - - def test_watermark_blocks_advance_until_cleared(mock_consumer: MockAIOKafkaConsumer) -> None: - pending = PendingCommits() - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=0, cancelled=True)) - pending.take_ready() # records the (consumer, tp) watermark at 0 - - # A later done task on the same partition must not advance past the floor. - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=1, done=True)) - assert pending.take_ready()[0].offsets == {} # withheld - - pending.clear_watermarks([_tp()]) - pending.absorb(make_commit_task(mock_consumer, _tp(), offset=2, done=True)) - assert pending.take_ready()[0].offsets == {_tp(): 3} # resumes after clear - - - def test_clear_watermarks_all_when_none(mock_consumer: MockAIOKafkaConsumer) -> None: - pending = PendingCommits() - pending.absorb(make_commit_task(mock_consumer, _tp(0), offset=0, cancelled=True)) - pending.absorb(make_commit_task(mock_consumer, _tp(1), offset=0, cancelled=True)) - pending.take_ready() - pending.clear_watermarks() # None → clear all - pending.absorb(make_commit_task(mock_consumer, _tp(0), offset=1, done=True)) - assert pending.take_ready()[0].offsets == {_tp(0): 2} - - - def test_two_consumers_same_partition_commit_independently() -> None: - a, b = MockAIOKafkaConsumer(), MockAIOKafkaConsumer() - pending = PendingCommits() - pending.absorb(make_commit_task(a, _tp(), offset=0, done=True)) - pending.absorb(make_commit_task(b, _tp(), offset=1, done=True)) - - ready = {id(rc.consumer): rc for rc in pending.take_ready()} - - assert ready[id(a)].offsets == {_tp(): 1} - assert ready[id(b)].offsets == {_tp(): 2} - ``` - -- [ ] **Step 3: Run the tests to verify they fail** - - Run: `uv run --no-sync pytest tests/test_pending_commits.py -v` - Expected: FAIL — `ImportError: cannot import name 'PendingCommits'`. - -- [ ] **Step 4: Implement `ReadyCommit` and `PendingCommits`** - - Append to `faststream_concurrent_aiokafka/_pending_state.py` (the module-level - `insert_sorted` / `extract_ready_prefixes` / `map_offsets_per_partition` stay as - the private implementation the class drives): - - ```python - @dataclasses.dataclass(frozen=True, slots=True) - class ReadyCommit: - # consumer typed Any to match KafkaCommitTask.consumer (avoids importing aiokafka at runtime) - consumer: typing.Any - offsets: dict[TopicPartition, int] - tasks: list[KafkaCommitTask] - - - class PendingCommits: - """Owns the per-partition pending commit tasks, the pending count, the - cancellation watermarks, and the partition owners. - - Synchronous and single-owner: the committer's streaming loop is the sole - mutator, so no locking is needed. Reads asyncio task state (done/cancelled) - but never awaits and never performs I/O. - """ - - def __init__(self) -> None: - self._pending: dict[TopicPartition, list[KafkaCommitTask]] = {} - self._count: int = 0 - self._watermarks: dict[tuple[int, TopicPartition], int] = {} - self._partition_owner: dict[TopicPartition, int] = {} - - def __len__(self) -> int: - return self._count - - def absorb(self, ct: KafkaCommitTask) -> None: - insert_sorted(self._pending.setdefault(ct.topic_partition, []), ct) - self._partition_owner[ct.topic_partition] = id(ct.consumer) - self._count += 1 - - def take_ready(self) -> list[ReadyCommit]: - # Extract each partition's contiguous-done prefix (cancelled = hard - # boundary), then group by consumer and apply the watermark floor. - # Atomic and synchronous: pending + watermark mutation both happen here, - # before any I/O the committer performs on the returned offsets. - ready, ready_count = extract_ready_prefixes(self._pending) - self._count -= ready_count - flat: list[KafkaCommitTask] = [t for tasks in ready.values() for t in tasks] - if not flat: - return [] - by_consumer: dict[int, list[KafkaCommitTask]] = {} - for task in flat: - by_consumer.setdefault(id(task.consumer), []).append(task) - result: list[ReadyCommit] = [] - for consumer_id, tasks in by_consumer.items(): - offsets = map_offsets_per_partition(consumer_id, tasks, self._watermarks) - result.append(ReadyCommit(consumer=tasks[0].consumer, offsets=offsets, tasks=tasks)) - return result - - def clear_watermarks(self, partitions: typing.Iterable[TopicPartition] | None = None) -> None: - if partitions is None: - self._watermarks.clear() - self._partition_owner.clear() - return - for partition in partitions: - owner = self._partition_owner.pop(partition, None) - if owner is not None: - self._watermarks.pop((owner, partition), None) - ``` - -- [ ] **Step 5: Run the tests to verify they pass** - - Run: `uv run --no-sync pytest tests/test_pending_commits.py -v` - Expected: PASS (all tests). - -- [ ] **Step 6: Lint** - - Run: `just lint` - Expected: clean (ruff format/check + ty). - -- [ ] **Step 7: Commit** - - ```bash - git add faststream_concurrent_aiokafka/_pending_state.py tests/test_pending_commits.py tests/mocks.py - git commit -m "refactor: add PendingCommits module owning pending/watermark state - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 2: Wire `PendingCommits` into the committer; reshape the commit path - -**Files:** -- Modify: `faststream_concurrent_aiokafka/batch_committer.py` -- Test: `tests/test_kafka_committer.py` (update committer-direct tests that touch - removed internals) - -**Interfaces:** -- Consumes: `PendingCommits`, `ReadyCommit` from Task 1. -- Produces (relied on by Task 3/4): - - `KafkaBatchCommitter._call_committer(self, rc: ReadyCommit) -> bool` - - `KafkaBatchCommitter._commit_ready(self, ready_commits: list[ReadyCommit]) -> bool` - - `KafkaBatchCommitter.clear_cancellation_watermarks` delegates to - `self._pending.clear_watermarks`. - - `self._pending: PendingCommits` replaces `_cancellation_watermarks`, - `_partition_owner`, and `_StreamingState.pending` / `pending_count`. - -This task is atomic: the offset state lives in one place, so the swap and the -commit-path reshape land together. The existing integration/rebalance/middleware -suites are the regression net. - -- [ ] **Step 1: Replace committer state and the streaming-state fields** - - In `KafkaBatchCommitter.__init__`, delete `self._cancellation_watermarks` and - `self._partition_owner`; add: - - ```python - self._pending: typing.Final = _pending_state.PendingCommits() - ``` - - In `_StreamingState`, delete the `pending` and `pending_count` fields (and the - two lines of their docstring invariants — the count invariant now lives inside - `PendingCommits`). - -- [ ] **Step 2: Point absorb/length/commit-trigger at `self._pending`** - - In `_run_commit_process`, change the loop guard: - - ```python - while not (state.should_shutdown and not self._pending): - ``` - - In `_streaming_iteration`, the absorb block: - - ```python - if not state.should_shutdown and state.queue_get_task.done(): - new_ct = state.queue_get_task.result() - self._track_user_task(new_ct) - self._pending.absorb(new_ct) - state.queue_get_task = asyncio.create_task(self._messages_queue.get()) - if state.timeout_deadline is None: - state.timeout_deadline = now + self._commit_batch_timeout_sec - ``` - - And the flush/deadline tail of `_streaming_iteration`: - - ```python - ready = await self._maybe_commit(state, timeout_fired) - if state.flush_in_progress and not self._pending: - state.flush_in_progress = False - - if ready or timeout_fired: - state.timeout_deadline = (loop.time() + self._commit_batch_timeout_sec) if self._pending else None - ``` - - In `_handle_flush_fired`, the drain loop: - - ```python - while True: - try: - ct = self._messages_queue.get_nowait() - except asyncio.QueueEmpty: - break - self._track_user_task(ct) - self._pending.absorb(ct) - ``` - -- [ ] **Step 3: Reshape `_maybe_commit` to use `take_ready`** - - ```python - async def _maybe_commit(self, state: "_StreamingState", timeout_fired: bool) -> list[_pending_state.ReadyCommit]: - commit_triggered: typing.Final = ( - len(self._pending) >= self._commit_batch_size - or timeout_fired - or state.flush_in_progress - or state.should_shutdown - ) - if not commit_triggered: - return [] - ready_commits: typing.Final = self._pending.take_ready() - if ready_commits: - await self._commit_ready(ready_commits) - return ready_commits - ``` - -- [ ] **Step 4: Replace `_commit_partitions` with `_commit_ready` and reshape `_call_committer`** - - Delete `_commit_partitions`. Add `_commit_ready`, which keeps the queue/count - bookkeeping and the concurrent per-consumer commit: - - ```python - async def _commit_ready(self, ready_commits: list[_pending_state.ReadyCommit]) -> bool: - # One commit per consumer, concurrently — each AIOKafkaConsumer commits its - # own partitions. task_done()/uncommitted_count balance the queue regardless - # of commit success (re-queued tasks are re-counted inside _call_committer). - results: typing.Final = await asyncio.gather(*(self._call_committer(rc) for rc in ready_commits)) - for rc in ready_commits: - for _ in rc.tasks: - self._messages_queue.task_done() - self._uncommitted_count -= sum(len(rc.tasks) for rc in ready_commits) - self._uncommitted_drained.set() - return all(results) - ``` - - Reshape `_call_committer` to take a `ReadyCommit`: - - ```python - async def _call_committer(self, rc: _pending_state.ReadyCommit) -> bool: - if not rc.offsets: - return True - try: - await rc.consumer.commit(rc.offsets) - except (CommitFailedError, IllegalStateError): - logger.exception("Cannot commit due to partition loss or rebalancing, ignoring batch") - return False - except KafkaError: - logger.exception("Error during commit to kafka, re-queuing batch") - for task in rc.tasks: - self._uncommitted_count += 1 - await self._messages_queue.put(task) - return False - else: - return True - ``` - -- [ ] **Step 5: Delegate watermark clearing; drop the static math delegators' call sites** - - Replace the body of `clear_cancellation_watermarks` (keep the method — it is a - real call path from `rebalance.py`): - - ```python - def clear_cancellation_watermarks(self, partitions: typing.Iterable[TopicPartition] | None = None) -> None: - self._pending.clear_watermarks(partitions) - ``` - - Leave the three static delegators (`_insert_sorted` module function, - `_extract_ready_prefixes`, `_map_offsets_per_partition` static methods) in place - for now — they are dead after this task but deleting them and repointing their - tests is Task 3. (This keeps Task 2 focused on behaviour, Task 3 on the seam.) - -- [ ] **Step 6: Update the committer-direct tests that referenced removed internals** - - In `tests/test_kafka_committer.py`: - - **`test_call_committer_*`** (the `_call_committer([...], {...})` tests): rewrite - to pass a `ReadyCommit`. Canonical transform: - - ```python - # before: result = await committer._call_committer([sample_task], partitions_to_offsets) - rc = _pending_state.ReadyCommit(consumer=sample_task.consumer, offsets=partitions_to_offsets, tasks=[sample_task]) - result = await committer._call_committer(rc) - ``` - The transient-`KafkaError` re-queue assertion (`committer._messages_queue` gets - the task back) is unchanged — `_call_committer` still re-queues `rc.tasks`. - - **`test_commit_partitions_*`**: rename target to `_commit_ready` and pass a - `list[ReadyCommit]`. Canonical transform: - - ```python - # before: await committer._commit_partitions({tp: tasks}) - rc = _pending_state.ReadyCommit( - consumer=tasks[0].consumer, - offsets=_pending_state.map_offsets_per_partition(id(tasks[0].consumer), tasks, {}), - tasks=tasks, - ) - await committer._commit_ready([rc]) - ``` - Keep the multi-consumer test (`test_commit_partitions_handles_multiple_consumers`) - — build one `ReadyCommit` per consumer and pass the list; it guards the - same-partition/two-consumer path. - - **`test_clear_cancellation_watermarks_*`**: these hand-seed - `committer._cancellation_watermarks[...]`, which no longer exists. Delete them - from this file — equivalent coverage now lives in - `tests/test_pending_commits.py` (`test_watermark_blocks_advance_until_cleared`, - `test_clear_watermarks_all_when_none`). - -- [ ] **Step 7: Run the unit suite** - - Run: `uv run --no-sync pytest tests/test_kafka_committer.py tests/test_pending_commits.py -v` - Expected: PASS. If any test still references `committer._cancellation_watermarks`, - `_partition_owner`, `_commit_partitions`, or `state.pending`, fix per Step 6. - -- [ ] **Step 8: Full gate (integration must be untouched and green)** - - Run: `just test` - Expected: PASS, including all of `test_integration.py`, `test_rebalance.py`, - `test_middleware.py` **with no edits to those files**. This is the proof the - refactor is behaviour-preserving. - -- [ ] **Step 9: Lint + commit** - - ```bash - just lint - git add faststream_concurrent_aiokafka/batch_committer.py tests/test_kafka_committer.py - git commit -m "refactor: route committer offset bookkeeping through PendingCommits - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 3: Delete the static math delegators; repoint their tests to `_pending_state` - -**Files:** -- Modify: `faststream_concurrent_aiokafka/batch_committer.py` -- Test: `tests/test_kafka_committer.py` - -The three delegators are shallow test handles: nothing in production calls them -after Task 2. The deletion test passes — removing them concentrates no -complexity; it only severs the committer from being the test door for the offset -math. - -- [ ] **Step 1: Confirm the delegators are dead in production** - - Run: `rg -n "_insert_sorted|_extract_ready_prefixes|_map_offsets_per_partition" faststream_concurrent_aiokafka/` - Expected: matches only in their own definitions in `batch_committer.py` (no - remaining call sites in production code). If `take_ready`/`absorb` call the - module-level `_pending_state.*` functions, the committer copies are unused. - -- [ ] **Step 2: Repoint the pure-function tests to `_pending_state` directly** - - In `tests/test_kafka_committer.py`, the `_map_offsets_per_partition` and - `_extract_ready_prefixes` test groups currently call the static methods. - Mechanical repoint (same args, same assertions): - - ```python - # before: KafkaBatchCommitter._map_offsets_per_partition(id(mock_consumer), tasks, {}) - _pending_state.map_offsets_per_partition(id(mock_consumer), tasks, {}) - - # before: KafkaBatchCommitter._extract_ready_prefixes(pending) - _pending_state.extract_ready_prefixes(pending) - ``` - - Apply to every test in the `# ---------- _map_offsets_per_partition ----------` - and `# ---------- _extract_ready_prefixes ----------` sections. These tests keep - the intricate offset-math coverage (earliest-cancelled-wins, contiguous prefix, - `max+1`, per-consumer watermark isolation) — now through the function that owns - it, not the committer. - -- [ ] **Step 3: Delete the three static delegators** - - Remove from `batch_committer.py`: - - the module-level `_insert_sorted` function, - - the `_map_offsets_per_partition` static method, - - the `_extract_ready_prefixes` static method. - - Remove any now-unused imports flagged by ruff. - -- [ ] **Step 4: Run the unit suite** - - Run: `uv run --no-sync pytest tests/test_kafka_committer.py tests/test_pending_commits.py -v` - Expected: PASS. Failures here mean a test still calls a deleted delegator — fix - per Step 2. - -- [ ] **Step 5: Lint + commit** - - ```bash - just lint - git add faststream_concurrent_aiokafka/batch_committer.py tests/test_kafka_committer.py - git commit -m "refactor: delete committer static delegators; test offset math directly - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 4: Seal — coverage gate, full integration, architecture promotion - -**Files:** -- Modify: `architecture/batch-committer.md` -- (Verification only on the rest.) - -Per the project workflow, promoting conclusions into `architecture/` is the -ship-time hand-edit that keeps it true. - -- [ ] **Step 1: Confirm no whitebox pokes remain past the seam** - - Run: `rg -n "committer\._cancellation_watermarks|committer\._partition_owner|committer\._commit_partitions|\._StreamingState|state\.pending" tests/` - Expected: no matches. Remaining `committer._messages_queue` references in the - `_commit_ready` / `send_task` tests are fine — the queue is genuinely the - committer's, tested at the committer's interface. - -- [ ] **Step 2: 100% coverage gate** - - Run: `just test-branch` - Expected: PASS at 100% coverage. If `PendingCommits` has an uncovered branch - (e.g. `clear_watermarks(None)` vs the partition path, or the empty-`offsets` - early-return in `_call_committer`), add the missing case to - `tests/test_pending_commits.py` and re-run. - -- [ ] **Step 3: Promote conclusions into `architecture/batch-committer.md`** - - Update the doc to reflect the shipped invariants: - - The committer delegates pending/watermark/owner state to a synchronous - `PendingCommits` (`_pending_state.py`); the committer keeps the queue, - backpressure, the `consumer.commit()` I/O, and the transient-error re-queue. - - `take_ready()` performs extraction + per-consumer offset mapping atomically - before any I/O, returning `ReadyCommit`s; the contiguous-prefix and - cancelled-as-hard-boundary rules now live in `PendingCommits`. - - `clear_cancellation_watermarks` is a thin delegator to - `PendingCommits.clear_watermarks` on the rebalance path. - -- [ ] **Step 4: Final lint + full gate** - - Run: `just lint && just test` - Expected: clean + all green. - -- [ ] **Step 5: Commit** - - ```bash - git add architecture/batch-committer.md - git commit -m "docs(architecture): record PendingCommits seam in batch-committer - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -## Self-review notes (for the executor) - -- **Spec coverage:** Task 1 = `PendingCommits` interface (design §1–§2); Task 2 = - loop wiring + commit-path reshape + delegator preservation (§3–§4); Task 3 = - static-delegator deletion (§5); Task 4 = test-door migration verification + - coverage + architecture promotion (Testing/Risk). `rebalance.py` is intentionally - never edited (Non-goal). -- **Behaviour preservation hinge:** `take_ready` must do extract → group-by-consumer - → `map_offsets_per_partition` with **no `await`** between them, exactly as today's - `_extract_ready_prefixes` → `_commit_partitions` ordering. The I/O-free constraint - enforces this. -- **Watermark timing:** recorded inside `take_ready` (synchronous) before the - committer's `consumer.commit()` await — same as today's record-before-commit. -- **Count invariant:** `__len__` decremented by the full `ready_count` from - `extract_ready_prefixes` (includes cancelled+after dropped into ready), matching - today's `state.pending_count -= ready_count`. diff --git a/planning/changes/2026-06-24.01-commit-scheduler-deep-module/design.md b/planning/changes/2026-06-24.01-commit-scheduler-deep-module.md similarity index 100% rename from planning/changes/2026-06-24.01-commit-scheduler-deep-module/design.md rename to planning/changes/2026-06-24.01-commit-scheduler-deep-module.md diff --git a/planning/changes/2026-06-24.01-commit-scheduler-deep-module/plan.md b/planning/changes/2026-06-24.01-commit-scheduler-deep-module/plan.md deleted file mode 100644 index 29540f5..0000000 --- a/planning/changes/2026-06-24.01-commit-scheduler-deep-module/plan.md +++ /dev/null @@ -1,576 +0,0 @@ -# commit-scheduler-deep-module — implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps -> use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Extract the streaming loop's when-to-commit decision state (timeout -deadline, flush lifecycle, shutdown lifecycle, batch-size trigger) out of -`KafkaBatchCommitter` into a pure synchronous `CommitScheduler`, so the four -loop invariants are enforced in one place and unit-testable without asyncio. - -**Spec:** [`design.md`](./design.md) - -**Branch:** `refactor/commit-scheduler-deep-module` - -**Commit strategy:** Per-task commits. - -## Global Constraints - -- Python ≥ 3.11. No `from __future__ import annotations`; eager annotations; - `typing.Self`/`typing.Never` used directly. -- All imports at module level — never inside function bodies. -- Type suppression is `# ty: ignore[rule-name]`, never `# type: ignore`. -- `CommitScheduler` is **synchronous and I/O-free**: it reads no clock (the - driver passes `now = loop.time()` in) and touches no asyncio object. -- `CommitScheduler` is **internal**: not added to `__init__.py`'s `__all__`. -- Public interface unchanged: `initialize_concurrent_processing`, - `KafkaConcurrentProcessingMiddleware`, `ConsumerRebalanceListener`, - `is_kafka_handler_healthy`, and the committer's surface - (`spawn` / `close` / `send_task` / `commit_all` / - `clear_cancellation_watermarks` / `is_healthy`). -- Strictly behaviour-preserving: same iteration semantics, same commit timing, - same shutdown sequence. `test_integration.py`, `test_rebalance.py`, - `test_middleware.py`, and the committer-level tests in - `test_kafka_committer.py` pass **untouched**. -- 100% coverage enforced (pytest-cov). `just lint` (ruff + ty) must pass. -- Unit tests run without Docker: `uv run --no-sync pytest tests/ -v`. - The full gate (integration vs Redpanda) runs via `just test`. Lint gate is - check-only: `just lint-ci` (eof-fixer may flag scratch files outside - `faststream_concurrent_aiokafka/` and `tests/` — only those two trees must be - ruff + ty clean). - ---- - -### Task 1: `CommitScheduler` + `Decision` in `_commit_scheduler.py` (TDD, new test file) - -**Files:** -- Create: `faststream_concurrent_aiokafka/_commit_scheduler.py` -- Test: `tests/test_commit_scheduler.py` (create) - -**Interfaces:** -- Consumes: nothing (pure, no project imports beyond stdlib `dataclasses`/`typing`). -- Produces (relied on by Task 2): - - `class Decision` — frozen dataclass: `should_commit: bool`, - `drain_queue_now: bool`, `timeout_fired: bool`. - - `class CommitScheduler` with: - - `__init__(self, *, commit_batch_size: int, commit_batch_timeout_sec: float)` - - `accepts_new_work(self) -> bool` - - `wait_timeout(self, now: float) -> float | None` - - `evaluate(self, *, now: float, absorbed: bool, flush_fired: bool, stop_requested: bool, pending_len: int) -> Decision` - - `note_committed(self, *, now: float, committed: bool, timeout_fired: bool, pending_empty: bool) -> None` - - `is_finished(self, *, pending_empty: bool) -> bool` - -- [ ] **Step 1: Write the failing tests** - - Create `tests/test_commit_scheduler.py`. These feed observation sequences with - zero asyncio. - - ```python - from faststream_concurrent_aiokafka._commit_scheduler import CommitScheduler, Decision - - - def _sched(batch_size: int = 10, timeout: float = 10.0) -> CommitScheduler: - return CommitScheduler(commit_batch_size=batch_size, commit_batch_timeout_sec=timeout) - - - def test_initial_state_accepts_work_no_deadline_not_finished() -> None: - s = _sched() - assert s.accepts_new_work() is True - assert s.wait_timeout(now=100.0) is None - assert s.is_finished(pending_empty=True) is False - - - def test_deadline_arms_on_first_absorb_and_does_not_rearm() -> None: - s = _sched(timeout=10.0) - s.evaluate(now=100.0, absorbed=False, flush_fired=False, stop_requested=False, pending_len=0) - assert s.wait_timeout(now=100.0) is None # nothing absorbed → no deadline - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - assert s.wait_timeout(now=100.0) == 10.0 # armed at now + timeout - s.evaluate(now=103.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=2) - assert s.wait_timeout(now=103.0) == 7.0 # ticks down; not re-armed - - - def test_wait_timeout_floors_at_zero_past_deadline() -> None: - s = _sched(timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - assert s.wait_timeout(now=115.0) == 0.0 - - - def test_timeout_fires_at_deadline_and_triggers_commit() -> None: - s = _sched(batch_size=10, timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - d = s.evaluate(now=105.0, absorbed=False, flush_fired=False, stop_requested=False, pending_len=1) - assert d.timeout_fired is False - assert d.should_commit is False - d = s.evaluate(now=110.0, absorbed=False, flush_fired=False, stop_requested=False, pending_len=1) - assert d.timeout_fired is True - assert d.should_commit is True - - - def test_batch_size_triggers_commit() -> None: - s = _sched(batch_size=3, timeout=10.0) - d = s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=3) - assert d.should_commit is True - assert d.timeout_fired is False - assert d.drain_queue_now is False - - - def test_flush_without_stop_commits_until_pending_drains() -> None: - s = _sched(batch_size=10, timeout=10.0) - d = s.evaluate(now=100.0, absorbed=False, flush_fired=True, stop_requested=False, pending_len=1) - assert d.should_commit is True # flush_in_progress drives commit - assert d.drain_queue_now is False - d2 = s.evaluate(now=101.0, absorbed=False, flush_fired=False, stop_requested=False, pending_len=1) - assert d2.should_commit is True # keeps committing while flush_in_progress - s.note_committed(now=102.0, committed=True, timeout_fired=False, pending_empty=True) - d3 = s.evaluate(now=103.0, absorbed=False, flush_fired=False, stop_requested=False, pending_len=0) - assert d3.should_commit is False # flag cleared once pending drained - - - def test_flush_with_stop_sets_shutdown_and_drain() -> None: - s = _sched() - d = s.evaluate(now=100.0, absorbed=False, flush_fired=True, stop_requested=True, pending_len=2) - assert d.drain_queue_now is True - assert d.should_commit is True - assert s.accepts_new_work() is False - assert s.is_finished(pending_empty=False) is False - assert s.is_finished(pending_empty=True) is True - - - def test_deadline_reset_keeps_ticking_when_pending_remains() -> None: - s = _sched(timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=5) - s.note_committed(now=104.0, committed=True, timeout_fired=False, pending_empty=False) - assert s.wait_timeout(now=104.0) == 10.0 # re-armed at fresh now + timeout - - - def test_deadline_cleared_when_pending_drains() -> None: - s = _sched(timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - s.note_committed(now=104.0, committed=True, timeout_fired=False, pending_empty=True) - assert s.wait_timeout(now=104.0) is None # invariant: pending empty ⇒ no deadline - - - def test_note_committed_resets_on_timeout_even_without_commit() -> None: - s = _sched(timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - s.note_committed(now=110.0, committed=False, timeout_fired=True, pending_empty=False) - assert s.wait_timeout(now=110.0) == 10.0 - - - def test_note_committed_no_reset_when_neither_committed_nor_timeout() -> None: - s = _sched(timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - s.note_committed(now=103.0, committed=False, timeout_fired=False, pending_empty=False) - assert s.wait_timeout(now=103.0) == 7.0 # deadline left ticking, not reset - - - def test_no_trigger_when_idle_below_batch() -> None: - s = _sched(batch_size=10, timeout=10.0) - s.evaluate(now=100.0, absorbed=True, flush_fired=False, stop_requested=False, pending_len=1) - d = s.evaluate(now=102.0, absorbed=False, flush_fired=False, stop_requested=False, pending_len=2) - assert d.should_commit is False - ``` - -- [ ] **Step 2: Run the tests to verify they fail** - - Run: `uv run --no-sync pytest tests/test_commit_scheduler.py -v` - Expected: FAIL — `ModuleNotFoundError: No module named '..._commit_scheduler'`. - -- [ ] **Step 3: Implement `Decision` and `CommitScheduler`** - - Create `faststream_concurrent_aiokafka/_commit_scheduler.py`: - - ```python - import dataclasses - - - @dataclasses.dataclass(frozen=True, slots=True) - class Decision: - should_commit: bool # the batch-size / timeout / flush / shutdown trigger fired - drain_queue_now: bool # flush fired with stop_requested → driver drains the queue into pending - timeout_fired: bool # surfaced so the driver hands it back to note_committed for the deadline reset - - - class CommitScheduler: - """Owns the streaming loop's when-to-commit decision state: the timeout - deadline, the flush lifecycle, and the shutdown lifecycle. - - Synchronous, I/O-free, single-owner: the committer's async driver is the - sole caller, on one asyncio task. Reads no clock and touches no asyncio - object — the driver passes ``now = loop.time()`` in. The driver never - writes a decision field; it only feeds observations (evaluate) and acts on - the returned Decision, so the invariants below are enforced here, not - annotated. - - Invariants: - * pending empty ⇒ timeout_deadline is None. - * flush_in_progress is set only when a flush fired without a stop request, - and cleared once pending drains. - * should_shutdown is set only when a flush fired with a stop request; once - set, is_finished() returns True as soon as pending drains. - """ - - def __init__(self, *, commit_batch_size: int, commit_batch_timeout_sec: float) -> None: - self._batch_size = commit_batch_size - self._batch_timeout = commit_batch_timeout_sec - self._timeout_deadline: float | None = None - self._should_shutdown: bool = False - self._flush_in_progress: bool = False - - def accepts_new_work(self) -> bool: - # While shutting down, the driver stops pulling new items from the queue. - return not self._should_shutdown - - def wait_timeout(self, now: float) -> float | None: - # Remaining time until the batch-timeout fires, for asyncio.wait. None when - # no deadline is armed (pending empty), so the select blocks until an event. - if self._timeout_deadline is None: - return None - return max(self._timeout_deadline - now, 0.0) - - def evaluate( - self, - *, - now: float, - absorbed: bool, - flush_fired: bool, - stop_requested: bool, - pending_len: int, - ) -> Decision: - # Arm the deadline on the first pending item (no-op once armed). - if absorbed and self._timeout_deadline is None: - self._timeout_deadline = now + self._batch_timeout - - timeout_fired = self._timeout_deadline is not None and now >= self._timeout_deadline - - drain_queue_now = False - if flush_fired: - if stop_requested: - self._should_shutdown = True - drain_queue_now = True - else: - self._flush_in_progress = True - - should_commit = ( - pending_len >= self._batch_size - or timeout_fired - or self._flush_in_progress - or self._should_shutdown - ) - return Decision( - should_commit=should_commit, - drain_queue_now=drain_queue_now, - timeout_fired=timeout_fired, - ) - - def note_committed( - self, - *, - now: float, - committed: bool, - timeout_fired: bool, - pending_empty: bool, - ) -> None: - # An active commit_all (flush without stop) keeps committing until pending - # drains; clear the flag once it does so messages_queue.join() can return. - if self._flush_in_progress and pending_empty: - self._flush_in_progress = False - # Re-arm the deadline after a commit round or a timeout firing; otherwise - # let it keep ticking. Invariant: pending empty ⇒ deadline None. - if committed or timeout_fired: - self._timeout_deadline = (now + self._batch_timeout) if not pending_empty else None - - def is_finished(self, *, pending_empty: bool) -> bool: - return self._should_shutdown and pending_empty - ``` - -- [ ] **Step 4: Run the tests to verify they pass** - - Run: `uv run --no-sync pytest tests/test_commit_scheduler.py -v` - Expected: PASS (all tests). - -- [ ] **Step 5: Confirm 100% coverage of the new module** - - Run: `uv run --no-sync pytest tests/test_commit_scheduler.py --cov=faststream_concurrent_aiokafka/_commit_scheduler --cov-report=term-missing` - Expected: `_commit_scheduler.py` at 100% (no missing lines/branches). If a - branch is uncovered, add the missing case before committing. - -- [ ] **Step 6: Lint + commit** - - ```bash - just lint-ci # faststream_concurrent_aiokafka/ + tests/ must be ruff + ty clean - git add faststream_concurrent_aiokafka/_commit_scheduler.py tests/test_commit_scheduler.py - git commit -m "refactor: add CommitScheduler owning the loop's when-to-commit state - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 2: Wire `CommitScheduler` into the committer's driver - -**Files:** -- Modify: `faststream_concurrent_aiokafka/batch_committer.py` - -**Interfaces:** -- Consumes: `CommitScheduler`, `Decision` from Task 1. -- Produces: the committer's async driver now delegates every when-to-commit - decision to `self._scheduler`; `_StreamingState` is replaced by `_LoopTasks` - holding only the three wait-tasks. - -This task is atomic — the loop's decision state lives in one place, so the swap -lands together. The integration/middleware/rebalance suites are the regression -net. NOT a TDD task; preserve behaviour, verify by keeping existing tests green. - -- [ ] **Step 1: Import the scheduler and construct it; drop the moved config fields** - - In `batch_committer.py`, extend the import: - - ```python - from faststream_concurrent_aiokafka import _commit_scheduler, _pending_state, consts - ``` - - In `KafkaBatchCommitter.__init__`, delete the two lines - `self._commit_batch_timeout_sec = commit_batch_timeout_sec` and - `self._commit_batch_size = commit_batch_size`, and add (next to `self._pending`): - - ```python - self._scheduler: typing.Final = _commit_scheduler.CommitScheduler( - commit_batch_size=commit_batch_size, - commit_batch_timeout_sec=commit_batch_timeout_sec, - ) - ``` - - (The constructor still accepts `commit_batch_size` / `commit_batch_timeout_sec` - — public API unchanged — it just forwards them to the scheduler.) - -- [ ] **Step 2: Replace `_StreamingState` with `_LoopTasks`** - - Replace the `_StreamingState` dataclass (the decision fields and the invariant - docstring move into `CommitScheduler`): - - ```python - @dataclasses.dataclass(kw_only=True, slots=True) - class _LoopTasks: - """The three asyncio wait-tasks the streaming select multiplexes.""" - - queue_get_task: asyncio.Task[KafkaCommitTask] - flush_wait_task: asyncio.Task[bool] - task_completed_wait_task: asyncio.Task[bool] - - def cancel_outstanding(self) -> None: - for task in (self.queue_get_task, self.flush_wait_task, self.task_completed_wait_task): - if not task.done(): - task.cancel() - ``` - -- [ ] **Step 3: Rewrite `_run_commit_process` to drive the scheduler** - - ```python - async def _run_commit_process(self) -> None: - tasks: typing.Final = _LoopTasks( - queue_get_task=asyncio.create_task(self._messages_queue.get()), - flush_wait_task=asyncio.create_task(self._flush_batch_event.wait()), - task_completed_wait_task=asyncio.create_task(self._task_completed_event.wait()), - ) - try: - while not self._scheduler.is_finished(pending_empty=not self._pending): - await self._streaming_iteration(tasks) - finally: - tasks.cancel_outstanding() - self._uncommitted_drained.set() - ``` - -- [ ] **Step 4: Rewrite `_streaming_iteration` as pure glue (no flag writes)** - - ```python - async def _streaming_iteration(self, tasks: "_LoopTasks") -> None: - loop: typing.Final = asyncio.get_running_loop() - - wait_targets: list[asyncio.Future[typing.Any]] = [ - tasks.flush_wait_task, - tasks.task_completed_wait_task, - ] - if self._scheduler.accepts_new_work(): - wait_targets.append(tasks.queue_get_task) - - remaining: typing.Final = self._scheduler.wait_timeout(loop.time()) - await asyncio.wait(wait_targets, return_when=asyncio.FIRST_COMPLETED, timeout=remaining) - - # Capture once after the wait — used for both the deadline arm and timeout_fired. - now: typing.Final = loop.time() - - absorbed: typing.Final = self._scheduler.accepts_new_work() and tasks.queue_get_task.done() - if absorbed: - new_ct = tasks.queue_get_task.result() - self._track_user_task(new_ct) - self._pending.absorb(new_ct) - tasks.queue_get_task = asyncio.create_task(self._messages_queue.get()) - - # Re-arm the completion event before deciding, so a task finishing during this - # iteration is captured next time instead of being lost between clear and re-wait. - if tasks.task_completed_wait_task.done(): - self._task_completed_event.clear() - tasks.task_completed_wait_task = asyncio.create_task(self._task_completed_event.wait()) - - flush_fired: typing.Final = tasks.flush_wait_task.done() - - decision: typing.Final = self._scheduler.evaluate( - now=now, - absorbed=absorbed, - flush_fired=flush_fired, - stop_requested=self._stop_requested, - pending_len=len(self._pending), - ) - - if flush_fired: - self._handle_flush_fired(tasks, drain_queue=decision.drain_queue_now) - - committed = False - if decision.should_commit: - ready = self._pending.take_ready() - if ready: - await self._commit_ready(ready) - committed = True - - self._scheduler.note_committed( - now=loop.time(), - committed=committed, - timeout_fired=decision.timeout_fired, - pending_empty=not self._pending, - ) - ``` - - Ordering note (behaviour-preserving): `evaluate` sets the flush flags **then** - computes the trigger in one call, so flags set this iteration drive this - iteration's commit — exactly as the old `_handle_flush_fired`-before-`_maybe_commit` - order. The shutdown drain runs (via `_handle_flush_fired`) before `take_ready`, - so drained items commit in the same iteration as today. - -- [ ] **Step 5: Reduce `_handle_flush_fired` to driver plumbing and delete `_maybe_commit`** - - `_handle_flush_fired` no longer writes any flag — it only drains (on shutdown) - and re-arms the flush wait-task: - - ```python - def _handle_flush_fired(self, tasks: "_LoopTasks", *, drain_queue: bool) -> None: - if drain_queue: - # Drain anything still buffered in messages_queue into pending so close() - # can commit it. Without this, items put before close() but not yet - # absorbed by queue_get would be silently dropped (offsets stay - # uncommitted; redelivered on restart, but close() callers expect - # everything enqueued to be processed). - while True: - try: - ct = self._messages_queue.get_nowait() - except asyncio.QueueEmpty: - break - self._track_user_task(ct) - self._pending.absorb(ct) - if not tasks.queue_get_task.done(): - tasks.queue_get_task.cancel() - self._flush_batch_event.clear() - tasks.flush_wait_task = asyncio.create_task(self._flush_batch_event.wait()) - ``` - - Delete the `_maybe_commit` method entirely (its trigger logic moved into - `CommitScheduler.evaluate`; its commit execution is inlined in - `_streaming_iteration` Step 4). - -- [ ] **Step 6: Run the committer unit suite** - - Run: `uv run --no-sync pytest tests/test_kafka_committer.py tests/test_commit_scheduler.py -v` - Expected: PASS. The one loop reference in the unit tests - (`patch.object(committer, "_run_commit_process", ...)`) still resolves — - `_run_commit_process` remains on the committer. If anything references the - removed `_StreamingState` / `_maybe_commit` / `_commit_batch_size`, fix it. - -- [ ] **Step 7: Full gate (integration must be untouched and green)** - - Run: `just test` - Expected: PASS, including all of `test_integration.py`, `test_rebalance.py`, - `test_middleware.py` **with no edits to those files**. This is the proof the - refactor is behaviour-preserving. (Requires Docker; if unavailable, STOP and - report DONE_WITH_CONCERNS naming exactly that, with the unit results you have — - do not claim the integration gate passed if you did not run it.) - -- [ ] **Step 8: Lint + commit** - - ```bash - just lint-ci - git add faststream_concurrent_aiokafka/batch_committer.py - git commit -m "refactor: drive commit timing through CommitScheduler - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -### Task 3: Seal — coverage gate, full integration, architecture promotion - -**Files:** -- Modify: `architecture/batch-committer.md` -- (Verification only on the rest.) - -- [ ] **Step 1: Confirm the loop's decision state is no longer reachable from tests** - - Run: `grep -rnE "_StreamingState|_maybe_commit|flush_in_progress|should_shutdown|timeout_deadline" tests/` - Expected: no matches. (A surviving `patch.object(committer, "_run_commit_process")` - reference is fine — it mocks the driver, not the decision state.) - -- [ ] **Step 2: 100% coverage gate** - - Run: `just test-branch` - Expected: PASS at 100% coverage. If `CommitScheduler` or the rewritten driver - has an uncovered branch, add the missing case (to `tests/test_commit_scheduler.py` - for decider gaps) and re-run. - -- [ ] **Step 3: Promote conclusions into `architecture/batch-committer.md`** - - Read the current doc, then add/adjust to record the shipped split (minimal, - accurate edits — do not rewrite wholesale): - - The committer's `_run_commit_process` is the **async driver**: it owns the - `asyncio.wait` select over the three wait-tasks (queue-get / flush / task-done) - and the queue, but delegates every when-to-commit decision to a pure - synchronous `CommitScheduler` (`_commit_scheduler.py`). - - `CommitScheduler` owns the timeout deadline, the flush lifecycle - (`flush_in_progress`), and the shutdown lifecycle (`should_shutdown`); the - driver feeds it observations (`evaluate`) and acts on the returned `Decision`, - never writing a decision field itself. - - The commit triggers are unchanged (pending ≥ `commit_batch_size`, the - `commit_batch_timeout_sec` deadline, or a `commit_all`/`close` flush); they - are now computed inside `CommitScheduler.evaluate`. - -- [ ] **Step 4: Final gate** - - Run: `just lint && just test` - Expected: clean + all green. - -- [ ] **Step 5: Commit** - - ```bash - git add architecture/batch-committer.md - git commit -m "docs(architecture): record CommitScheduler / async-driver split - - Co-Authored-By: Claude Opus 4.8 (1M context) " - ``` - ---- - -## Self-review notes (for the executor) - -- **Spec coverage:** Task 1 = `CommitScheduler` interface + invariants (design - §1–§2); Task 2 = driver rewrite delegating to the scheduler (§3–§4); Task 3 = - seal + architecture promotion (Testing/Risk). The async machinery, produce - side, backpressure, queue, and commit I/O are intentionally untouched (Non-goals). -- **Behaviour-preservation hinges (Task 2):** `now` is captured once after the - await and used for both the deadline arm and `timeout_fired`; `note_committed` - uses a **fresh** `loop.time()` and resets on `committed OR timeout_fired`; - `flush_in_progress` clears on post-commit `pending_empty` before the reset; - flush flags set in `evaluate` drive the same-iteration commit; the shutdown - drain runs before `take_ready`. -- **No whitebox migration:** the loop decision logic had no unit tests, so no - test repointing is needed — Task 1 adds net-new coverage. diff --git a/planning/changes/2026-06-24.02-route-classify-extraction/change.md b/planning/changes/2026-06-24.02-route-classify-extraction.md similarity index 100% rename from planning/changes/2026-06-24.02-route-classify-extraction/change.md rename to planning/changes/2026-06-24.02-route-classify-extraction.md From c08bf32dbe79dc0670d4befa42e474a6e9ed36db Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Wed, 8 Jul 2026 10:38:40 +0300 Subject: [PATCH 3/5] docs(planning): merge convention 2.0.0 prose into README, CLAUDE.md, Justfile Replace the 1.x Conventions section in planning/README.md with the 2.0.0 canonical body (Quick path, glossary, flat change files) verbatim from the lesnik512/planning-convention APPLY.md kit. Retire "bundle" wording in CLAUDE.md and the Justfile comment in favor of "change"/ "change file". Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 6 +-- Justfile | 2 +- planning/README.md | 118 +++++++++++++++++++++++++++++++++++---------- 3 files changed, 96 insertions(+), 30 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 201b63b..4216c95 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,10 +18,10 @@ changing `pyproject.toml`. Non-obvious notes: Planning uses a portable two-axis convention: `architecture/` (repo root) is the living **truth home** and promotion target; `planning/changes/` holds the -per-change bundles. **Start at the Quick path** in +per-change files. **Start at the Quick path** in [`planning/README.md`](planning/README.md) to choose a lane (Full / Lightweight -/ Tiny), create a bundle, and ship — that file is the authoritative spec. Run -`just check-planning` to validate bundles and `just index` to print the listing. +/ Tiny), create a change file, and ship — that file is the authoritative spec. Run +`just check-planning` to validate changes and `just index` to print the listing. Release notes: copy `planning/releases/TEMPLATE.md` to `planning/releases/.md` (bare version, no `v` prefix) when cutting a release. diff --git a/Justfile b/Justfile index 59b7311..d02b6b6 100644 --- a/Justfile +++ b/Justfile @@ -33,7 +33,7 @@ lint-ci: index: uv run python planning/index.py -# Validate planning bundles + decisions; CI runs this. +# Validate planning changes + decisions; CI runs this. check-planning: uv run python planning/index.py --check diff --git a/planning/README.md b/planning/README.md index 892d2cb..f69feab 100644 --- a/planning/README.md +++ b/planning/README.md @@ -5,29 +5,77 @@ living truth about *what the system does now* lives in [`architecture/`](../architecture/) at the repo root; this directory records *how it got there*. +## Quick path (start here) + +> The fast lane for making a change. The full reference is in +> [Conventions](#conventions) below — read it only when this isn't enough. + +**1. Choose a lane — first matching rule wins:** + +1. Any of: needs design judgment · new file/module · public-API change · + cross-cutting or multi-file · non-trivial test design → **Full** (design template) +2. Purely mechanical: typo · dep bump · linter/formatter/CI tweak · + mechanical rename · single-line config → **Tiny** (no change file, conventional + commit) +3. Small-but-real, none of the above: ≲30 LOC net · ≤2 files · no new file · + no public-API change · one straightforward test → **Lightweight** (change template) + +Ambiguous between two? Take the heavier. A lightweight change file that outgrows its lane is rewritten from the design template. + +**2. Create the change file** (Full / Lightweight only): +`planning/changes/YYYY-MM-DD.NN-.md`, where `.NN` is a zero-padded +intra-day counter — copied from the matching template (design or change) in +[`_templates/`](_templates/). + +**3. Ship in the implementing PR:** hand-edit the affected +`architecture/.md`, finalize the change file's `summary:` to the +realized result, and run `just check-planning` before pushing. + ## Conventions -> This section is the portable convention — identical across the -> modern-python repos. The generated change listing (`just index`) and the `## Other` pointers below are repo-local. To adopt elsewhere, -> copy this section plus [`_templates/`](_templates/) and point that repo's -> `CLAUDE.md` Workflow + truth home at it. +> This is the portable convention, sourced from the canonical repo +> [`lesnik512/planning-convention`](https://github.com/lesnik512/planning-convention) +> (applied version in [`.convention-version`](.convention-version)). To update +> it, run that repo's `APPLY.md` flow. The generated change index (`just index`) +> and the `## Other` pointers below are repo-local. ### Two axes, never mixed -- **`architecture/` (repo root) — the present.** One file per capability, - living prose, updated in the same PR that ships the change. The truth home. -- **`planning/changes/` — the past-and-pending.** One folder per change, +- **`architecture/` (repo root) — the present.** One file per capability, plus + a single `glossary.md` (the ubiquitous language); living prose, updated in the + same PR that ships the change. The truth home. +- **`planning/changes/` — the past-and-pending.** One file per change, kept in place after ship. A change **promotes** its conclusions into the affected `architecture/.md` by hand **in the implementing PR, alongside the code** — the edit rides in the same diff and is reviewed with it, never applied as a separate post-merge step. That hand-edit is what keeps `architecture/` -true; the bundle stays in `changes/` as the *why*. +true; the change file stays in `changes/` as the *why*. + +### Glossary + +`architecture/glossary.md` is the project's **ubiquitous language** — one page +defining the domain terms that code, specs, and capability pages all share. Like +the capability files beside it, it is living prose with **no frontmatter**, dated +by git, and authored lazily: it appears when the first term is worth pinning down. + +Each entry is a term, a one-or-two-sentence definition of what it *is* (not what +it does), and an optional `_Avoid_:` line naming the synonyms to reject: -### Change bundles +```md +**Timer**: +A scheduled future delivery, identified by a timer id. +_Avoid_: job, task, alarm +``` -A change is a folder `changes/YYYY-MM-DD.NN-/`: +Keep it a glossary, not a spec — no implementation detail. A change that +introduces or sharpens a term updates `glossary.md` in the same PR, the same way +a behavior change promotes into a capability file. + +### Change files + +A change is a file `changes/YYYY-MM-DD.NN-.md`: - `YYYY-MM-DD` — proposal date; `.NN` — zero-padded intra-day counter (`.01`, `.02`, …) that breaks same-date ties so the timeline sorts stably. @@ -36,43 +84,61 @@ A change is a folder `changes/YYYY-MM-DD.NN-/`: `summary` is written when the change is created (the intent one-liner) and **finalized at ship** to state the realized result — set in the implementing PR, alongside the code and the `architecture/` promotion. No post-merge -bookkeeping, no folder move. `date` and `slug` are never written — they are -read from the bundle's directory name. +bookkeeping, no file move. `date` and `slug` are never written — they are +read from the file name. ### Three lanes | Lane | Artifacts | Use when | |------|-----------|----------| -| **Full** | `design.md` + `plan.md` | design judgment; new file/module; public-API change; cross-cutting/multi-file; non-trivial test design | -| **Lightweight** | `change.md` | small-but-real: ≲30 LOC net, ≤2 files, no new file, no public-API change, single straightforward test | +| **Full** | one change file from the design template | design judgment; new file/module; public-API change; cross-cutting/multi-file; non-trivial test design | +| **Lightweight** | one change file from the change template | small-but-real: ≲30 LOC net, ≤2 files, no new file, no public-API change, single straightforward test | | **Tiny** | none — conventional commit | typo, dep bump, linter/formatter/CI tweak, mechanical rename, single-line config | -Heavier lane wins on ambiguity. A `change.md` that outgrows its lane splits -into `design.md` + `plan.md`. +Heavier lane wins on ambiguity. A lightweight change file that outgrows its lane is rewritten from the design template. + +### Plans are ephemeral + +The executable plan — task checklists, embedded code, commit sequences, +whatever the executor needs — is a working artifact, not history. Keep it out +of `changes/` and out of version control (git-ignored scratch, e.g. +`.superpowers/`). Once the change ships, the diff and the PR are the record +of execution; a committed plan duplicates them. `check-planning` rejects +anything in `changes/` that is not a flat change file. + +### Lean specs + +The change file is the single home of a change's rationale: + +- The PR body summarizes and links to the change file — it never restates it. +- Rejected alternatives live in `decisions/` and are referenced, not retold. +- Show a sketch when the design needs code; never the full diff-to-be. +- Delete template sections that don't apply — an empty section is ceremony. +- Most designs fit well under ~700 words; length must buy information. ### Artifacts at a glance -- **`design.md`** — the spec: the *thinking* (why, design, trade-offs, scope). -- **`plan.md`** — the plan: the *sequencing* (the executor's task checklist). -- **`change.md`** — both, condensed, for the lightweight lane. -- **`decisions/-.md`** — one file per design decision taken - (especially options *rejected*), each with a revisit trigger, so reviews don't - re-litigate them; listed by `just index`. +- **design template** — the spec: the *thinking* (why, design, trade-offs, + scope); the change file it produces is the single home of rationale (see + [Lean specs](#lean-specs)). +- **change template** — the condensed spec for the lightweight lane. - **`releases/.md`** — per-release user-facing notes. - **`audits/-.md`** — findings from a code/docs/bug-hunt sweep; spawns fix changes. - **`retros/-.md`** — what we learned after a body of work. - **`deferred.md`** — real-but-unscheduled items, each with a revisit trigger. +- **`decisions/-.md`** — one file per design decision taken + (especially options *rejected*), each with a revisit trigger; listed by + `just index`. Templates live in [`_templates/`](_templates/). ### Frontmatter -`date` and `slug` are **derived from the directory / file name** — never +`date` and `slug` are **derived from the file name** — never repeated in frontmatter. So: -- `design.md` / `change.md`: `summary` (single line) only. -- `plan.md`: **no frontmatter** — its identity is the bundle directory. +- `changes/*.md`: `summary` (single line) only. - `decisions/*.md`: `status` (accepted|superseded), `summary`, and optional `supersedes` / `superseded_by`. - Files in `architecture/` carry **no** frontmatter — living prose, dated by git. @@ -85,7 +151,7 @@ only field the index renders. The listing is **generated**, not maintained — run `just index` to print it: changes newest-first, then decisions newest-first. The `summary` in each -bundle / decision file is the single source of truth; there is no committed +change file / decision file is the single source of truth; there is no committed copy to drift. ## Other From b45f48ac60d8a7dd946c3cc08f69d4a60d0755a8 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Wed, 8 Jul 2026 10:38:40 +0300 Subject: [PATCH 4/5] chore(planning): bump convention to 2.0.0 Co-Authored-By: Claude Opus 4.8 (1M context) --- planning/.convention-version | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/planning/.convention-version b/planning/.convention-version index 3eefcb9..227cea2 100644 --- a/planning/.convention-version +++ b/planning/.convention-version @@ -1 +1 @@ -1.0.0 +2.0.0 From 038cdcf26ef26511d9aa066c5df94cc77c165afa Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Wed, 8 Jul 2026 12:19:22 +0300 Subject: [PATCH 5/5] docs: retire "change bundle" wording in living-truth prose Promotion-rule prose now says "change file" (2.0.0 flat-file model). Co-Authored-By: Claude Opus 4.8 (1M context) --- architecture/README.md | 2 +- planning/deferred.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/architecture/README.md b/architecture/README.md index cf1ab20..eec4a4a 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -31,5 +31,5 @@ These files carry **no frontmatter** — they are prose, dated by git. ## Promotion rule Shipping a change hand-edits the affected capability file(s) here to match the -new reality, in the same PR as the code. The change bundle stays in place under +new reality, in the same PR as the code. The change file stays in place under [`../planning/changes/`](../planning/changes/) — no folder move. diff --git a/planning/deferred.md b/planning/deferred.md index 5e02e69..f63fc7c 100644 --- a/planning/deferred.md +++ b/planning/deferred.md @@ -3,7 +3,7 @@ Items raised in reviews or audits that are real but not actionable now. Each is parked here with the reason it's deferred and the concrete trigger that should bring it back. This is the long-tail register — not a backlog -of planned work. When an item is picked up it graduates to a change bundle +of planned work. When an item is picked up it graduates to a change file in [`changes/active/`](changes/active/); see [CLAUDE.md](../CLAUDE.md#workflow). ## Open