|
2 | 2 |
|
3 | 3 | **Branch:** `mollifier-phase-3` (continuation) |
4 | 4 | **Date:** 2026-05-19 |
5 | | -**Status:** Q1, Q2, Q3, Q4, Q5 all locked. Endpoint inventory complete. Ready for TDD implementation. |
| 5 | +**Status:** Q1, Q2, Q3, Q4, Q5 all locked. Endpoint inventory complete. **Phase A complete.** Phase B is the next chunk. |
| 6 | + |
| 7 | +## Progress tracking |
| 8 | + |
| 9 | +> Always update this section after each phase commits, so a fresh session can resume cleanly without rereading every git log entry. |
| 10 | +
|
| 11 | +| Phase | Status | Commits | Notes | |
| 12 | +|---|---|---|---| |
| 13 | +| Merge of origin/main | ✅ Done | `8c01cf0eb` | 8 conflicts resolved; phase-3 versions kept; picked up one doc comment from main about shadow-mode counter writes | |
| 14 | +| Design docs + parity script | ✅ Done | `c8d036aa0` | 6 plan docs + `scripts/mollifier-api-parity.sh` | |
| 15 | +| **Phase A — read endpoints** | ✅ **Done** | `6b8a54e43`, `e21dbee5e` | See "Phase A patterns established" below | |
| 16 | +| Phase B — shared infrastructure | ⏳ Next | — | ZSET migration, drainer ack semantics, mutateSnapshot Lua, helpers | |
| 17 | +| Phase C — mutation endpoints | ⏳ Pending | — | cancel first (drives B), then tags/metadata-put/reschedule/replay | |
| 18 | +| Phase D — dashboard internals | ⏳ Pending | — | reuse C paths | |
| 19 | +| Phase E — listing endpoints | ⏳ Pending | — | Q1 design | |
| 20 | +| Phase F — test surface lockdown | ⏳ Pending | — | strict parity script + integration tests | |
| 21 | + |
| 22 | +## Phase A patterns established (reference for B/C/D) |
| 23 | + |
| 24 | +Six read endpoints implemented in A1-A6. Three got new code, two needed nothing, one had a pre-existing route bug fixed: |
| 25 | + |
| 26 | +| # | Endpoint | Implementation | Pattern used | |
| 27 | +|---|---|---|---| |
| 28 | +| A1 | `GET /api/v1/runs/{id}/trace` | `findResource` discriminated union → empty trace shape for buffered | New pattern (see below) | |
| 29 | +| A2 | `GET /api/v1/runs/{id}/spans/{spanId}` | Same discriminated union → minimal span shape if spanId matches snapshot, 404 otherwise | Same as A1 | |
| 30 | +| A3 | `GET /api/v1/runs/{id}/events` | **No change** — works via `ApiRetrieveRunPresenter.findRun`'s existing buffer fallback; querying events for a buffered traceId returns `{events:[]}` naturally | Inherits existing infra | |
| 31 | +| A4 | `GET /api/v1/runs/{id}/result` | **No change** — existing 404 message "Run either doesn't exist or is not finished" already covers buffered (not-in-PG) and PG-delayed (not-finished) cases | No-op | |
| 32 | +| A5 | `GET /api/v1/runs/{id}/attempts` | Added missing `loader` (route only had `action`); returns `{attempts:[]}` for both PG and buffered | New loader + parity stub | |
| 33 | +| A6 | `GET /api/v1/runs/{id}/metadata` | Same: added missing `loader`; returns `{metadata, metadataType}` from PG or buffer snapshot | New loader + buffer probe | |
| 34 | + |
| 35 | +### The discriminated union pattern (for A1, A2, and reusable for Phase B/C/D mutations) |
| 36 | + |
| 37 | +```ts |
| 38 | +type ResolvedRun = |
| 39 | + | { source: "pg"; run: <Prisma TaskRun shape> } |
| 40 | + | { source: "buffer"; run: NonNullable<Awaited<ReturnType<typeof findRunByIdWithMollifierFallback>>> }; |
| 41 | + |
| 42 | +findResource: async (params, auth): Promise<ResolvedRun | null> => { |
| 43 | + const pgRun = await $replica.taskRun.findFirst({...}); |
| 44 | + if (pgRun) return { source: "pg", run: pgRun }; |
| 45 | + |
| 46 | + const buffered = await findRunByIdWithMollifierFallback({ |
| 47 | + runId, environmentId: auth.environment.id, organizationId: auth.environment.organizationId, |
| 48 | + }); |
| 49 | + if (buffered) return { source: "buffer", run: buffered }; |
| 50 | + return null; |
| 51 | +} |
| 52 | + |
| 53 | +authorization.resource: (resolved) => { |
| 54 | + if (resolved.source === "pg") { /* existing PG-shape resources */ } |
| 55 | + else { /* synthetic from SyntheticRun shape (no batchId; tags from buffered.tags) */ } |
| 56 | +} |
| 57 | + |
| 58 | +handler: async ({ resource: resolved }) => { |
| 59 | + if (resolved.source === "buffer") { |
| 60 | + // synthesise endpoint-specific empty/minimal shape |
| 61 | + return json({...}, { status: 200 }); |
| 62 | + } |
| 63 | + // existing PG handler logic |
| 64 | +} |
| 65 | +``` |
| 66 | + |
| 67 | +**Important detail:** `SyntheticRun` (in `apps/webapp/app/v3/mollifier/readFallback.server.ts`) lacks a `batchId` field. Buffered runs have no `batch` (batchTrigger bypasses the gate by design). The authorization branch for buffer source must not include batch resources. |
| 68 | + |
| 69 | +### What's NOT in `SyntheticRun` today |
| 70 | + |
| 71 | +If Phase B/C endpoints need additional fields from the buffer snapshot, extend `SyntheticRun` in `readFallback.server.ts`. Current fields cover: friendlyId, status, taskIdentifier, createdAt, payload, payloadType, metadata, metadataType, idempotencyKey, idempotencyKeyOptions, isTest, depth, ttl, tags, lockedToVersion, resumeParentOnCompletion, parentTaskRunId, traceId, spanId, parentSpanId, error. Missing: `taskEventStore`, `runtimeEnvironmentId`, `concurrencyKey`, `machinePreset`, `workerQueue`, `realtimeStreamsVersion`, `idempotencyKeyExpiresAt`, `seedMetadata`, `seedMetadataType`, `parentSpanId` etc. needed by various downstream services (replay, etc). |
| 72 | + |
| 73 | +Q2 (replay) explicitly calls out the synthesiser extension — when implementing Phase C5 (replay), extend `SyntheticRun` with the full set of fields `ReplayTaskRunService` reads. |
| 74 | + |
| 75 | +## Phase B — shared infrastructure (NEXT) |
| 76 | + |
| 77 | +Start here. Implements the building blocks that unblock Phase C. Detailed in [`2026-05-19-mollifier-listing-design.md`](2026-05-19-mollifier-listing-design.md) (Q1), [`2026-05-19-mollifier-mutation-race-design.md`](2026-05-19-mollifier-mutation-race-design.md) (Q3), and [`2026-05-19-mollifier-idempotency-design.md`](2026-05-19-mollifier-idempotency-design.md) (Q5). |
| 78 | + |
| 79 | +Order: |
| 80 | + |
| 81 | +- **B1.** ZSET migration in `packages/redis-worker/src/mollifier/buffer.ts`. `acceptMollifierEntry` Lua → `ZADD queue createdAtMicros runId`. `popAndMarkDraining` Lua → `ZPOPMIN`. `requeueMollifierEntry` Lua → `ZADD`. Listing read goes via `ZREVRANGEBYSCORE`. **Forward-compat:** ship drainer-side changes first, API-side second. |
| 82 | +- **B2.** Drainer ack semantics — replace `DEL entry` with atomic `HSET materialised=true; EXPIRE +30s`. Touches `MollifierBuffer.ack` + the underlying Lua. |
| 83 | +- **B3.** `MollifierBuffer.mutateSnapshot(runId, patch)` — atomic Lua. Three return codes: `applied_to_snapshot`, `not_found`, `busy`. Patch types: `append_tags`, `set_metadata`, `set_delay`, `mark_cancelled`. Idempotency-key patch comes in Q5 work. |
| 84 | +- **B4.** Snapshot-to-TaskRun synthesiser extension — extend `SyntheticRun` in `readFallback.server.ts` to include the fields `ReplayTaskRunService` reads (see Q2 doc table). The Phase C5 work depends on this. |
| 85 | +- **B5.** `mutateWithFallback` helper in `apps/webapp/app/v3/mollifier/mutateWithFallback.server.ts`. Signature in Q3 doc (`bufferPatch`, `pgMutation`, `synthesisedResponse`, optional `maxWaitMs`). Composes Lua call + writer-side spin-wait for the busy case. |
| 86 | +- **B6.** Idempotency lookup wiring per Q5 — extend `acceptMollifierEntry` Lua with SETNX on `mollifier:idempotency:{env}:{task}:{key}`; extend ack Lua with DEL of same; add `lookupIdempotency` and `resetIdempotency` methods. |
| 87 | + |
| 88 | +Phase B has no customer-visible API changes by itself. It's the substrate for Phase C. |
| 89 | + |
| 90 | +## Phase C — mutation endpoints (after B) |
| 91 | + |
| 92 | +Order: |
| 93 | + |
| 94 | +- **C1.** Cancel — drives the drainer-bifurcation work in `engine.createCancelledRun` (Q4 design). Hardest first. |
| 95 | +- **C2.** Tags — fixes the live 500 documented in the parity script results. |
| 96 | +- **C3.** Metadata PUT — straight snapshot patch. |
| 97 | +- **C4.** Reschedule — snapshot patch on `delayUntil`; PG-side terminal-status rejection (status !== "DELAYED") inherits naturally via wait-and-bounce. |
| 98 | +- **C5.** Replay — extend `SyntheticRun` (B4), pass synthesised TaskRun to existing `ReplayTaskRunService`. |
| 99 | + |
| 100 | +## Resuming guidance for a fresh session |
| 101 | + |
| 102 | +If context is lost and a new session needs to resume: |
| 103 | + |
| 104 | +1. `git log --oneline -10 mollifier-phase-3` to see what's been done. |
| 105 | +2. Read this master plan's **Progress tracking** section. |
| 106 | +3. For each unfinished phase, read its companion design doc. |
| 107 | +4. The bash parity script (`scripts/mollifier-api-parity.sh`) is the integration regression guard — run it after each phase to see drift count drop. |
| 108 | +5. The discriminated-union pattern from Phase A is the reference shape for Phase B/C `findResource` work. Don't reinvent. |
| 109 | +6. `SyntheticRun` in `readFallback.server.ts` is the canonical "what fields does the buffer snapshot expose to consumers" type. Extend it (never recreate) when Phase C endpoints need more fields. |
| 110 | +7. **All five Q-docs are locked** — don't relitigate decisions. If a design corner needs revision, update the relevant Q doc + bump the master plan's status line. |
6 | 111 |
|
7 | 112 | ## Why this exists |
8 | 113 |
|
|
0 commit comments