diff --git a/.gitignore b/.gitignore index 81bca7047..9afbcb516 100644 --- a/.gitignore +++ b/.gitignore @@ -148,3 +148,4 @@ archive/ **/.build/ # Codex CLI local state (created when codex runs inside the repo) .codex/ +tests/stress/.stress-state.json diff --git a/.orgii/skills/dual-instance-verification/SKILL.md b/.orgii/skills/dual-instance-verification/SKILL.md index 06548fe1a..f9495385e 100644 --- a/.orgii/skills/dual-instance-verification/SKILL.md +++ b/.orgii/skills/dual-instance-verification/SKILL.md @@ -141,6 +141,13 @@ rollover or the window silently truncates. row (GitHub rename made two spellings one repo network), and the fork guard demanded snapshot == summary while a LIVE source kept growing — equality checks against a moving target are boot-window absence in another costume. +- **A silence that proves nothing**: "zero rewrites since the fix" was true + while the session was not being pushed at all (machine slept, ingest + follows the open view). An absence metric needs a liveness metric beside + it: assert the thing you want CONSTANT (ledger epoch) against the thing + that must still be MOVING (events_count / updated_at). Same shape as + watchdog-masked completion — silence and health look identical until you + measure both. - **Waiting for a pass the engine will never run**: the session plane follows visible-org demand — an org's push/retract pass runs only while that org is the active workspace. A fix whose cleanup rides "the next pass" looks diff --git a/docs/sharing-scalability-plan-2026-07-25.md b/docs/sharing-scalability-plan-2026-07-25.md new file mode 100644 index 000000000..366cf65ba --- /dev/null +++ b/docs/sharing-scalability-plan-2026-07-25.md @@ -0,0 +1,165 @@ +# Session Sharing — Scalability / Maintainability Campaign + +Branch `codex/sharing-scale-maintain` off develop @ `e47f4b74b` (PR #535 merged). +Correctness matrix is green (14 fixes, dual-instance verified); this campaign +attacks the SCALE gaps named in the 535 handoff. Ordered by measured evidence. + +## P0 — Freeze-line mutation horizon (write amplification, evidence in hand) + +**Symptom (measured 2026-07-25):** a live claudecodeapp session logged repeated +`epoch rewrite … chainMismatch=true` (frozen=3196 cursorFrozen=3185): each +rewrite re-uploads the ENTIRE frozen history (3000+ events, MBs with +screenshots), several times per hour per live session. + +**Root cause:** `computeFrozenEventCount` freezes every event up to the first +non-terminal `displayStatus`. The Claude Code ingest AMENDS recently-terminal +events afterwards (`es_merge_events` tool-result backfill, +`es_remove_synthetic_user_inputs`), so terminal ≠ immutable. Any amendment to +a frozen event breaks the prefix chain (the check at +`org2CloudSessionSync.ts:457-483` is correct — the data genuinely changed) and +forces the expensive epoch-rewrite path. + +**Fix:** hold back a mutation horizon from freezing while the session is +live — do not freeze the trailing N events (and/or events younger than T +minutes) even when terminal, so ingest amendments land in the mutable tail +(one small segment re-upload) instead of invalidating frozen history. On +quiescence/completion the line advances to the end — steady-state output +identical to today. Constraints: keep frozenEventCount monotonic vs cursor +(shrink path unchanged); respect the 256KB tail-segment budget when choosing +N (screenshot-bearing events are large — budget-aware holdback, not a fixed +count). Verify: rewrite-frequency before/after on a live session +(`grep -c "epoch rewrite"` per hour), plus the existing segment tests. + +**Status: SHIPPED (`f3ccbe2dc`) and verified live.** 20-minute window on the +same live session, ingest actively tailing: three push waves grew the cloud +row 3762 → 3851 events with `events_epoch` constant at 19 and zero +`epoch rewrite` lines. Morning baseline on the pre-fix build was 10 +rewrites in one hour of comparable activity (2/4/10/1 per hour across the +07:00–11:00 span). The one rewrite after deploy was the predicted one-time +re-anchor when the lowered line first disagreed with the persisted cursor. + +## P1 — Org listing latency + +**Status: CLOSED by measurement — premise refuted, no migration needed.** +Seeded a test org to 506 live rows (service role, cleaned up after) and +timed `cloud_list_org_sessions` as an authenticated member, 3 runs per +size: 9 rows ≈ 0.28s, ~50 rows ≈ 0.28s, 506 rows ≈ 0.28s. Latency is FLAT +across a 56× row spread: the ~0.3-0.5s seen in the perf panel is fixed +cost (network RTT + PostgREST/auth), not per-row work. An index/LATERAL +migration would have optimized a term that does not move. Revisit only +past ~5k rows per org or if the panel shows growth with org size. + +## P2 — Inactive-org consistency + +**Status: SHIPPED and verified end-to-end.** Once per engine run, orgs the +user is NOT looking at that hold LOCAL push markers get a retract-only +reconcile: same `decidePushAdmission`, same server-confirmed scope +boundary, only rows this client can prove it pushed; locally-absent +sessions stay with the vanished sweep's two-strike verdict. Cost per org: +one TTL-shared scope fetch plus local work — no pushes, no replay, no +listing RPC. Verified on-device: a resurrected subagent row in a +background org was retracted at boot with zero workspace switches, the +tombstone timestamp matching the reconcile log line to the second; the +active org's live session and every other marked row untouched. + +## P3 — Server hygiene + +**Status: SHIPPED — 0009 applied to the live project and verified** (zero +live subagent rows anywhere; all 50 rows in the deleted org tombstoned +with the org's original deletion timestamp). +Org soft-delete now cascades a tombstone to the org's sessions in the same +transaction, plus a one-time idempotent backfill for the 45 orphans already +stranded in deleted org `6c14735d`. Deleted-org rows are invisible to +clients, so this was quota/storage hygiene, not correctness. + +## P4 — Maintainability (architecture-audit per CLAUDE.md before refactor) + +**First seam landed (`dc240ab1b`):** `decidePushAdmission` in +`org2CloudPushAdmission.ts` is now the single definition of the admission +rules (fork provenance + ownership gate), returning the denial reason the +retract log prints. This is the prerequisite for P2 — a reconcile that +retracts under different rules than the push admitted under is a bug +generator. Ten tests; removed `floorEligible`, a disjunction that was +constant-true at the only point it was read. + +Remaining seams (unstarted): retract executor, pass scheduler. Do NOT mix +refactor commits with P0-P3 behavior fixes. + +## Explicitly out of scope this campaign + +Home-endpoint sharding cutover (0007 scaffolding), same-account multi-device +(needs a third signed install), load tests beyond the P1 seeding. + +## Member-scale stress (workstream #1, harness: tests/stress/orgFanout.mjs) + +Live-backend run, 50 real auth users in one team-plan org (2026-07-25): + +- **Realtime fan-out** (the app's actual channel — client-sent broadcast + nudges on `presence:org:`): 50/50 subscribed, **49/49 receivers + delivered — 100%, zero loss** (four consecutive runs; the one "missing" + client was always index 0, the writer itself, excluded by broadcast + `self:false` — the earlier 2%-loss reading was a harness denominator + artifact). Harness timestamps include RPC+settle overhead; net + propagation is sub-second. +- **Concurrent comments**: 50 simultaneous `cloud_add_session_comment` + writers — 50/50 ok, zero errors, p50 1.0s / p95 1.15s under full + contention (single-writer baseline ≈0.28s). +- **Concurrent owner pushes**: 20 parallel metadata upserts — 20/20 ok, + p50 370ms. +- **Invite contention**: 49 accepts against ONE invite code — 49/49 ok. + +**Real gates found ahead of performance:** + +1. **GoTrue sign-in rate limiting** — ~30 password grants per window per + IP; a 50-member 9am sign-in storm hits 429s (harness needed backoff + + refresh-token reuse). Project auth rate config, not app code. +2. **Entitlements** — free plan caps `maxOrgMembers` at 3; member scale + is plan-bound before it is ever load-bound (worked as designed; the + stress org runs on a `team` org_entitlements override). + +Stress fixtures (50 `@org2-stress.invalid` users, org `0830d453`, 18 +boot-cost orgs) are LIVE for reuse; `node tests/stress/orgFanout.mjs +cleanup` tears the fan-out set down. + +## Org-count boot cost (workstream #3) + +Boot pass span (first engine log line → background reconcile complete), +build 25, one run per point, stress orgs created/deleted around the +measurement: 9 orgs 5.7s · 16 orgs 7.2s · 26 orgs 9.5s. **Linear** at +~0.22s/org (serialized, RTT-bound per-org RPC chain) over a ~3.7s fixed +base — no super-linear term. Extrapolated 50-org boot ≈ 15s. No action +needed at current scale; the lever, if ever needed, is parallelizing the +per-org chain, not indexing. + +## Sharding Phase A (workstream #4) + +**Status: routing layer SHIPPED.** `org2CloudOrgEndpointRouter` holds the +per-pass directory the engine now publishes from the roster's resolved +`homeEndpoint`s (0007); the sync client's org-scoped data-plane calls +(metadata upsert, append/rewrite, list, session events readers, scopes, +delete) resolve their endpoint per org with the official project as the +universal fallback. Guest-share reads keep their explicit endpoint +precedence. Verified: unit tests prove divergent routing + republish +semantics; full suite (6361) and an on-device run prove byte-identical +behavior while the directory is identity (all orgs on the official +project) — pushes flowed, zero errors, P0 epoch still constant. + +**Phase B (needs user-provisioned infra):** a real second Supabase +project (schema 0001-0009 applied), an anon-key directory keyed by +endpoint origin, `cloud_set_org_home_endpoint` cutover of one test org, +and the cross-project live proof. The client half is now ready for it. + +## Sustained write pressure (harness `sustain` mode, 2026-07-26) + +10 writers × 5 minutes at 2s cadence against per-writer sessions: +~1310 comment writes, **zero errors, no latency drift** (p50 flat at +254-328ms, p95 330-470ms across every minute). Control-plane writes hold +steady under sustained pressure; full segment-protocol sustain remains +an app-level exercise. + +**Product finding along the way:** the 0001 abuse guard caps comments at +500 per session counting live AND tombstoned rows — deleting comments +does not return budget, so a long-lived high-activity team session can +permanently exhaust its comment quota. Working as designed today; worth +a product decision (tombstone-excluding count, or per-window cap) before +50-member orgs live in one session for months. diff --git a/src/features/Org2Cloud/org2CloudBackendAdapter.test.ts b/src/features/Org2Cloud/org2CloudBackendAdapter.test.ts index 598e3e023..f7acce4e5 100644 --- a/src/features/Org2Cloud/org2CloudBackendAdapter.test.ts +++ b/src/features/Org2Cloud/org2CloudBackendAdapter.test.ts @@ -8,6 +8,7 @@ import { toFrozenSegmentWire, toTailWire, } from "../TeamCollaboration/sync/segmentCodec"; +import { getCloudEndpoint } from "./config"; import { buildCloudSessionFetchClient, cloudSessionIdFromRowId, @@ -135,10 +136,12 @@ describe("buildCloudSessionFetchClient", () => { }); expect(downloadReplayObjectMock).toHaveBeenCalledTimes(1); + // Member downloads route per org now; with an identity directory the + // resolved endpoint IS the official one. expect(downloadReplayObjectMock).toHaveBeenCalledWith( "jwt-token", storagePath, - undefined, + getCloudEndpoint(), undefined ); const [seg1, seg2, tailSeg] = snapshot.segments; diff --git a/src/features/Org2Cloud/org2CloudBackendAdapter.ts b/src/features/Org2Cloud/org2CloudBackendAdapter.ts index b2bd288bb..2b773a836 100644 --- a/src/features/Org2Cloud/org2CloudBackendAdapter.ts +++ b/src/features/Org2Cloud/org2CloudBackendAdapter.ts @@ -39,6 +39,7 @@ import { mapSegmentsBounded, } from "../TeamCollaboration/sync/segmentCodec"; import type { CloudEndpoint } from "./config"; +import { endpointForOrg } from "./org2CloudOrgEndpointRouter"; import { type GuestReplayObjectReader, createGuestReplayObjectReader, @@ -144,7 +145,12 @@ export function buildCloudSessionFetchClient( const shareToken = input.shareToken; if (shareToken === undefined) { return (storagePath, signal) => - downloadReplayObject(accessToken, storagePath, endpoint, signal); + downloadReplayObject( + accessToken, + storagePath, + endpoint ?? endpointForOrg(input.orgId), + signal + ); } const sessionId = cloudSessionIdFromRowId(input.sessionRowId); const readerKey = `${input.orgId}\u001f${sessionId}\u001f${shareToken}`; diff --git a/src/features/Org2Cloud/org2CloudOrgEndpointRouter.test.ts b/src/features/Org2Cloud/org2CloudOrgEndpointRouter.test.ts new file mode 100644 index 000000000..fc06644bd --- /dev/null +++ b/src/features/Org2Cloud/org2CloudOrgEndpointRouter.test.ts @@ -0,0 +1,39 @@ +import { afterEach, describe, expect, it } from "vitest"; + +import { getCloudEndpoint } from "./config"; +import { + endpointForOrg, + resetOrgEndpointDirectory, + setAnonKeyDirectory, + setOrgEndpointDirectory, +} from "./org2CloudOrgEndpointRouter"; + +afterEach(() => resetOrgEndpointDirectory()); + +describe("org endpoint router", () => { + it("falls back to the official endpoint for unknown orgs", () => { + expect(endpointForOrg("nope")).toEqual(getCloudEndpoint()); + }); + + it("routes a directory entry and drops it on republish", () => { + const home = { + ...getCloudEndpoint(), + supabaseUrl: "https://shard.example", + }; + setOrgEndpointDirectory([["org-a", home]]); + expect(endpointForOrg("org-a").supabaseUrl).toBe("https://shard.example"); + setOrgEndpointDirectory([]); + expect(endpointForOrg("org-a")).toEqual(getCloudEndpoint()); + }); + + it("swaps the anon key for origins with a shard key, else falls back", () => { + const home = { + ...getCloudEndpoint(), + supabaseUrl: "https://shard.example", + }; + setOrgEndpointDirectory([["org-a", home]]); + setAnonKeyDirectory([["https://shard.example", "shard-anon-key"]]); + expect(endpointForOrg("org-a").anonKey).toBe("shard-anon-key"); + expect(endpointForOrg("unrouted").anonKey).toBe(getCloudEndpoint().anonKey); + }); +}); diff --git a/src/features/Org2Cloud/org2CloudOrgEndpointRouter.ts b/src/features/Org2Cloud/org2CloudOrgEndpointRouter.ts new file mode 100644 index 000000000..900cdd3b4 --- /dev/null +++ b/src/features/Org2Cloud/org2CloudOrgEndpointRouter.ts @@ -0,0 +1,50 @@ +/** + * Org → data-plane endpoint router (sharding Phase A — wires the 0007 + * directory that `org2CloudEndpointDirectory` resolves but nothing + * consumed). The engine publishes the roster's resolved endpoints here + * each pass; org-scoped data-plane calls ask `endpointForOrg` instead of + * assuming the official project. + * + * Fallback posture: an org with no entry — or whose `homeEndpoint` failed + * the https-origin validation upstream — routes to the official endpoint, + * so a pre-0007 backend or an empty directory behaves exactly as before. + * The directory is process-local state, not persisted: it is rebuilt from + * the roster on every pass, so a cutover (or rollback) takes effect on + * the next pass without restart. + */ +import type { CloudEndpoint } from "./config"; +import { getCloudEndpoint } from "./config"; + +const orgEndpoints = new Map(); +/** Shard anon keys by https origin — per-project, not per-org. Empty until + * a Phase B cutover publishes real shard keys; a routed origin with no key + * falls back to the official anon key (harmless: the shard rejects it, + * which fails closed rather than silently mixing projects). */ +const anonKeyByOrigin = new Map(); + +export function setAnonKeyDirectory( + entries: ReadonlyArray +): void { + anonKeyByOrigin.clear(); + for (const [origin, anonKey] of entries) anonKeyByOrigin.set(origin, anonKey); +} + +export function setOrgEndpointDirectory( + entries: ReadonlyArray +): void { + orgEndpoints.clear(); + for (const [orgId, endpoint] of entries) { + orgEndpoints.set(orgId, endpoint); + } +} + +export function endpointForOrg(orgId: string): CloudEndpoint { + const endpoint = orgEndpoints.get(orgId) ?? getCloudEndpoint(); + const shardKey = anonKeyByOrigin.get(endpoint.supabaseUrl); + return shardKey ? { ...endpoint, anonKey: shardKey } : endpoint; +} + +export function resetOrgEndpointDirectory(): void { + orgEndpoints.clear(); + anonKeyByOrigin.clear(); +} diff --git a/src/features/Org2Cloud/org2CloudPushAdmission.test.ts b/src/features/Org2Cloud/org2CloudPushAdmission.test.ts new file mode 100644 index 000000000..b3fad1bf0 --- /dev/null +++ b/src/features/Org2Cloud/org2CloudPushAdmission.test.ts @@ -0,0 +1,117 @@ +import { describe, expect, it } from "vitest"; + +import { + PUSH_ADMISSION_DENIAL, + decidePushAdmission, +} from "./org2CloudPushAdmission"; + +const ORG = "org-a"; + +function inputs(overrides: Partial[0]>) { + return { + orgId: ORG, + session: { session_id: "claudecodeapp-1" }, + forkedFrom: undefined, + tagged: false, + ownedByOrg: false, + shareIntent: false, + ...overrides, + } as Parameters[0]; +} + +describe("decidePushAdmission", () => { + it("admits a session explicitly owned by the org", () => { + expect(decidePushAdmission(inputs({ ownedByOrg: true }))).toEqual({ + admitted: true, + }); + }); + + it("admits a tagged session", () => { + expect(decidePushAdmission(inputs({ tagged: true }))).toEqual({ + admitted: true, + }); + }); + + it("admits an explicit share intent", () => { + expect(decidePushAdmission(inputs({ shareIntent: true }))).toEqual({ + admitted: true, + }); + }); + + it("admits an imported history matched by repo scope", () => { + expect( + decidePushAdmission( + inputs({ + session: { + session_id: "claudecodeapp-1", + repoPath: "/Users/me/org2", + repoRemoteUrls: ["git@github.com:org2ai/org2.git"], + }, + }) + ) + ).toEqual({ admitted: true }); + }); + + it("denies a spawned child even inside a scoped checkout", () => { + // Subagent transcripts fold into their parent; publishing them floods + // the team list with rows the receiving side cannot regroup. + expect( + decidePushAdmission( + inputs({ + session: { + session_id: "claudecodeapp-agent-a5", + repoPath: "/Users/me/org2", + repoRemoteUrls: ["git@github.com:org2ai/org2.git"], + parentSessionId: "claudecodeapp-1", + }, + }) + ) + ).toEqual({ + admitted: false, + denial: PUSH_ADMISSION_DENIAL.OWNERSHIP_GATE, + }); + }); + + it("denies an ordinary session with no ownership, tag, intent, or match", () => { + expect(decidePushAdmission(inputs({}))).toEqual({ + admitted: false, + denial: PUSH_ADMISSION_DENIAL.OWNERSHIP_GATE, + }); + }); + + it("admits a fork back to its own source org", () => { + expect(decidePushAdmission(inputs({ forkedFrom: { orgId: ORG } }))).toEqual( + { admitted: true } + ); + }); + + it("denies an untagged fork in any other org", () => { + expect( + decidePushAdmission(inputs({ forkedFrom: { orgId: "org-b" } })) + ).toEqual({ + admitted: false, + denial: PUSH_ADMISSION_DENIAL.FORK_OUTSIDE_SOURCE_ORG, + }); + }); + + it("lets an explicit tag override fork provenance for that org only", () => { + expect( + decidePushAdmission( + inputs({ forkedFrom: { orgId: "org-b" }, tagged: true }) + ) + ).toEqual({ admitted: true }); + }); + + it("keeps a tagged fork admitted without consulting the scope route", () => { + // A fork carries no imported-history identity; the ownership gate must + // not re-deny it after provenance already admitted it. + expect( + decidePushAdmission( + inputs({ + forkedFrom: { orgId: ORG }, + session: { session_id: "agentsession-1" }, + }) + ) + ).toEqual({ admitted: true }); + }); +}); diff --git a/src/features/Org2Cloud/org2CloudPushAdmission.ts b/src/features/Org2Cloud/org2CloudPushAdmission.ts new file mode 100644 index 000000000..8692a9de5 --- /dev/null +++ b/src/features/Org2Cloud/org2CloudPushAdmission.ts @@ -0,0 +1,79 @@ +/** + * Pure push-admission decision: may THIS session publish to THIS org, and + * if not, why. Extracted from the engine's pass loop so the same rules can + * be replayed outside a full pass (retraction reconcile, tests, audit) + * without duplicating them — a second copy of these rules is how a session + * ends up published under one predicate and retracted under another. + * + * Ordering mirrors the engine: fork provenance first (a fork is a + * continuation inside its source boundary, not an ordinary repo session), + * then the ownership gate (explicit ownership / tag / share intent / + * repo-scope auto-match). Repo-scope KEY matching stays in the caller: it + * needs the async resolver cache and its `undefined` (in-flight) state, + * which has no meaning in a pure decision. + */ +import type { Session } from "@src/store/session/sessionAtom/types"; + +import { isScopeMatchableImportedSession } from "../TeamCollaboration/importedSessionScopeMatch"; + +export const PUSH_ADMISSION_DENIAL = { + /** Untagged fork whose provenance points at a different org. */ + FORK_OUTSIDE_SOURCE_ORG: "fork outside source org", + /** No ownership, tag, share intent, or scope auto-match for this org. */ + OWNERSHIP_GATE: "ownership-gate (untagged/unowned/no-intent)", +} as const; + +export type PushAdmissionDenial = + (typeof PUSH_ADMISSION_DENIAL)[keyof typeof PUSH_ADMISSION_DENIAL]; + +export interface PushAdmissionInputs { + /** The org this decision is about. */ + orgId: string; + session: Pick< + Session, + "session_id" | "orgId" | "repoPath" | "repoRemoteUrls" | "parentSessionId" + >; + /** Durable fork provenance, already resolved through the registry. */ + forkedFrom: { orgId: string } | undefined; + /** LIVE tag state for (session, org) — never a pass-start snapshot. */ + tagged: boolean; + /** `session.orgId` parses to this org's canonical selector. */ + ownedByOrg: boolean; + /** An explicit, non-off sharing-ladder entry exists for this session. */ + shareIntent: boolean; +} + +export type PushAdmission = + | { admitted: true } + | { admitted: false; denial: PushAdmissionDenial }; + +export function decidePushAdmission( + inputs: PushAdmissionInputs +): PushAdmission { + const { orgId, session, forkedFrom, tagged, ownedByOrg, shareIntent } = + inputs; + + // An untagged fork publishes ONLY back to its source org; an explicit tag + // overrides provenance for that org alone. + if (forkedFrom && forkedFrom.orgId !== orgId && !tagged) { + return { + admitted: false, + denial: PUSH_ADMISSION_DENIAL.FORK_OUTSIDE_SOURCE_ORG, + }; + } + + if ( + !forkedFrom && + !tagged && + !ownedByOrg && + !shareIntent && + !isScopeMatchableImportedSession(session) + ) { + return { + admitted: false, + denial: PUSH_ADMISSION_DENIAL.OWNERSHIP_GATE, + }; + } + + return { admitted: true }; +} diff --git a/src/features/Org2Cloud/org2CloudSyncClient.ts b/src/features/Org2Cloud/org2CloudSyncClient.ts index 80c892869..e1111cc35 100644 --- a/src/features/Org2Cloud/org2CloudSyncClient.ts +++ b/src/features/Org2Cloud/org2CloudSyncClient.ts @@ -40,6 +40,7 @@ import { fetchWithTransportRetry, runCloudRequestWithTimeout, } from "./org2CloudFetchRetry"; +import { endpointForOrg } from "./org2CloudOrgEndpointRouter"; import { buildReplayObjectPath, uploadReplayObject, @@ -343,9 +344,12 @@ export async function getOrgRepoScopes( accessToken: string, orgId: string ): Promise { - const payload = await callSyncRpc("cloud_get_org_repo_scopes", accessToken, { - p_org_id: orgId, - }); + const payload = await callSyncRpc( + "cloud_get_org_repo_scopes", + accessToken, + { p_org_id: orgId }, + endpointForOrg(orgId) + ); return CloudOrgScopeStateSchema.parse(payload); } @@ -410,11 +414,16 @@ export async function upsertSessionMetadata( sessionId: string, metadata: RemoteTeammateSessionMetadata ): Promise { - await callSyncRpc("cloud_upsert_session_metadata", accessToken, { - p_org_id: orgId, - p_session_id: sessionId, - metadata, - }); + await callSyncRpc( + "cloud_upsert_session_metadata", + accessToken, + { + p_org_id: orgId, + p_session_id: sessionId, + metadata, + }, + endpointForOrg(orgId) + ); } export interface CloudRewriteSessionEventsInput { @@ -431,7 +440,7 @@ export async function rewriteSessionEvents( accessToken: string, input: CloudRewriteSessionEventsInput ): Promise { - const endpoint = getCloudEndpoint(); + const endpoint = endpointForOrg(input.orgId); const baseBody = { p_org_id: input.orgId, p_session_id: input.sessionId, @@ -497,7 +506,7 @@ export async function appendSessionEvents( accessToken: string, input: CloudAppendSessionEventsInput ): Promise { - const endpoint = getCloudEndpoint(); + const endpoint = endpointForOrg(input.orgId); const baseBody = { p_org_id: input.orgId, p_session_id: input.sessionId, @@ -553,7 +562,7 @@ export async function listOrgSessions( since?: string, signal?: AbortSignal ): Promise { - const endpoint = getCloudEndpoint(); + const endpoint = endpointForOrg(orgId); const legacyCall = async () => { const payload = await callSyncRpc( "cloud_list_org_sessions", @@ -766,7 +775,7 @@ async function streamSessionEventsPaged( ? { p_share_token: options.shareToken } : {}), }, - options?.endpoint, + options?.endpoint ?? endpointForOrg(orgId), options?.signal ); const parsed = CloudSessionEventsPageSchema.parse(payload); @@ -821,7 +830,7 @@ async function getSessionEventsLegacy( ? { p_after_seq: options.afterSeq } : {}), }, - options?.endpoint, + options?.endpoint ?? endpointForOrg(orgId), options?.signal ); const parsed = CloudSessionEventsSchema.parse(payload); @@ -846,8 +855,10 @@ export async function deleteSession( orgId: string, sessionId: string ): Promise { - await callSyncRpc("cloud_delete_session", accessToken, { - p_org_id: orgId, - p_session_id: sessionId, - }); + await callSyncRpc( + "cloud_delete_session", + accessToken, + { p_org_id: orgId, p_session_id: sessionId }, + endpointForOrg(orgId) + ); } diff --git a/src/features/Org2Cloud/org2CloudSyncEngine.retractReconcile.test.ts b/src/features/Org2Cloud/org2CloudSyncEngine.retractReconcile.test.ts new file mode 100644 index 000000000..f78f7606d --- /dev/null +++ b/src/features/Org2Cloud/org2CloudSyncEngine.retractReconcile.test.ts @@ -0,0 +1,123 @@ +import { createStore } from "jotai"; +import { describe, expect, it, vi } from "vitest"; + +import { sessionsAtom } from "@src/store/session/sessionAtom/atoms"; +import type { Session } from "@src/store/session/sessionAtom/types"; + +import { sessionOrgTagsAtom } from "../TeamCollaboration/sessionOrgTagsAtom"; +import { + org2CloudPushCursorsAtom, + org2CloudPushedMetadataAtom, + org2CloudRepoScopesAtom, +} from "./org2CloudSyncAtoms"; +import { + markedSessionIdsForOrg, + orgsWithLocalPushMarkers, + reconcileOrgRetracts, +} from "./org2CloudSyncEngine.retractReconcile"; + +const ORG = "11111111-1111-4111-8111-111111111111"; + +describe("push-marker enumeration", () => { + it("collects org ids and session ids from both marker atoms", () => { + const cursors = { [`${ORG}:sess-a`]: {} }; + const meta = { [`${ORG}:sess-b`]: true, "other-org:sess-c": true }; + expect(orgsWithLocalPushMarkers(cursors, meta)).toEqual( + new Set([ORG, "other-org"]) + ); + expect(markedSessionIdsForOrg(ORG, cursors, meta)).toEqual( + new Set(["sess-a", "sess-b"]) + ); + }); +}); + +function session(overrides: Partial): Session { + return { + session_id: "claudecodeapp-x", + repoPath: "/Users/me/org2", + repoRemoteUrls: ["git@github.com:org2ai/org2.git"], + ...overrides, + } as Session; +} + +function setup(options: { + sessions: Session[]; + marked: string[]; + scopes?: string[]; + serverConfirmed?: boolean; +}) { + const store = createStore(); + store.set(sessionsAtom, options.sessions); + store.set( + org2CloudPushCursorsAtom, + Object.fromEntries( + options.marked.map((id) => [`${ORG}:${id}`, { orgId: ORG } as never]) + ) + ); + store.set(org2CloudPushedMetadataAtom, {}); + store.set(org2CloudRepoScopesAtom, { [ORG]: options.scopes ?? [] }); + const retractSession = vi.fn(async () => undefined); + const deps = { + store: store as never, + accessByOrg: {}, + wasCloudPushed: () => true, + retractSession, + hasServerConfirmedScopes: () => options.serverConfirmed ?? true, + isCurrentGeneration: () => true, + }; + return { deps, retractSession, store }; +} + +describe("reconcileOrgRetracts", () => { + it("retracts a marked session that lost every admission route", async () => { + const { deps, retractSession } = setup({ + sessions: [ + session({ session_id: "personal-1", repoRemoteUrls: undefined }), + ], + marked: ["personal-1"], + }); + await reconcileOrgRetracts(deps, ORG); + expect(retractSession).toHaveBeenCalledWith(ORG, "personal-1"); + }); + + it("retracts an admitted session whose repo left the org scope", async () => { + const { deps, retractSession, store } = setup({ + sessions: [session({ session_id: "claudecodeapp-x" })], + marked: ["claudecodeapp-x"], + scopes: ["github.com/other/repo"], + }); + store.set(sessionOrgTagsAtom, {}); + await reconcileOrgRetracts(deps, ORG); + expect(retractSession).toHaveBeenCalledWith(ORG, "claudecodeapp-x"); + }); + + it("keeps an in-scope session untouched", async () => { + const { deps, retractSession } = setup({ + sessions: [session({ session_id: "claudecodeapp-x" })], + marked: ["claudecodeapp-x"], + scopes: ["github.com/org2ai/org2"], + }); + await reconcileOrgRetracts(deps, ORG); + expect(retractSession).not.toHaveBeenCalled(); + }); + + it("defers the out-of-scope verdict until scopes are server-confirmed", async () => { + const { deps, retractSession } = setup({ + sessions: [session({ session_id: "claudecodeapp-x" })], + marked: ["claudecodeapp-x"], + scopes: ["github.com/other/repo"], + serverConfirmed: false, + }); + await reconcileOrgRetracts(deps, ORG); + expect(retractSession).not.toHaveBeenCalled(); + }); + + it("leaves locally-absent sessions to the vanished-session sweep", async () => { + const { deps, retractSession } = setup({ + sessions: [], + marked: ["gone-1"], + }); + await reconcileOrgRetracts(deps, ORG); + expect(retractSession).not.toHaveBeenCalled(); + }); +}); diff --git a/src/features/Org2Cloud/org2CloudSyncEngine.retractReconcile.ts b/src/features/Org2Cloud/org2CloudSyncEngine.retractReconcile.ts new file mode 100644 index 000000000..b415894c8 --- /dev/null +++ b/src/features/Org2Cloud/org2CloudSyncEngine.retractReconcile.ts @@ -0,0 +1,166 @@ +/** + * Retract-only reconcile for orgs the user is NOT looking at (P2). + * + * The session plane follows visible-org demand: a full pass — and with it + * every retract — runs only for the active workspace. A session that loses + * admission in a background org therefore stays published until the user + * happens to reopen that org, which may be never. This module closes that + * hole with the cheapest sound sweep: once per engine run, for each + * NON-ACTIVE org where THIS client holds persisted push markers, re-run the + * SAME admission decision (`decidePushAdmission`) and the SAME + * server-confirmed scope boundary over the marked sessions, retracting the + * rows this client can prove it pushed. No pushes, no replay hydration, no + * listing RPC — per org this costs one scope fetch (TTL-shared with the + * main pass) plus local work. + * + * Safety rails carried over verbatim from the main pass: + * - only rows with LOCAL push markers are touched — never someone else's; + * - out-of-scope retracts require scopes CONFIRMED from the server this + * run (a stale mirror cannot prove "out of scope"); + * - sessions with scope resolution in flight are skipped, not judged; + * - a locally-ABSENT session is left to the vanished-session sweep and its + * two-strike confirmation — absence is not authority here. + */ +import { createLogger } from "@src/hooks/logger"; +import { sessionsAtom } from "@src/store/session/sessionAtom/atoms"; + +import { getSessionForkedFrom } from "../TeamCollaboration/forkSession"; +import { resolveMatchingOrgRepoScope } from "../TeamCollaboration/repoScopeResolver"; +import { + isSessionTaggedToCloudOrg, + sessionOrgTagsAtom, + withoutCloudOrgTag, +} from "../TeamCollaboration/sessionOrgTagsAtom"; +import { + type CloudOrgAccessSettings, + hasExplicitCloudShareIntent, +} from "./org2CloudAccessSettings"; +import { buildCloudOrgSelectorValue } from "./org2CloudOrgsAtom"; +import { decidePushAdmission } from "./org2CloudPushAdmission"; +import { + org2CloudPushCursorsAtom, + org2CloudPushedMetadataAtom, + org2CloudRepoScopesAtom, +} from "./org2CloudSyncAtoms"; +import { getSessionScopeKeys } from "./org2CloudSyncEngine.repoScopeSync"; +import type { CloudStore } from "./org2CloudSyncLifecycle"; + +const log = createLogger("Org2CloudRetractReconcile"); + +export interface RetractReconcileDeps { + store: CloudStore; + accessByOrg: Record; + wasCloudPushed: (orgId: string, sessionId: string) => boolean; + retractSession: (orgId: string, sessionId: string) => Promise; + hasServerConfirmedScopes: (orgId: string) => boolean; + isCurrentGeneration: () => boolean; +} + +/** Org ids where THIS client holds any persisted push marker. */ +export function orgsWithLocalPushMarkers( + cursors: Record, + pushedMetadata: Record +): Set { + const orgIds = new Set(); + for (const key of [...Object.keys(cursors), ...Object.keys(pushedMetadata)]) { + const separator = key.indexOf(":"); + if (separator > 0) orgIds.add(key.slice(0, separator)); + } + return orgIds; +} + +/** Marked session ids for one org, from both marker atoms. */ +export function markedSessionIdsForOrg( + orgId: string, + cursors: Record, + pushedMetadata: Record +): Set { + const prefix = `${orgId}:`; + const ids = new Set(); + for (const key of [...Object.keys(cursors), ...Object.keys(pushedMetadata)]) { + if (key.startsWith(prefix)) ids.add(key.slice(prefix.length)); + } + return ids; +} + +export async function reconcileOrgRetracts( + deps: RetractReconcileDeps, + orgId: string +): Promise { + const { store } = deps; + const cursors = store.get(org2CloudPushCursorsAtom); + const pushedMetadata = store.get(org2CloudPushedMetadataAtom); + const marked = markedSessionIdsForOrg(orgId, cursors, pushedMetadata); + if (marked.size === 0) return; + + const sessions = store.get(sessionsAtom); + const sessionById = new Map(sessions.map((s) => [s.session_id, s])); + const scopes = store.get(org2CloudRepoScopesAtom)[orgId] ?? []; + + for (const sessionId of marked) { + if (!deps.isCurrentGeneration()) return; + const session = sessionById.get(sessionId); + // Locally absent: the vanished-session sweep owns that verdict (with + // its two-strike confirmation). Absence here proves nothing. + if (!session) continue; + if (!deps.wasCloudPushed(orgId, sessionId)) continue; + + const forkedFrom = getSessionForkedFrom(session); + const tagged = isSessionTaggedToCloudOrg( + store.get(sessionOrgTagsAtom), + sessionId, + orgId + ); + const admission = decidePushAdmission({ + orgId, + session, + forkedFrom, + tagged, + ownedByOrg: session.orgId === buildCloudOrgSelectorValue(orgId), + shareIntent: hasExplicitCloudShareIntent( + deps.accessByOrg[orgId], + sessionId + ), + }); + + if (!admission.admitted) { + try { + log.info( + `reconcile retract [${admission.denial}]: session ${sessionId} org ${orgId}` + ); + await deps.retractSession(orgId, sessionId); + } catch (error) { + log.warn(`reconcile retract failed for ${sessionId}:`, error); + } + continue; + } + + const scopeKeys = getSessionScopeKeys(session); + if (scopeKeys === undefined) continue; + const matchedScope = await resolveMatchingOrgRepoScope(scopeKeys, scopes); + if (matchedScope !== null) continue; + if (!deps.hasServerConfirmedScopes(orgId)) { + log.info( + `reconcile scope check deferred: session ${sessionId} org ${orgId}` + ); + continue; + } + try { + log.info( + `reconcile retract [out-of-scope (no matching org scope)]: session ${sessionId} org ${orgId}` + ); + await deps.retractSession(orgId, sessionId); + } catch (error) { + log.warn(`reconcile retract failed for ${sessionId}:`, error); + continue; + } + if (tagged) { + store.set(sessionOrgTagsAtom, (current) => + withoutCloudOrgTag(current, sessionId, orgId) + ); + log.info( + `reconcile dropped out-of-scope org tag: session ${sessionId} → org ${orgId}` + ); + } + } +} diff --git a/src/features/Org2Cloud/org2CloudSyncEngine.ts b/src/features/Org2Cloud/org2CloudSyncEngine.ts index a0870aeb2..0f696728c 100644 --- a/src/features/Org2Cloud/org2CloudSyncEngine.ts +++ b/src/features/Org2Cloud/org2CloudSyncEngine.ts @@ -61,7 +61,6 @@ import { chatPanelSelectedCloudOrgAtom } from "@src/store/ui/chatPanelAtom"; import type { ProjectSyncBridge } from "../TeamCollaboration/engine/projectSyncBridge"; import { tauriProjectSyncBridge } from "../TeamCollaboration/engine/projectSyncBridge"; import { getSessionForkedFrom } from "../TeamCollaboration/forkSession"; -import { isScopeMatchableImportedSession } from "../TeamCollaboration/importedSessionScopeMatch"; import { resolveMatchingOrgRepoScope } from "../TeamCollaboration/repoScopeResolver"; import { isSessionTaggedToCloudOrg, @@ -82,6 +81,8 @@ import { org2CloudAuthAtom, } from "./org2CloudAuthAtom"; import { ensureFreshSession, schemaVersion } from "./org2CloudClient"; +import { resolveOrgEndpoint } from "./org2CloudEndpointDirectory"; +import { setOrgEndpointDirectory } from "./org2CloudOrgEndpointRouter"; import { buildCloudOrgSelectorValue, org2CloudOrgsAtom, @@ -89,12 +90,18 @@ import { } from "./org2CloudOrgsAtom"; import type { Org2CloudOrg } from "./org2CloudOrgsAtom"; import * as org2CloudProjectsClient from "./org2CloudProjectsClient"; +import { + PUSH_ADMISSION_DENIAL, + decidePushAdmission, +} from "./org2CloudPushAdmission"; import { Org2CloudSessionSync, type Org2CloudSyncClientDeps, } from "./org2CloudSessionSync"; import { isCloudPushCandidate } from "./org2CloudSessionSync"; import { + org2CloudPushCursorsAtom, + org2CloudPushedMetadataAtom, org2CloudRepoScopesAtom, org2CloudSyncEnabledAtom, } from "./org2CloudSyncAtoms"; @@ -115,6 +122,10 @@ import { Org2CloudRepoScopeSync, getSessionScopeKeys, } from "./org2CloudSyncEngine.repoScopeSync"; +import { + orgsWithLocalPushMarkers, + reconcileOrgRetracts, +} from "./org2CloudSyncEngine.retractReconcile"; import { Org2CloudSchemaGate, type Org2CloudSchemaVersionProbe, @@ -153,6 +164,8 @@ export class Org2CloudSyncEngine extends Org2CloudSyncLifecycle { * rationale (kept together there since a policy signal touches more than * one at once). */ private readonly orgBackoff: Org2CloudOrgBackoffTracker; + /** Generation whose background-org retract reconcile already ran (P2). */ + private reconciledGeneration = -1; /** TTL-gated `org2CloudRepoScopesAtom` mirror hydration, split out to * `Org2CloudRepoScopeSync`. */ private readonly repoScopeSync: Org2CloudRepoScopeSync; @@ -305,6 +318,17 @@ export class Org2CloudSyncEngine extends Org2CloudSyncLifecycle { // made one open workspace scan/replay sessions across every matching team // and kept inactive-org scope RPCs alive. Switching/opening an org causes // its Realtime subscription to request an immediate full session pass. + // Sharding Phase A: publish the roster's resolved home endpoints so + // org-scoped data-plane calls route to each org's home project. Rebuilt + // every pass — a directory cutover (or rollback) takes effect on the + // next pass without restart. Empty/absent homeEndpoint keeps the + // official endpoint, so pre-0007 backends behave exactly as before. + setOrgEndpointDirectory( + orgs.map((org) => [ + org.orgId, + resolveOrgEndpoint(org, getCloudEndpoint()), + ]) + ); const activeSessionOrgs = orgs.filter((org) => this.isActiveOrg(org.orgId)); await this.repoScopeSync.hydrateRepoScopes( fresh, @@ -359,11 +383,25 @@ export class Org2CloudSyncEngine extends Org2CloudSyncLifecycle { session.session_id, org.orgId ); - if (forkedFrom && forkedFrom.orgId !== org.orgId && !tagged) { + const admission = decidePushAdmission({ + orgId: org.orgId, + session, + forkedFrom, + tagged, + ownedByOrg: session.orgId === buildCloudOrgSelectorValue(org.orgId), + shareIntent: hasExplicitCloudShareIntent( + accessByOrg[org.orgId], + session.session_id + ), + }); + if ( + !admission.admitted && + admission.denial === PUSH_ADMISSION_DENIAL.FORK_OUTSIDE_SOURCE_ORG + ) { if (this.sessionSync.wasCloudPushed(org.orgId, session.session_id)) { try { log.info( - `cloud retract [fork outside source org]: session ${session.session_id} org ${org.orgId}` + `cloud retract [${admission.denial}]: session ${session.session_id} org ${org.orgId}` ); await this.sessionSync.retractSession( fresh, @@ -398,24 +436,11 @@ export class Org2CloudSyncEngine extends Org2CloudSyncLifecycle { // This prevents a Personal session from leaking into every team org // that happens to configure the same Git remote. Fork provenance has // already constrained untagged forks to their source org above. - const ownedByOrg = - session.orgId === buildCloudOrgSelectorValue(org.orgId); - const shareIntent = hasExplicitCloudShareIntent( - accessByOrg[org.orgId], - session.session_id - ); - const scopeAutoMatched = isScopeMatchableImportedSession(session); - if ( - !forkedFrom && - !tagged && - !ownedByOrg && - !shareIntent && - !scopeAutoMatched - ) { + if (!admission.admitted) { if (this.sessionSync.wasCloudPushed(org.orgId, session.session_id)) { try { log.info( - `cloud retract [ownership-gate (untagged/unowned/no-intent)]: session ${session.session_id} org ${org.orgId}` + `cloud retract [${admission.denial}]: session ${session.session_id} org ${org.orgId}` ); await this.sessionSync.retractSession( fresh, @@ -518,17 +543,14 @@ export class Org2CloudSyncEngine extends Org2CloudSyncLifecycle { // Settings while withholding the matching cloud push would make the // rendered policy lie. Ordinary Personal sessions still require org // ownership, a tag, fork provenance, or explicit share intent. - const floorEligible = - Boolean(forkedFrom) || - tagged || - ownedByOrg || - shareIntent || - scopeAutoMatched; + // Reaching here means admission passed, and every admission route + // (provenance, tag, ownership, intent, scope match) is floor-eligible + // — so the floor always applies at this point. const access = resolveCloudPushAccess( accessByOrg[org.orgId], session.session_id, tagged, - floorEligible ? floorByOrg[org.orgId] : undefined + floorByOrg[org.orgId] ); if (!access) { // Effective-off and NOT tagged: the ladder grants nothing this @@ -687,6 +709,56 @@ export class Org2CloudSyncEngine extends Org2CloudSyncLifecycle { } } } + + // P2: retract-only reconcile for orgs the user is NOT looking at. The + // session plane above follows visible-org demand, so a session that + // lost admission in a background org would otherwise stay published + // until that org is reopened — possibly never. Once per engine run: + // same admission decision, same server-confirmed scope boundary, only + // rows THIS client push-marked. See the module header for the rails. + if (this.reconciledGeneration !== generation) { + this.reconciledGeneration = generation; + const cursors = store.get(org2CloudPushCursorsAtom); + const pushedMetadata = store.get(org2CloudPushedMetadataAtom); + const markedOrgIds = orgsWithLocalPushMarkers(cursors, pushedMetadata); + const backgroundOrgs = orgs.filter( + (org) => + markedOrgIds.has(org.orgId) && + !this.isActiveOrg(org.orgId) && + enabledByOrg[org.orgId] !== false && + !this.orgBackoff.isOrgBackedOff(org.orgId) + ); + if (backgroundOrgs.length > 0) { + await this.repoScopeSync.hydrateRepoScopes( + fresh, + backgroundOrgs, + generation, + (gen) => this.generation === gen + ); + if (this.generation !== generation) return; + log.info( + `retract reconcile: covering ${backgroundOrgs.length} background org(s) with local push markers` + ); + for (const org of backgroundOrgs) { + if (this.generation !== generation) return; + if (getCloudEndpoint().supabaseUrl !== passSupabaseUrl) return; + await reconcileOrgRetracts( + { + store, + accessByOrg, + wasCloudPushed: (orgId, sessionId) => + this.sessionSync.wasCloudPushed(orgId, sessionId), + retractSession: (orgId, sessionId) => + this.sessionSync.retractSession(fresh, orgId, sessionId), + hasServerConfirmedScopes: (orgId) => + this.repoScopeSync.hasServerConfirmedScopes(orgId), + isCurrentGeneration: () => this.generation === generation, + }, + org.orgId + ); + } + } + } } protected override clearOrgBackoff(orgId: string): void { diff --git a/src/features/TeamCollaboration/engine/collabSegmentPlanning.ts b/src/features/TeamCollaboration/engine/collabSegmentPlanning.ts index 724a2b08d..f15ae6d09 100644 --- a/src/features/TeamCollaboration/engine/collabSegmentPlanning.ts +++ b/src/features/TeamCollaboration/engine/collabSegmentPlanning.ts @@ -157,17 +157,46 @@ function isProvablyStuck( * the skip-over: if a "provably" stuck event does mutate after all, the push * detects the chain mismatch and re-anchors with one epoch rewrite. */ -export function computeFrozenEventCount(events: SessionEvent[]): number { +/** Recently-terminal events are still amendable by the ingest (tool-result + * backfill, synthetic-input cleanup): terminal ≠ immutable. Freezing them + * turns every amendment into a full epoch rewrite of the whole history, so + * the freeze line holds back events younger than this horizon; amendments + * then land in the mutable tail (one small segment re-upload). A quiescent + * session has nothing inside the horizon and freezes to the end. */ +const FREEZE_MUTATION_HORIZON_MS = 10 * 60_000; +/** Bound on horizon holdback — the tail ships as ONE segment, so a busy + * span must not grow it without limit. Events older than the horizon or + * beyond this cap freeze even while the session is live. */ +const FREEZE_HORIZON_MAX_EVENTS = 40; + +export function computeFrozenEventCount( + events: SessionEvent[], + nowMs: number = Date.now() +): number { let proof: StuckSentinelProof | null = null; + let line = events.length; for (let index = 0; index < events.length; index += 1) { const event = events[index]; const status = event?.displayStatus; if (typeof status === "string" && !TERMINAL_EVENT_STATUSES.has(status)) { proof ??= buildStuckSentinelProof(events); - if (!isProvablyStuck(event, index, proof)) return index; + if (!isProvablyStuck(event, index, proof)) { + line = index; + break; + } } } - return events.length; + const horizonFloor = nowMs - FREEZE_MUTATION_HORIZON_MS; + let held = 0; + while (line > 0 && held < FREEZE_HORIZON_MAX_EVENTS) { + // Only a PROVABLY recent event is held back; a missing/invalid + // timestamp freezes as before (the hash chain still catches mutation). + const createdAt = Date.parse(events[line - 1]?.createdAt ?? ""); + if (!Number.isFinite(createdAt) || createdAt < horizonFloor) break; + line -= 1; + held += 1; + } + return line; } /** Per-segment size budget (design §7.3 step 3a), measured pre-gzip. */ diff --git a/src/features/TeamCollaboration/engine/collabSyncEngineHelpers.test.ts b/src/features/TeamCollaboration/engine/collabSyncEngineHelpers.test.ts index f35436f38..34520a6d3 100644 --- a/src/features/TeamCollaboration/engine/collabSyncEngineHelpers.test.ts +++ b/src/features/TeamCollaboration/engine/collabSyncEngineHelpers.test.ts @@ -144,6 +144,30 @@ describe("computeFrozenEventCount frozen line + stuck-sentinel skip-over", () => expect(computeFrozenEventCount(events)).toBe(1); }); + it("holds recently-terminal events inside the mutation horizon in the tail", () => { + // Terminal ≠ immutable while the ingest can still amend (tool-result + // backfill): freezing them made every amendment a full epoch rewrite. + const now = Date.parse("2026-07-25T12:00:00Z"); + const old = { createdAt: "2026-07-25T11:00:00Z" }; + const recent = { createdAt: "2026-07-25T11:55:00Z" }; + const events = [event(old), event(old), event(recent), event(recent)]; + expect(computeFrozenEventCount(events, now)).toBe(2); + }); + + it("freezes everything once the session is quiescent past the horizon", () => { + const now = Date.parse("2026-07-25T12:00:00Z"); + const old = { createdAt: "2026-07-25T11:00:00Z" }; + const events = [event(old), event(old), event(old)]; + expect(computeFrozenEventCount(events, now)).toBe(3); + }); + + it("caps horizon holdback so a busy span cannot grow the tail unbounded", () => { + const now = Date.parse("2026-07-25T12:00:00Z"); + const recent = { createdAt: "2026-07-25T11:59:00Z" }; + const events = Array.from({ length: 60 }, () => event(recent)); + expect(computeFrozenEventCount(events, now)).toBe(20); + }); + it("counts a missing displayStatus as terminal (hash chain catches mutation)", () => { const events = [event({ displayStatus: undefined as never }), event({})]; expect(computeFrozenEventCount(events)).toBe(2); diff --git a/tests/stress/orgFanout.mjs b/tests/stress/orgFanout.mjs new file mode 100644 index 000000000..67e4aa7f6 --- /dev/null +++ b/tests/stress/orgFanout.mjs @@ -0,0 +1,421 @@ +#!/usr/bin/env node +/** + * 50-member org stress harness (scalability campaign, workstream #1). + * + * Measures the three member-scale unknowns against the LIVE backend: + * 1. realtime fan-out — N clients subscribe postgres_changes on + * cloud_session_comments for one org; one comment write; measure + * per-client delivery latency and loss; + * 2. concurrent comments — all N users call cloud_add_session_comment + * simultaneously; measure latency distribution and error rate; + * 3. concurrent owner pushes — M users upsert their own metadata rows + * simultaneously (the multi-owner push front door). + * + * Test identities live under the @org2-stress.invalid domain and are + * created/removed via the GoTrue admin API. Everything this script makes + * is torn down by `cleanup` (org delete cascades sessions via 0009). + * + * node tests/stress/orgFanout.mjs setup [N=50] + * node tests/stress/orgFanout.mjs run + * node tests/stress/orgFanout.mjs cleanup + */ +import { createClient } from "@supabase/supabase-js"; +import { createHash } from "node:crypto"; +import { readFileSync, writeFileSync, existsSync } from "node:fs"; +import { resolve, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const STATE_FILE = resolve(here, ".stress-state.json"); + +function loadEnv() { + const envPath = resolve(here, "../e2e/.env"); + const out = {}; + for (const line of readFileSync(envPath, "utf8").split("\n")) { + const m = line.match(/^([A-Z0-9_]+)=(.*)$/); + if (m) out[m[1]] = m[2].replace(/^"|"$/g, ""); + } + return out; +} + +const env = loadEnv(); +const URL_ = env.E2E_CLOUD_SUPABASE_URL; +const SERVICE = env.E2E_CLOUD_SERVICE_KEY; +const ANON = env.E2E_CLOUD_ANON_KEY; +const PASSWORD = "Stress-0725!pass"; +const EMAIL = (i) => `stress-${String(i).padStart(3, "0")}@org2-stress.invalid`; + +const admin = createClient(URL_, SERVICE, { + auth: { autoRefreshToken: false, persistSession: false }, +}); + +async function rpc(token, fn, args) { + const res = await fetch(`${URL_}/rest/v1/rpc/${fn}`, { + method: "POST", + headers: { + apikey: ANON, + authorization: `Bearer ${token}`, + "content-type": "application/json", + "content-profile": "org2_cloud", + }, + body: JSON.stringify(args), + }); + const text = await res.text(); + if (!res.ok) throw new Error(`${fn} ${res.status}: ${text.slice(0, 200)}`); + return text ? JSON.parse(text) : null; +} + +let rateLimited = 0; +async function signIn(i, attempt = 0) { + const res = await fetch(`${URL_}/auth/v1/token?grant_type=password`, { + method: "POST", + headers: { apikey: ANON, "content-type": "application/json" }, + body: JSON.stringify({ email: EMAIL(i), password: PASSWORD }), + }); + if (res.status === 429 && attempt < 8) { + rateLimited += 1; + await new Promise((r) => setTimeout(r, 45000)); + return signIn(i, attempt + 1); + } + if (!res.ok) throw new Error(`signIn ${i}: ${res.status}`); + return res.json(); +} + +async function mapLimit(items, limit, fn) { + const out = new Array(items.length); + let next = 0; + await Promise.all( + Array.from({ length: Math.min(limit, items.length) }, async () => { + while (next < items.length) { + const idx = next++; + out[idx] = await fn(items[idx], idx); + } + }) + ); + return out; +} + +function quantiles(values) { + const s = [...values].sort((a, b) => a - b); + const q = (p) => s[Math.min(s.length - 1, Math.floor(p * s.length))]; + return { p50: q(0.5), p95: q(0.95), max: s[s.length - 1], n: s.length }; +} + +async function setup(n) { + const users = []; + for (let i = 0; i < n; i += 1) { + const { data, error } = await admin.auth.admin.createUser({ + email: EMAIL(i), + password: PASSWORD, + email_confirm: true, + user_metadata: { display_name: `Stress ${i}` }, + }); + if (error && !`${error.message}`.includes("already")) throw error; + if (data?.user) users.push(data.user.id); + else { + const list = await admin.auth.admin.listUsers({ perPage: 1000 }); + const hit = list.data.users.find((u) => u.email === EMAIL(i)); + users.push(hit.id); + } + } + const prior = existsSync(STATE_FILE) + ? JSON.parse(readFileSync(STATE_FILE, "utf8")) + : null; + const tokens = prior?.tokens ?? {}; + const owner = await signIn(0); + tokens[0] = owner.refresh_token; + let orgId = prior?.orgId; + if (!orgId) { + const org = await rpc(owner.access_token, "create_org", { + org_name: "Stress Fanout 0725", + }); + orgId = org.orgId ?? org.id; + } + // Membership goes through the REAL invite flow — org_memberships is + // write-hardened even for service_role, and concurrent accepts on one + // invite are themselves a contention scenario worth measuring. + const code = `stress-code-${Date.now()}`; + const hash = createHash("sha256").update(code).digest("hex"); + await rpc(owner.access_token, "create_invite", { + p_org_id: orgId, + invite_code_hash: hash, + invite_role: "member", + max_uses: n + 10, + expires_at: null, + }); + const acceptStart = Date.now(); + const accepts = await mapLimit( + Array.from({ length: n - 1 }, (_, idx) => idx), + 8, + async (idx) => { + const member = await signIn(idx + 1); + tokens[idx + 1] = member.refresh_token; + try { + await rpc(member.access_token, "accept_invite", { + invite_code_hash: hash, + }); + return true; + } catch (error) { + console.log("accept failed:", String(error).slice(0, 120)); + return false; + } + } + ); + console.log( + `invite accepts: ${accepts.filter(Boolean).length}/${n - 1} in ${ + Date.now() - acceptStart + }ms` + ); + const sessionId = prior?.sessionId ?? `stress-fanout-target-${Date.now()}`; + const sess = await fetch(`${URL_}/rest/v1/cloud_sessions`, { + method: "POST", + headers: { + apikey: SERVICE, + authorization: `Bearer ${SERVICE}`, + "content-type": "application/json", + "content-profile": "org2_cloud", + prefer: "return=minimal,resolution=merge-duplicates", + }, + body: JSON.stringify({ + org_id: orgId, + owner_user_id: users[0], + session_id: sessionId, + access_mode: "metadata_only", + visibility: "org", + last_activity_at: new Date().toISOString(), + metadata: { title: "fanout target" }, + }), + }); + if (!sess.ok) throw new Error(`session: ${sess.status} ${await sess.text()}`); + writeFileSync(STATE_FILE, JSON.stringify({ users, orgId, sessionId, n, tokens })); + console.log(`setup done: org ${orgId}, ${users.length} users`); +} + +async function run() { + const state = JSON.parse(readFileSync(STATE_FILE, "utf8")); + const { orgId, sessionId, n } = state; + console.log(`signing in ${n} users…`); + const t0 = Date.now(); + const refresh = async (i) => { + const saved = state.tokens?.[i]; + if (saved) { + const res = await fetch(`${URL_}/auth/v1/token?grant_type=refresh_token`, { + method: "POST", + headers: { apikey: ANON, "content-type": "application/json" }, + body: JSON.stringify({ refresh_token: saved }), + }); + if (res.ok) { + const fresh = await res.json(); + state.tokens[i] = fresh.refresh_token; + return fresh; + } + } + return signIn(i); + }; + const sessions = await mapLimit( + Array.from({ length: n }, (_, i) => i), + 8, + (i) => refresh(i) + ); + writeFileSync(STATE_FILE, JSON.stringify(state)); + console.log( + `sign-in storm: ${n} tokens in ${Date.now() - t0}ms (429 retries: ${rateLimited})` + ); + + console.log("subscribing realtime clients…"); + const deliveries = new Map(); + const clients = sessions.map((s, i) => { + const c = createClient(URL_, ANON, { + auth: { autoRefreshToken: false, persistSession: false }, + realtime: { params: { eventsPerSecond: 10 } }, + }); + c.realtime.setAuth(s.access_token); + return c; + }); + let subscribed = 0; + await Promise.all( + clients.map( + (c, i) => + new Promise((done) => { + // Mirror the app: comments ride the org presence channel as + // client-sent broadcast nudges (org2CloudCommentsBus), not + // postgres_changes on the comments table. + const ch = c + .channel(`presence:org:${orgId}`, { + config: { broadcast: { self: false } }, + }) + .on("broadcast", { event: "*" }, () => + deliveries.set(i, Date.now()) + ) + .subscribe((status) => { + if (status === "SUBSCRIBED") { + subscribed += 1; + done(); + } + if (status === "CHANNEL_ERROR" || status === "TIMED_OUT") done(); + }); + setTimeout(done, 15000); + }) + ) + ); + console.log(`subscribed: ${subscribed}/${n}`); + + const writeAt = Date.now(); + await rpc(sessions[0].access_token, "cloud_add_session_comment", { + p_org_id: orgId, + p_session_id: sessionId, + p_body: `fanout probe ${writeAt}`, + }); + const writerChannel = clients[0] + .channel(`presence:org:${orgId}:writer`, {}) + .subscribe(); + await new Promise((r) => setTimeout(r, 1500)); + await clients[0] + .channel(`presence:org:${orgId}`, { config: { broadcast: { self: false } } }) + .send({ + type: "broadcast", + event: "comments_changed", + payload: { sessionId, at: writeAt }, + }); + await new Promise((r) => setTimeout(r, 8000)); + const missing = Array.from({ length: n }, (_, i) => i).filter( + (i) => !deliveries.has(i) + ); + console.log(`missing indices: ${JSON.stringify(missing)}`); + const latencies = [...deliveries.values()].map((t) => t - writeAt); + // The writer's own client is excluded by broadcast self:false — count + // RECEIVERS, not subscribers. Four consecutive runs each missed exactly + // index 0 (the writer): delivery to actual receivers is 49/49. + console.log( + `fanout: delivered ${deliveries.size}/${subscribed - 1} receivers — ${JSON.stringify( + latencies.length ? quantiles(latencies) : {} + )}` + ); + for (const c of clients) await c.removeAllChannels(); + + console.log(`concurrent comments: ${n} simultaneous writers…`); + const results = await Promise.all( + sessions.map(async (s, i) => { + const start = Date.now(); + try { + await rpc(s.access_token, "cloud_add_session_comment", { + p_org_id: orgId, + p_session_id: sessionId, + p_body: `concurrent ${i}`, + }); + return { ms: Date.now() - start }; + } catch (error) { + return { err: String(error).slice(0, 120) }; + } + }) + ); + const ok = results.filter((r) => r.ms !== undefined); + const errs = results.filter((r) => r.err); + console.log( + `comments: ${ok.length} ok, ${errs.length} errors — ${JSON.stringify( + quantiles(ok.map((r) => r.ms)) + )}` + ); + if (errs.length) console.log("sample error:", errs[0].err); + + console.log("concurrent owner metadata pushes (service-role upserts)…"); + const pushStart = Date.now(); + const pushes = await Promise.all( + state.users.slice(0, 20).map(async (userId, i) => { + const start = Date.now(); + const res = await fetch(`${URL_}/rest/v1/cloud_sessions`, { + method: "POST", + headers: { + apikey: SERVICE, + authorization: `Bearer ${SERVICE}`, + "content-type": "application/json", + "content-profile": "org2_cloud", + prefer: "resolution=merge-duplicates,return=minimal", + }, + body: JSON.stringify({ + org_id: orgId, + owner_user_id: userId, + session_id: `stress-owner-${i}`, + access_mode: "metadata_only", + visibility: "org", + last_activity_at: new Date().toISOString(), + metadata: { title: `owner push ${i}` }, + }), + }); + return { ok: res.ok, ms: Date.now() - start }; + }) + ); + console.log( + `owner pushes: ${pushes.filter((p) => p.ok).length}/20 ok in ${ + Date.now() - pushStart + }ms — ${JSON.stringify(quantiles(pushes.map((p) => p.ms)))}` + ); +} + +async function sustain(minutes) { + const state = JSON.parse(readFileSync(STATE_FILE, "utf8")); + const { orgId, sessionId } = state; + const writers = 10; + const sessions = await mapLimit( + Array.from({ length: writers }, (_, i) => i), + 4, + (i) => signIn(i) + ); + console.log(`sustain: ${writers} writers × ${minutes}min, 2s cadence`); + const perMinute = new Map(); + const endAt = Date.now() + minutes * 60_000; + let errors = 0; + await Promise.all( + sessions.map(async (sess, w) => { + while (Date.now() < endAt) { + const start = Date.now(); + try { + // Per-writer target sessions dodge the 500-comment/session cap + // (an 0001 abuse guard counting live AND tombstoned rows) so this + // measures infra drift, not the cap. + await rpc(sess.access_token, "cloud_add_session_comment", { + p_org_id: orgId, + p_session_id: `stress-owner-${w}`, + p_body: `sustain w${w} ${start}`, + }); + const minute = Math.floor((start - (endAt - minutes * 60_000)) / 60_000); + const bucket = perMinute.get(minute) ?? []; + bucket.push(Date.now() - start); + perMinute.set(minute, bucket); + } catch (error) { + errors += 1; + if (errors <= 3) console.log("sustain error:", String(error).slice(0, 160)); + } + await new Promise((r) => setTimeout(r, 2000)); + } + }) + ); + for (const [minute, values] of [...perMinute.entries()].sort((a, b) => a[0] - b[0])) { + console.log(`minute ${minute}: ${JSON.stringify(quantiles(values))}`); + } + console.log(`sustain done: errors=${errors}`); +} + +async function cleanup() { + if (!existsSync(STATE_FILE)) return console.log("no state"); + const { users, orgId } = JSON.parse(readFileSync(STATE_FILE, "utf8")); + const owner = await signIn(0); + try { + await rpc(owner.access_token, "cloud_delete_org", { p_org_id: orgId }); + console.log("org deleted (0009 cascades sessions)"); + } catch (error) { + console.log("org delete:", String(error).slice(0, 160)); + } + for (const id of users) { + await admin.auth.admin.deleteUser(id).catch(() => undefined); + } + console.log(`deleted ${users.length} stress users`); +} + +const mode = process.argv[2] ?? "run"; +const n = Number(process.argv[3] ?? 50); +if (mode === "setup") await setup(n); +else if (mode === "run") await run(); +else if (mode === "sustain") await sustain(Number(process.argv[3] ?? 5)); +else if (mode === "cleanup") await cleanup(); +else console.log("modes: setup [n] | run | sustain [min] | cleanup");