diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 9c3e132..a3eff28 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -6,8 +6,8 @@ file at the **end of every step** so we never jump ahead on a deferred item or skip a main phase. - **Last updated:** 2026-07-23 - see [PROGRESS Β§7 Changelog](#7-changelog). This field carries the date only; the narrative lives in Β§7. -- **Active branch:** `docs/close-45-and-log-design-items` - this docs-only PR CLOSES register **#45** (the live duplicate Vercel ingestion service is gone and `INGESTION_TOKEN` was rotated) by converting its row in place (πŸ”΄β†’βœ…, original lead preserved after "Was:"), and logs four new product-design items as registers **#46-#49** (settings-page premium polish, per-material actions, landing-page nav menu, animated terminal demo). Touches **only** this file (the #45 row, four new register rows, one new Β§7 block, and these three header fields). **No code, no other register row's substance, no Β§1-Β§6 phase content, no B5b row, `PIVOT_PLAN.md` untouched.** Register **#17** stays OPEN. -- **Main tip:** `ecb1ed5` (PR **#67** merge - the duplicate-ingestion-service removal: the `experimentalServices.ingestion` sub-key dropped from `vercel.json`, keeping `frontend`), on top of `2d0856d` (PR **#66** - the Β§7 changelog migration off the line-8 header field, all 17 entry bodies byte-identical) and `c119f6b` (PR **#65** - register #17 WIP preservation snapshot). Both required checks green on `ecb1ed5` (`tsc`, `db-types`). +- **Active branch:** `docs/b5b-scoping-record` - this docs-only PR RECORDS the **B5b upload-content-hardening scoping** as register **#50** and creates the decision record [`b5b-scoping.md`](./b5b-scoping.md): the established+verified constraints (the real ceiling is Vercel's hard ~4.5 MB edge cap, not the app's broken 50 MB promise; duration is the unmeasured true limit), the corrections (the cap does not stop ZIP bombs; `PIVOT_PLAN.md:269`'s 50 MB claim is false), the B5b-1/B5b-2 split plus enablers (b1)/(b2), and **seven OPEN founder decisions** with an OUTSTANDING duration measurement. Touches **only** two files: `docs/b5b-scoping.md` (new) and this file (one new register row #50, one new Β§7 block, and these three header fields). **No code, no route/validation/size-constant change, no other register row's substance, no Β§1-Β§6 phase content; B5b Β§1 row stays ⬜ (scoping is not progress), `PIVOT_PLAN.md` untouched.** Register **#17** stays OPEN. +- **Main tip:** `1e37c03` (PR **#68** merge - the docs-only close of register #45 and logging of design items #46-#49), on top of `ecb1ed5` (PR **#67** - the duplicate Vercel ingestion service removed from `vercel.json` + `INGESTION_TOKEN` rotated) and `2d0856d` (PR **#66** - the Β§7 changelog migration off the line-8 header field, all 17 entry bodies byte-identical). Both required checks green on `1e37c03` (`tsc`, `db-types`). > βœ… **RESOLVED 2026-07-04 β€” Supabase project back online (register #21).** The > project **auto-paused** (free-tier inactivity) and then hit Supabase's free @@ -146,6 +146,7 @@ project memory, and per-step review debates. "Where" is the target phase/trigger | 47 πŸ”Ž **OPEN β€” product-design item, raised 2026-07-23 (Abo Jad design review). DESIGN-QUALITY, NOT A BUG.** | **Materials page has no per-item management** β€” a material, once listed, cannot be acted on. | Not a defect β€” a missing-affordance/UX item; no phase assigned. | Wanted: **per-material actions** β€” **rename**, **delete**, and similar. | | 48 πŸ”Ž **OPEN β€” product-design item, raised 2026-07-23 (Abo Jad design review). DESIGN-QUALITY, NOT A BUG.** | **Pre-login landing page has no navigation menu.** | Not a defect β€” a missing-nav/UX item for the mobile app; no phase assigned. | Wanted: a **hamburger (three-line) icon** opposite the KnowFlow wordmark in the header, opening a **main dropdown menu** for the mobile app. | | 49 πŸ”Ž **OPEN β€” product-design item, raised 2026-07-23 (Abo Jad design review). Carries a correctness rider, not pure polish.** | **The landing page's terminal-style demo window shows static text.** **Cross-ref #20:** that window currently displays **"Ready in 0.4s"**, a timing claim **#20 already flags**, and this session proved **ingestion is synchronous and takes real seconds** β€” so the number MUST be corrected or removed as part of this work rather than re-shipping a false claim. | Not pure polish β€” the embedded "0.4s" IS a false claim (see **#20**), so this row carries a correctness rider; no phase assigned. | Wanted: **animate the terminal** so the text **types out live from start to finish**; as part of the same work, **correct or remove "Ready in 0.4s"** (per **#20**) rather than re-ship it. | +| 50 πŸ”Ž **SCOPING RECORDED 2026-07-23 β€” B5b upload content hardening. Decisions 1-7 OPEN pending founder rulings; the duration measurement OUTSTANDING. Scoping is NOT progress β€” B5b (Β§1) stays ⬜.** | **B5b (deep upload content hardening: magic-byte, decompression-bomb / nested-archive limits, content scanning) is now scoped in [`b5b-scoping.md`](./b5b-scoping.md).** Established + verified this session: the real upload ceiling is **Vercel's hard ~4.5 MB edge cap** (platform constant, unraisable), **not** the 50 MB the app promises in **7 places** (`route.ts:51/:52`, `DropZone.tsx:26`, four i18n strings β€” the `:52` message is hardcoded English, register-#17 class); ingestion runs on **Railway** (`INGESTION_SERVICE_URL`), so the `route.ts`β†’`/convert` hop faces no second Vercel cap but a **9-13Γ— return-path blob** (each chunk's 1024-float embedding inline + full markdown, materialized by `await pyResponse.json()`); and **no `maxDuration` is set anywhere** (grep-empty), so the true user-facing ceiling is an **UNMEASURED duration limit**, not size. Corrections recorded so they are not re-inherited: the 4.5 MB cap does **NOT** mitigate decompression bombs (compressed bytes are bounded, expansion is not); the pre-auth body-buffering concern is **bounded** by the edge cap, so the `getUser`-above-`formData` reorder is a low-priority ride-along (`checkDocumentLimit` **cannot** move β€” needs `kbId` from the body; `enforceLimit` **should not** β€” it would burn an upload credit on a rejected request); and **`PIVOT_PLAN.md:269`'s "Size cap (50 MB) already exists" is FALSE** (the effective cap is the platform's ~4.5 MB). | Not a bug and not deferred work β€” a **decision record** written before implementation so the founder's rulings are captured, not inferred. The split is recorded (B5b-1 request-path magic-byte + declared-size ZIP-ratio pre-check + the auth reorder, doable now; B5b-2 enforced expansion bounds + nested depth + AV/scanning, genuinely gated on **B6**; plus enabler moves (b1) persist chunks from Railway and (b2) signed-URL direct-to-storage). **No code, no route/validation/size-constant change; `PIVOT_PLAN.md` untouched.** | **Seven OPEN founder decisions** (each stated with consequences in the doc, none picked here): (1) max file size β€” accept ~4.5 MB vs. deliver real 50 MB via (b2); (2) format list β€” keep all six vs. drop the docx/pptx/xlsx ZIP trio that carries all bomb risk; (3) scanning scope β€” AV (nothing in the image today), PII, and **prompt-injection in extracted text** (flows unfiltered into Haiku via `` interpolation β€” currently unmitigated, needs at minimum a documented accepted-risk decision); (4) failure posture β€” reject-at-upload (415) vs. accept-then-quarantine; (5) enforcement point β€” Next, Railway, or both (Railway trusts its caller on a single bearer token, so Next-only is bypassed if `INGESTION_TOKEN` leaks); (6) the three ZIP numbers (expansion ratio, absolute expanded cap, nesting depth) β€” each measured from the spoofable declared central directory (cheap) or actual expansion (accurate, needs B6); (7) the signature table (`%PDF-`; `PK\x03\x04` + inner part-path for the OOXML trio; the txt/md non-signature UTF-8/NUL rule). **OUTSTANDING measurement:** at what file size does the upload time out β€” plain-`.txt`-on-Preview procedure + failure-signature table recorded in the doc Β§5 (each upload burns a rate-limit credit and the #22 free tier β€” throwaway subject, delete rows after). Related: **#45** (the duplicate ingestion service closed while scoping this), **#22** (bounds (b2) and the Β§5 test budget), **#24** (why `enforceLimit` cannot move), **#17** (the `:52` hardcoded-English class). | **Founder-owned open items (PIVOT_PLAN Β§10) β€” not engineering-blocked, tracked for launch:** Next.js hosting decision (Vercel Pro vs self-host, ~Phase 7/10) Β· **Supabase backend @@ -207,6 +208,10 @@ bodies total 37,095 bytes. **This retires the Option C frozen-tail invariant by existed only to police a boundary inside an unreviewable single line, and the append-only rule above supersedes it. No bespoke hash is needed for future updates: the diff is the proof. +### 2026-07-23 - B5b upload-hardening scoping recorded (register #50, docs/b5b-scoping.md) + +(**B5b (deep upload content hardening) is now SCOPED, not started β€” register #50 opened, `docs/b5b-scoping.md` created; Β§1 B5b stays ⬜ because scoping is not progress on the gate.** The doc records this session's established + verified facts, its corrections, the split, and seven OPEN founder decisions. **Established (each verified against the code this session):** the real upload ceiling is Vercel's **hard ~4.5 MB edge cap** (platform constant, unraisable β€” `413 FUNCTION_PAYLOAD_TOO_LARGE` before the handler), **not** the 50 MB the app promises in **7 places** (`route.ts:51` check / `:52` hardcoded-English 413 message β€” register-#17 class, `DropZone.tsx:26`, and four i18n strings in `en.ts`/`ar.ts`); ingestion runs on **Railway** (`INGESTION_SERVICE_URL = knowflow-production.up.railway.app`), so the `route.ts`β†’`/convert` hop faces no second Vercel cap but a **9-13Γ— return-path blob** β€” `/convert` returns each chunk's **1024-float embedding inline plus full markdown** (`main.py`: `EMBED_DIM=1024`, `CHUNK_TOKENS=512`, `CHUNK_OVERLAP=64`), and `await pyResponse.json()` materializes it in the Next function's memory; and **no `maxDuration` is configured anywhere** (grep-empty across `vercel.json`/`next.config.ts`/`package.json`/`src`), with Voyage embeds batched 128-at-60s and Supabase inserts batched 50, all sequential in one invocation β€” so the true user-facing ceiling is an **UNMEASURED duration limit**, not size. **Corrections recorded so they are not re-inherited:** the 4.5 MB cap does **NOT** mitigate decompression bombs (it bounds compressed bytes, never the expansion β€” 4.5 MB of crafted ZIP still exhausts the service); the pre-auth body-buffering concern is **bounded** by the edge cap, making the `getUser`-above-`formData` reorder a low-priority ride-along (`checkDocumentLimit` **cannot** move β€” it needs `kbId` from the body; `enforceLimit` **should not** β€” it would burn an upload credit on a rejected request, register #24); and **`PIVOT_PLAN.md:269`'s "Size cap (50 MB) already exists" is a documented FALSE claim** (the effective cap is the platform's ~4.5 MB), corrected here, `PIVOT_PLAN.md` deliberately untouched. **The split:** B5b-1 (request-path, doable now β€” magic-byte verification, a declared-size ZIP-ratio pre-check from the central directory, the auth reorder); B5b-2 (gated on **B6** β€” enforced expansion bounds, nested-archive depth, AV/content scanning, all needing actual expansion or a worker); plus enabler moves **(b1)** move chunk+embedding persistence into the Railway service (kills the 9-13Γ— blob, relocates slow work to the only component without a duration ceiling β€” the enabler of B6, not B6 itself) and **(b2)** signed-URL direct-to-Supabase-storage upload (the only way past the 4.5 MB inbound cap; gated on the file-size ruling; #22's 5 GB/mo egress limits its payoff to ~100 uploads/mo at 50 MB). **Seven OPEN decisions (stated with consequences, none picked): (1)** max file size β€” ~4.5 MB vs. real 50 MB via (b2), gating the enforcement point and sync/async posture; **(2)** format list β€” keep six vs. drop the docx/pptx/xlsx ZIP trio that carries all bomb risk; **(3)** scanning scope β€” AV (nothing in the image today), PII, and **prompt-injection in extracted text** (flows unfiltered into Haiku via `` interpolation across summarize/quiz/agent β€” unmitigated, needs at minimum a documented accepted-risk decision); **(4)** failure posture β€” reject-at-upload (415) vs. accept-then-quarantine; **(5)** enforcement point β€” Next, Railway, or both (Railway trusts its caller on a single bearer token, so Next-only is bypassed if `INGESTION_TOKEN` leaks); **(6)** the three ZIP numbers (expansion ratio, absolute expanded cap, nesting depth) β€” each measured from the spoofable declared central directory vs. actual expansion; **(7)** the signature table (`%PDF-`; `PK\x03\x04` + inner part-path for the OOXML trio; the txt/md non-signature UTF-8/NUL rule β€” txt/md have no magic bytes at all). **OUTSTANDING measurement:** the duration ceiling β€” plain-`.txt`-on-Preview procedure + a failure-signature table (`200`+chunk_count=pass; `413`+Vercel HTML=edge cap; `413`+app JSON=app check; `500`+"Ingestion failed"; `504`/none=duration), each upload burning a rate-limit credit and the #22 free tier. Related registers: **#45** (the duplicate ingestion service closed while scoping this), **#22**, **#24**, **#17**. **Docs-only: no code, no migration, no DB, no route/validation/size-constant change; only `docs/b5b-scoping.md` (new) and this file changed; `Phase 7` stays ⬜, Β§1 B5b row untouched, `PIVOT_PLAN.md` untouched, register #17 stays OPEN.**) + ### 2026-07-23 - closed register #45 and logged four product-design items (#46-#49) (**Register #45 CLOSED and four new design registers opened.** #45 β€” the live DUPLICATE Vercel ingestion service at `/_/ingestion` β€” is now closed: the `experimentalServices.ingestion` sub-key removal shipped in PR #67 (merge `ecb1ed5`), and the closure evidence was MEASURED, not asserted β€” before merge `curl https://tryknowflow.com/_/ingestion/health` returned **200 with no redirect**; after merge the same curl returned **307 β†’ `/ar/_/ingestion/health`** and, following it, **404** (Next's own page, not the service JSON), proving nothing intercepts the path at the edge any more and that **config removal alone sufficed** β€” no Vercel-dashboard service deletion was needed. `INGESTION_TOKEN` was then **rotated Railway-first then Vercel** and verified by a **real production upload reaching READY**. The #45 row was CONVERTED IN PLACE (πŸ”΄β†’βœ…, original lead preserved after "Was:"), per the #38/#40 keep-history convention. Separately, Abo Jad's design review raised **four product-design items**, logged as registers **#46-#49**, all πŸ”Ž OPEN β€” overarching framing: the app reads as visually **empty/cold** and does not match a **paid premium product**; these are DESIGN-QUALITY items, not bugs. **#46** the settings page needs premium polish (language-switcher buttons, a dark-mode toggle, further refinement); **#47** the materials page needs per-item actions (rename, delete, and similar); **#48** the pre-login landing page needs a hamburger nav menu opposite the wordmark for the mobile app; **#49** the landing page's terminal demo should type out live from start to finish, AND its **"Ready in 0.4s"** claim β€” already flagged by **#20** and disproven this session (ingestion is synchronous, takes real seconds) β€” must be corrected or removed rather than re-shipped. **Docs-only: no code, no migration, no DB; `Phase 7` stays ⬜, Β§1 B5b row untouched, `PIVOT_PLAN.md` untouched, register #17 stays OPEN.**) diff --git a/docs/b5b-scoping.md b/docs/b5b-scoping.md new file mode 100644 index 0000000..585ef9b --- /dev/null +++ b/docs/b5b-scoping.md @@ -0,0 +1,327 @@ +# B5b β€” Upload Content Hardening: Scoping & Decision Record + +**Status:** scoping only. **Nothing in this document changes code.** No route change, no +validation logic, no size-constant edit. It records the constraints this session +**established and verified**, the misdiagnoses it **corrected**, the **split** it proposes, +and the **decisions that remain the founder's to make**. B5b (PIVOT_PLAN Β§8) and Phase 7 both +stay ⬜ NOT STARTED β€” a scoping document is not progress on the gate. + +**Recorded:** 2026-07-23. **Author context:** written during the same session that closed +register #45 (the live duplicate Vercel ingestion service). This repo records **WHY, not just +what** β€” every value below is accompanied by the reasoning that makes it binding, and every +fact by how it was verified. + +**Scope boundary.** B5b is "deep upload content hardening: magic-byte verification, bomb / +nested-archive limits, content scanning" (PIVOT_PLAN Β§8, `B5b`). It is distinct from **B5a** +(the extension/MIME allowlist, already βœ… done, Phase 0) and from **B6** (synchronous ingestion +β†’ background job). B5b and B6 are **coupled but not identical**; Β§3 records exactly where the +seam falls. + +--- + +## Section 1 β€” Established constraints (facts, each with how it was verified) + +### 1.1 β€” Vercel enforces a hard 4.5 MB request-body cap at the edge + +Vercel Functions reject any request whose body exceeds **4.5 MB** with +`413 FUNCTION_PAYLOAD_TOO_LARGE`, **before the handler runs**. This is a documented platform +constant, **not configurable** β€” it cannot be raised from `vercel.json`, `next.config.ts`, or +route code. It therefore binds **every upload today**, and it binds *below* the 50 MB the app +currently promises (Β§1.6). + +**Why it matters:** the app's own 50 MB check (`route.ts:51`, `if (file.size > 52428800)`) +can *never fire for the reason a user would expect* β€” a 10 MB file is rejected by Vercel's edge +at 4.5 MB long before Next sees it. The app's check only ever fires for files between... nothing: +4.5 MB < 50 MB, so the app's 413 is **unreachable for oversize** on Vercel. It could only fire +if the edge cap were absent (e.g. the Railway service, or a self-hosted Next). + +**How verified:** documented Vercel platform limit; corroborated by register #45's measurement +work this session, where the Vercel-served ingestion duplicate's `/convert` was noted as +edge-capped at 4.5 MB inbound (unlike Railway). + +### 1.2 β€” Ingestion runs on Railway, not Vercel + +`INGESTION_SERVICE_URL = https://knowflow-production.up.railway.app` +(`.env.local:4`; consumed at `route.ts:143` and `src/lib/ingestion.ts:6`). The Python ingestion +service (MarkItDown β†’ chunk β†’ Voyage embed) runs on **Railway**, verified from the Vercel +dashboard during the register-#45 work. + +**Consequence:** the `route.ts` β†’ `/convert` hop is **Next (Vercel) β†’ Railway**. It faces **no +second Vercel inbound cap** (Railway is not behind Vercel's edge) and **no Vercel response cap**. +The 4.5 MB edge cap of Β§1.1 applies to the **browser β†’ Next** leg only. + +### 1.3 β€” Return-path amplification (memory/latency pressure, not a hard 413) + +`/convert` returns, in one JSON body, the **full markdown** *plus* **every chunk with its +1024-float embedding inline**. Verified in `services/ingestion/main.py:170-173` (returns +`{ markdown, chunks }`) and `:166-168` (each chunk gets `c["embedding"] = emb`), and consumed at +`route.ts:165` (`const result = await pyResponse.json()`). + +From `CHUNK_TOKENS = 512`, `CHUNK_OVERLAP = 64` (`main.py:27-28`) and `EMBED_DIM = 1024` +(`main.py:20`), the JSON response is roughly **9–13Γ— the size of the extracted text**: the +embeddings dominate (1024 floats serialized as ~11 bytes each β‰ˆ ~11 KB per chunk, versus ~2 KB of +chunk text), and the ~512/448 overlap re-emits text at ~1.14Γ—. So roughly **350–480 KB of +extracted text produces a ~4.5 MB response**. + +**Why it is NOT a hard 413:** `await pyResponse.json()` at `route.ts:165` **materializes the +entire response in the Next function's memory**, but because ingestion is on **Railway** (Β§1.2), +this is **memory + latency pressure inside the Next invocation**, not a Vercel response-size +rejection. There is no platform ceiling on this leg β€” it degrades, it does not cliff. + +### 1.4 β€” Duration likely binds before size, and it is UNMEASURED + +**No `maxDuration` is configured anywhere.** Verified grep-empty across `vercel.json`, +`next.config.ts`, `package.json`, **and** all of `src/`. The function runs under the platform +default. + +Inside a **single** `/api/ingest` invocation the work is **sequential**: +- Voyage embeds in batches of **128** inputs, each with a **60 s** client timeout + (`main.py:113-114`, `timeout=60.0`, `for i in range(0, len(texts), 128)`). +- Supabase chunk inserts run in batches of **50** (`route.ts:184`, `BATCH = 50`), looped. + +A 4.5 MB text file implies on the order of **~2,600 chunks** (512-token chunks, 64 overlap) and +therefore **~21 sequential embed batches**. The real user-facing ceiling is thus a **duration +ceiling**, and **it is UNMEASURED**. See Β§5 for the measurement procedure that must precede any +user-facing number. + +**Why this ordering matters:** debating a file-size number (Β§4 decision 1) while the true +binding limit is *time* would set a size cap that either never fires (duration times out first) +or is set to the wrong value. The measurement in Β§5 tells us which limit actually governs. + +### 1.5 β€” Accepted formats today + +`ALLOWED_FILE_TYPES = ['pdf', 'docx', 'pptx', 'xlsx', 'txt', 'md']` +(`src/types/index.ts:45`, single source of truth; B5a / register #42/#56 closed the drift). + +The gate: **extension is the primary check** (`route.ts:60-64`, 415 if the extension is not in +the allowlist). **MIME is only a secondary cross-check and is bypassable** β€” `route.ts:70-73` +tolerates an **empty** MIME *and* `application/octet-stream`, rejecting only a *specific* MIME +that contradicts the extension. A caller who sends no `Content-Type` (or `octet-stream`) passes +the MIME check with any allowed extension. + +**NOTE β€” txt and md have no magic bytes.** Four of the six formats have file signatures +(`%PDF-` for pdf; `PK\x03\x04` for the docx/pptx/xlsx OOXML trio). **`txt` and `md` have no +signature at all** β€” they are arbitrary text. A magic-byte rule (Β§3, B5b-1) **cannot** cover +them; they need a **different** rule: UTF-8 decodability / NUL-byte rejection, not a signature +match. Any signature table that forgets this would reject every legitimate `.txt`/`.md` upload. + +### 1.6 β€” The 50 MB promise is BROKEN in production, asserted in 7 places + +The app promises 50 MB in **seven** sites, but Β§1.1 makes the real ceiling ~4.5 MB, so the +promise is **false in production today**: + +1. `src/app/api/ingest/route.ts:51` β€” the check `if (file.size > 52428800)` (52428800 = 50 MiB) +2. `src/app/api/ingest/route.ts:52` β€” the 413 body `'File too large. Maximum size is 50MB.'`, + a **hardcoded English string, not routed through `t.*`** β€” the **same defect class as + register #17** (hardcoded English bypassing i18n) +3. `src/components/upload/DropZone.tsx:26` β€” the client check `if (file.size > 52428800)` +4. `src/lib/i18n/locales/en.ts:248` β€” `fileTooBig: "File too large. Maximum size is 50MB."` +5. `src/lib/i18n/locales/en.ts:251` β€” `supported: "… (Max: 50MB)"` +6. `src/lib/i18n/locales/ar.ts:250` β€” `fileTooBig: "… 50 Ω…ΩŠΨ¬Ψ§Ψ¨Ψ§ΩŠΨͺ."` +7. `src/lib/i18n/locales/ar.ts:253` β€” `supported: "… (Ψ§Ω„Ψ­Ψ― Ψ§Ω„Ψ£Ω‚Ψ΅Ω‰: 50 Ω…ΩŠΨ¬Ψ§Ψ¨Ψ§ΩŠΨͺ)"` + +**How verified:** direct grep of each file this session. Two of the seven (route.ts:52 and +the two `supported` strings) also carry the register-#17 i18n defect: correcting the number is +an opportunity to route route.ts:52 through `t.*` at the same time. + +--- + +## Section 2 β€” Corrections to record (so future readers do not inherit our misdiagnoses) + +These are places where an earlier, plausible reading was **wrong**. They are recorded so nobody +re-derives the mistake. + +### 2.1 β€” The 4.5 MB cap does NOT mitigate decompression bombs + +It is tempting to read Β§1.1's edge cap as a partial defense against ZIP bombs ("at least the +input is bounded to 4.5 MB"). **This is the single most dangerous inference available here, and +it is false.** ZIP expansion ratios are astronomical β€” a few dozen KB of crafted archive can +expand to **petabytes**. **4.5 MB of crafted ZIP still exhausts the ingestion service.** The +inbound cap bounds the *compressed* bytes; the bomb's damage is in the *expansion*, which the cap +never touches. Bomb mitigation must measure or bound the **expanded** size (Β§3, B5b-2), not the +uploaded size. + +### 2.2 β€” The pre-auth body-buffering concern is BOUNDED, not unbounded β€” a low-priority ride-along + +The request path parses the body **before** it authenticates: +`formData()` at `route.ts:43` β†’ `getUser()` at `route.ts:79` β†’ `enforceLimit` at `route.ts:102`. +It is tempting to call this an urgent "unauthenticated attacker streams unbounded bytes into our +function" hole. **It is bounded by the platform at 4.5 MB (Β§1.1), so it is NOT urgent.** + +- The `getUser()`-above-`formData()` **reorder is verified safe** but is now a **low-priority + ride-along, not an urgent fix**: `createClient` reads cookies only, so a cookieless caller + returns `user: null` with zero network calls β€” moving the auth check above the body parse + rejects an anonymous caller before buffering, at no correctness cost. +- **`checkDocumentLimit` CANNOT move** above the body parse: it needs `kbId`, which is read + **from the body** (`route.ts:45`, `formData.get('kb_id')`). No body, no `kbId`, no check. +- **`enforceLimit` should NOT move** above the body parse: it **increments the usage counter** + (register #24), so running it before a request is known-good would **burn an upload credit on + a rejected request**. + +So the only movable piece is `getUser()`, and its payoff is small because the edge cap already +bounds the exposure. Record it; do not prioritize it. + +### 2.3 β€” PIVOT_PLAN.md:269 asserts a FALSE claim ("Size cap (50 MB) already exists") + +`PIVOT_PLAN.md:269` (the **B5a** row) states *"Size cap (50 MB) already exists."* This is +**false**: the effective cap is **~4.5 MB** and is imposed by the **platform** (Β§1.1), not by +that code, and the app's own 50 MB check is unreachable-for-oversize on Vercel (Β§1.1). Recorded +here so the contradiction is on the record. **Do NOT edit `PIVOT_PLAN.md` in this PR** β€” this +document is the correction; the plan edit, if any, is a separate decision. + +--- + +## Section 3 β€” The split (with the reasoning, not just the labels) + +The hardening work divides along **what is doable now on the request path** versus **what +genuinely requires B6's off-request re-handling**, plus two enabling infrastructure moves. + +### B5b-1 β€” Request path, doable NOW, independent of B6 + +Each item here is a pure function of bytes the handler **already holds** (the uploaded file), so +none of them needs the file re-handled off the request path: + +- **Magic-byte verification** β€” check the leading signature against the declared extension + (`%PDF-` for pdf; `PK\x03\x04` for the OOXML trio). Pure function of bytes in hand. +- **Declared-size ZIP ratio pre-check** β€” read the **central directory** of the OOXML/ZIP + container and sum the *declared* uncompressed sizes. **O(entries), no expansion.** This is a + *cheap first filter*, not a truthful bound (see B5b-2 on why a lying directory survives it). +- **The auth reorder ride-along** (Β§2.2) β€” move `getUser()` above `formData()`. Low priority. + +### B5b-2 β€” Genuinely gated on B6 / register #22 + +Each item here requires either **actual expansion** (which you do not want to do synchronously on +the request path) or a **worker / external service** that does not exist today: + +- **Enforced expansion bounds** β€” a *truthful* measurement requires **actually expanding** the + archive and stopping at a byte ceiling. A **lying central directory** (declaring small sizes + while the streams expand hugely) passes the B5b-1 pre-check and only fails **here**, at real + expansion. Expansion is slow and memory-hungry β†’ it belongs off the request path (B6). +- **Nested-archive depth** β€” a bomb can be archives-within-archives; bounding depth means + recursively opening, i.e. expansion, i.e. B6. +- **Any AV / content scanning** β€” slow, and needs a scanner binary or service that is **not in + the image today** (Β§4 decision 3). Needs a worker. + +### (b1) β€” Move chunk+embedding persistence INTO the Railway service (the enabler of B6) + +Today the Railway `/convert` returns a **9–13Γ— JSON blob** (Β§1.3) that the Next route materializes +and inserts. Instead, have the **Railway service write chunks+embeddings to Supabase directly**. + +- **Size-independent** β€” helps regardless of any file-size ruling. +- **Fixes the return-path wall** (Β§1.3) β€” the 9–13Γ— blob never crosses the network; Next stops + holding it in memory. +- **Relocates the slow work** (embed batches, chunk inserts) to Railway β€” **the only component + without a duration ceiling** (Β§1.4; Vercel has the unmeasured default, Railway does not sit + behind it). +- **It is the ENABLER of B6, not B6 itself.** Even after (b1), the Next route would still + `await` Railway until **fire-and-forget** is adopted (return early, let Railway finish). So (b1) + removes the *payload* wall but not the *duration* wall; B6 proper is the fire-and-forget step. +- **Stands regardless of the file-size ruling** (Β§4 decision 1). + +### (b2) β€” Signed-URL direct-to-Supabase-storage upload + +The browser uploads **straight to Supabase storage** via a signed URL, bypassing Next entirely. + +- It is the **ONLY** way past the **4.5 MB inbound cap** (Β§1.1) β€” the bytes never traverse + Vercel's edge. +- **Gated entirely on the founder's file-size ruling** (Β§4 decision 1): if ~4.5 MB is accepted, + (b2) is unnecessary; only a *real* 50 MB (or larger) target justifies it. +- **Payoff is limited by register #22** β€” the free-tier Supabase **5 GB/month egress** caps the + benefit at roughly **~100 uploads/month at 50 MB** before the tier is exhausted. A 50 MB + capability the backend cannot afford to serve is a promise in the same family as the broken + 50 MB of Β§1.6. + +--- + +## Section 4 β€” Open decisions awaiting Abo Jad's ruling + +**Each is stated as a question with its consequences. This document does NOT pick values.** + +**1. Max file size.** Accept **~4.5 MB** (correct all 7 sites of Β§1.6, and route the message +through `t.*`) **vs.** deliver a **real 50 MB** via (b2) signed-URL upload. +*Consequence:* this decision **gates** decision 5 (enforcement point) and the sync/async posture β€” +a real-50 MB path is inherently async (b2 + B6), a ~4.5 MB path can stay synchronous. + +**2. Format list.** Keep **all six**, or **drop the ZIP-container trio (docx / pptx / xlsx)** that +carries **all** the decompression-bomb risk (Β§2.1). pdf/txt/md carry no ZIP-bomb surface. +*Consequence:* dropping the trio removes the B5b-2 expansion-bound and nested-depth work entirely; +keeping it makes that work mandatory before Phase 8. + +**3. What "scanning" means β€” in or out of scope.** +- **AV** β€” needs a scanner binary/service; **nothing in the image today** provides it. +- **PII detection** β€” its own service/model. +- **Prompt-injection in extracted text** β€” **record explicitly:** extracted document text flows + **unfiltered** into the Haiku prompt via `` interpolation in the summarize / quiz / + agent routes. **This is currently unmitigated.** At minimum it needs a **documented + accepted-risk decision**, even if active mitigation is deferred. +*Consequence:* each sub-item independently either enters scope (with a worker, per B5b-2) or is +recorded as an accepted risk. + +**4. Failure posture.** **Reject at upload** (user-visible **415**) **vs. accept-then-quarantine** +(store, scan out-of-band, release or delete). +*Consequence:* reject-at-upload is synchronous and simple but blocks the request on the check; +quarantine needs the async worker (B6) and a quarantine state on `documents`. + +**5. Enforcement point.** **Next route**, **Railway service**, or **both**. +*Consequence:* **the Railway service currently trusts its caller completely (bearer token only)** β€” +`_check_auth` is a single-token equality. Anything enforced **only in Next is bypassed if that +token leaks** (exactly the register-#45 exposure). Enforcing in Railway (or both) is the only +posture robust to a leaked `INGESTION_TOKEN`. + +**6. The three ZIP numbers.** Each needs **a value AND a measurement decision**: +- **expansion ratio** (max expanded Γ· compressed) +- **absolute expanded cap** (hard byte ceiling) +- **nesting depth** (max archive-within-archive levels) +For each: is it measured from the **declared central-directory sizes** (cheap, **spoofable** β€” a +lying directory passes) or from **actual expansion** (accurate, **expensive**, needs B6)? +*Consequence:* declared-size checks are B5b-1 (now); actual-expansion checks are B5b-2 (gated on +B6). + +**7. Signature table.** Fix the exact signatures: +- `%PDF-` for **pdf** +- `PK\x03\x04` **plus inner part-path verification** for the **OOXML trio** β€” a bare ZIP check + **cannot distinguish docx from pptx from xlsx from a bomb**; the inner part path + (`word/`, `ppt/`, `xl/`) is what identifies the real type. +- the **txt / md non-signature rule** (Β§1.5) β€” UTF-8 decodability / NUL rejection, **not** a + signature match. +*Consequence:* the table is only correct if it treats the OOXML trio structurally and txt/md +without a signature; a naive "starts with PK" rule both mis-types and mis-defends. + +--- + +## Section 5 β€” Required measurement before any user-facing number ships + +**Question:** at what file size does the upload **time out** (Β§1.4)? + +**Procedure.** Upload **plain `.txt`** files to a **PREVIEW** deployment. Plain text is used +deliberately: **extracted text β‰ˆ file size**, so the variable is clean. A PDF's file size is +dominated by images and tells you **nothing** about the extracted-text volume that actually drives +chunk count / embed batches / duration. + +**Failure-signature table (how to read each result):** + +| Observed result | Meaning | +|---|---| +| `200` + a `chunk_count` | **pass** β€” the file ingested | +| `413` + **Vercel HTML** | the **inbound edge cap** (Β§1.1) β€” browserβ†’Next leg | +| `413` + **the app's JSON** message | the **app's own** 50 MB check (`route.ts:52`) | +| `500` + `"Ingestion failed"` | the `!pyResponse.ok` path (`route.ts:161-162`) | +| `504` / no response | **duration** β€” the ceiling we are actually hunting (Β§1.4) | + +**Cost discipline:** each upload **burns a rate-limit credit** (`enforceLimit('upload')`, +`route.ts:102`) and **consumes the register-#22 free-tier database** (storage + egress). Use a +**throwaway subject** and **delete the rows afterward**. + +--- + +## Cross-references + +- **Register #45** β€” the live duplicate Vercel ingestion service, found and closed this session + while scoping B5b. Distinct incident; its closure is why decision 5 stresses the leaked-token + posture. +- **Register #22** β€” free-tier backend limits; bounds (b2)'s payoff (Β§3) and Β§5's test budget. +- **Register #24** β€” `enforceLimit` charges before the paid call; why `enforceLimit` must not + move above the body parse (Β§2.2) and why Β§5 uploads cost credits. +- **Register #17** β€” hardcoded English bypassing `t.*`; the same defect class as `route.ts:52` + (Β§1.6, site 2). +- **B6 (PIVOT_PLAN Β§8)** β€” synchronous ingestion; the gate for every B5b-2 item and for (b1)/(b2).