diff --git a/.github/workflows/ingestion-image.yml b/.github/workflows/ingestion-image.yml new file mode 100644 index 0000000..c37791d --- /dev/null +++ b/.github/workflows/ingestion-image.yml @@ -0,0 +1,168 @@ +name: ingestion-image + +# Builds the Railway ingestion image the way Railway builds it, then proves the +# module actually imports and the app actually serves inside that image. +# +# WHY THIS EXISTS. The two existing required checks are `tsc` and `db-types`. +# Neither inspects a single line of Python. A PR that changes only +# services/ingestion/ therefore passes both VACUOUSLY -- a green checkmark that +# vouches for nothing -- and Railway's "Wait for CI" is OFF, so the deploy does +# not even pause for them. The failure this closes is specific: main.py imports +# `acreate_client`/`AsyncClient`/`AsyncClientOptions` from `supabase` at MODULE +# SCOPE, and `tiktoken.get_encoding("cl100k_base")` also runs at module scope, so +# a bad resolve or an unreachable encoder file is a BOOT-time crash. All four +# endpoints -- /health, /embed, /convert, /ingest -- live in one uvicorn process, +# so a boot crash takes the Ask path down with the upload path. +# +# WHAT IT DELIBERATELY DOES NOT PROVE. It does not prove the Railway environment +# is correct: the CI container has no SUPABASE_URL or SUPABASE_ANON_KEY, so its +# /health correctly reports "supabase_configured": false. That distinction is +# load-bearing -- see docs/b1-verification-protocol.md items 1 and 2. This job is +# item 1's CI twin; item 2 has no CI twin and cannot have one. +# +# BUILD PARITY. Railway's "knowflow" service has Root Directory /services/ingestion +# and an empty Dockerfile Path, so its build CONTEXT is services/ingestion and its +# Dockerfile is services/ingestion/Dockerfile. The build below uses exactly that +# context. Changing either here without changing it there makes this job green on +# an image production never builds. +# +# NOT PINNED BY DIGEST, DELIBERATELY. services/ingestion/Dockerfile says +# `FROM python:3.11-slim`, a mutable tag. Pinning the base by digest HERE but not +# THERE would make CI test an image production does not build -- a silent +# false-green, which is worse than the flakiness it would avoid. If the base +# should be pinned, pin it in the Dockerfile so both paths move together. See +# register #43's residual (1): Docker Hub is anonymous-rate-limited per IP and +# GitHub runners share NAT pools, so this job CAN red on a third party's traffic. +# That is the explicit reason it is NOT a required check -- see the PR body. +# +# Actions minutes are not billed: this repository is PUBLIC. + +on: + pull_request: + branches: [main] + push: + branches: [main] + +# No `paths:` filter, deliberately. A path-filtered job never reports a status on +# PRs it skips; if this check is ever promoted to required, a rule demanding a +# context that nothing publishes blocks every unrelated PR indefinitely. Running +# a few unnecessary minutes on TypeScript-only PRs is the cheaper mistake, and +# minutes are free here. + +# Least privilege: this job reads the tree and builds a container from it. It never +# writes to the repo, never pushes an image, never posts a comment, and needs NO +# secret of any kind -- not a registry credential, not a Supabase key, not the +# INGESTION_TOKEN. It cannot reach, read, or mutate production. +permissions: + contents: read + +concurrency: + group: ingestion-image-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + # Job id is the status-check context string. Keep it stable: renaming it + # silently detaches any branch-protection rule that requires the old name, and + # a rule requiring a context nothing publishes blocks every PR indefinitely. + # (Same warning as db-types.yml:39-41, for the same reason.) + ingestion-image: + name: ingestion-image + # Pinned, not `ubuntu-latest`: `latest` silently rolls to the next LTS image + # and can change the preinstalled Docker version with no commit to this repo. + runs-on: ubuntu-24.04 + timeout-minutes: 20 + + steps: + # Actions are pinned to immutable commit SHAs, not tags. Tags like `v7` are + # mutable and are republished in place, so a tag pin would NOT prevent an + # action's behaviour from changing under us. + - name: Check out the repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + + # Build with Railway's context, not the repo root. `docker build` is used + # rather than buildx + a cache backend: a cache would make a green run + # depend on a cache entry rather than on a clean build, which is the + # opposite of what this gate is for. + - name: Build the ingestion image + run: | + docker build \ + -f services/ingestion/Dockerfile \ + -t knowflow-ingestion:ci \ + services/ingestion + + # Import the module INSIDE the built image, at the image's own WORKDIR + # (/app), with the image's own interpreter. This is the check that a bad + # `supabase` resolve or a missing tiktoken encoder is caught here rather + # than on Railway. The full traceback is printed so a NETWORK failure + # (tiktoken fetching cl100k_base, PyPI, DNS) is distinguishable from a real + # ImportError -- they are not the same defect and must not read the same. + # + # `-i` IS LOAD-BEARING. DO NOT REMOVE IT. `python -` reads its program from + # STDIN, and `docker run` WITHOUT `-i` does not attach stdin to the + # container. The first version of this workflow omitted it (PR #71, run + # 30218194135): python read EOF, executed an EMPTY program, exited 0, and + # the step reported GREEN having asserted nothing -- zero bytes of stdout, + # no ROUTES line, no IMPORT OK. A workflow whose entire purpose is to end + # vacuous green checks shipped one. Proven by direct experiment, not + # inferred: without `-i` the heredoc never reaches the interpreter and the + # exit code is 0; with `-i` it runs. + # + # The trailing `grep` is what actually fails this step, and it is the + # STRUCTURAL fix rather than a second opinion. GitHub runs steps with + # `bash -e` and NOT `-o pipefail` (see `shell: /usr/bin/bash -e {0}` in any + # job log), so piping through `tee` MASKS a non-zero `docker run`. Asserting + # on the step's OUTPUT instead of on its exit code is the only thing that + # makes a silent no-op impossible to pass again -- which is the same lesson + # register #52 exists to record. + - name: Import the module inside the image + run: | + docker run --rm -i knowflow-ingestion:ci python - <<'PY' | tee /tmp/import.log + import traceback, sys + try: + import main + except Exception: + print("IMPORT FAILED -- full traceback follows.", flush=True) + print("If this is a network error (tiktoken/cl100k_base, PyPI, DNS),", flush=True) + print("it is register #43's class, NOT a code defect. Read before re-running.", flush=True) + traceback.print_exc() + sys.exit(1) + + paths = sorted({r.path for r in main.app.routes}) + print("ROUTES:", paths, flush=True) + + # Assert only the endpoints that exist in EVERY version of this service: + # before PR A (/convert, /embed, /health), after PR A (all four), and + # after PR C (/ingest, /embed, /health). Asserting a version-specific + # route here would make this workflow need editing in lockstep with the + # service, which is exactly the drift it exists to prevent. + for required in ("/health", "/embed"): + assert required in paths, f"missing route: {required}" + + print("IMPORT OK", flush=True) + PY + grep -q "IMPORT OK" /tmp/import.log + + # Stronger than an import: actually run the image's own CMD and prove uvicorn + # binds and serves. This catches ASGI/lifespan startup failures that a bare + # import does not. + - name: Boot the container and probe /health + run: | + docker run -d --name ingestion-ci -p 8000:8000 knowflow-ingestion:ci + for i in $(seq 1 30); do + if curl -fsS http://127.0.0.1:8000/health > /tmp/health.json; then + echo "HEALTH: $(cat /tmp/health.json)" + # Assert the boot invariant only. `supabase_configured` is EXPECTED + # to be false here -- CI has no Supabase env vars, by design. Only + # the live probe (protocol item 2 / V2) can assert it is true. + grep -q '"ok":true' /tmp/health.json + exit 0 + fi + sleep 2 + done + echo "Container never served /health. Logs follow:" + docker logs ingestion-ci + exit 1 + + - name: Container logs (always) + if: always() + run: docker logs ingestion-ci || true diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index a3eff28..6224300 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -5,9 +5,9 @@ commit history + [`PIVOT_PLAN.md`](./PIVOT_PLAN.md) + project memory. Update thi file at the **end of every step** so we never jump ahead on a deferred item or skip a main phase. -- **Last updated:** 2026-07-23 - see [PROGRESS §7 Changelog](#7-changelog). This field carries the date only; the narrative lives in §7. -- **Active branch:** `docs/b5b-scoping-record` - this docs-only PR RECORDS the **B5b upload-content-hardening scoping** as register **#50** and creates the decision record [`b5b-scoping.md`](./b5b-scoping.md): the established+verified constraints (the real ceiling is Vercel's hard ~4.5 MB edge cap, not the app's broken 50 MB promise; duration is the unmeasured true limit), the corrections (the cap does not stop ZIP bombs; `PIVOT_PLAN.md:269`'s 50 MB claim is false), the B5b-1/B5b-2 split plus enablers (b1)/(b2), and **seven OPEN founder decisions** with an OUTSTANDING duration measurement. Touches **only** two files: `docs/b5b-scoping.md` (new) and this file (one new register row #50, one new §7 block, and these three header fields). **No code, no route/validation/size-constant change, no other register row's substance, no §1-§6 phase content; B5b §1 row stays ⬜ (scoping is not progress), `PIVOT_PLAN.md` untouched.** Register **#17** stays OPEN. -- **Main tip:** `1e37c03` (PR **#68** merge - the docs-only close of register #45 and logging of design items #46-#49), on top of `ecb1ed5` (PR **#67** - the duplicate Vercel ingestion service removed from `vercel.json` + `INGESTION_TOKEN` rotated) and `2d0856d` (PR **#66** - the §7 changelog migration off the line-8 header field, all 17 entry bodies byte-identical). Both required checks green on `1e37c03` (`tsc`, `db-types`). +- **Last updated:** 2026-07-26 - see [PROGRESS §7 Changelog](#7-changelog). This field carries the date only; the narrative lives in §7. +- **Active branch:** `chore/b1a0-ingestion-image-ci` - **PR A0** of the three-PR (b1) sequence, and **CI-ONLY**. Adds `.github/workflows/ingestion-image.yml`, which builds `services/ingestion/` through **Railway's own build context** and then imports the module and boots the container **inside the built image**. It exists because `tsc` and `db-types` inspect **no Python at all**, so a service-only PR passes both **vacuously** while Railway's "Wait for CI" is **OFF**. Opens register **#52** (the vacuous-green gap + this check's promotion rule) and register **#53** (the `tiktoken` cold-start network dependency the workflow surfaced, which is **pre-existing on `main`** and **not** a (b1) regression). **#51 is deliberately skipped here and reserved for (b1)'s own row, which PR A adds.** Touches **two** files: the new workflow and this one (two register rows, one §7 block, these three header fields). **Zero files under `src/`, zero files under `services/`; no service code, no migration, no DB change, no dashboard change; the check is deliberately NOT added to branch protection.** Register **#17** stays OPEN. +- **Main tip:** `140bc31` (PR **#69** merge - the docs-only record of the B5b upload-hardening scoping, register #50 + `docs/b5b-scoping.md`), on top of `1e37c03` (PR **#68** - the close of register #45 and logging of design items #46-#49) and `ecb1ed5` (PR **#67** - the duplicate Vercel ingestion service removed from `vercel.json` + `INGESTION_TOKEN` rotated). Both required checks green on `140bc31` (`tsc`, `db-types`) - verified this session via the check-runs API, not remembered. > ✅ **RESOLVED 2026-07-04 — Supabase project back online (register #21).** The > project **auto-paused** (free-tier inactivity) and then hit Supabase's free @@ -147,6 +147,8 @@ project memory, and per-step review debates. "Where" is the target phase/trigger | 48 🔎 **OPEN — product-design item, raised 2026-07-23 (Abo Jad design review). DESIGN-QUALITY, NOT A BUG.** | **Pre-login landing page has no navigation menu.** | Not a defect — a missing-nav/UX item for the mobile app; no phase assigned. | Wanted: a **hamburger (three-line) icon** opposite the KnowFlow wordmark in the header, opening a **main dropdown menu** for the mobile app. | | 49 🔎 **OPEN — product-design item, raised 2026-07-23 (Abo Jad design review). Carries a correctness rider, not pure polish.** | **The landing page's terminal-style demo window shows static text.** **Cross-ref #20:** that window currently displays **"Ready in 0.4s"**, a timing claim **#20 already flags**, and this session proved **ingestion is synchronous and takes real seconds** — so the number MUST be corrected or removed as part of this work rather than re-shipping a false claim. | Not pure polish — the embedded "0.4s" IS a false claim (see **#20**), so this row carries a correctness rider; no phase assigned. | Wanted: **animate the terminal** so the text **types out live from start to finish**; as part of the same work, **correct or remove "Ready in 0.4s"** (per **#20**) rather than re-ship it. | | 50 🔎 **SCOPING RECORDED 2026-07-23 — B5b upload content hardening. Decisions 1-7 OPEN pending founder rulings; the duration measurement OUTSTANDING. Scoping is NOT progress — B5b (§1) stays ⬜.** | **B5b (deep upload content hardening: magic-byte, decompression-bomb / nested-archive limits, content scanning) is now scoped in [`b5b-scoping.md`](./b5b-scoping.md).** Established + verified this session: the real upload ceiling is **Vercel's hard ~4.5 MB edge cap** (platform constant, unraisable), **not** the 50 MB the app promises in **7 places** (`route.ts:51/:52`, `DropZone.tsx:26`, four i18n strings — the `:52` message is hardcoded English, register-#17 class); ingestion runs on **Railway** (`INGESTION_SERVICE_URL`), so the `route.ts`→`/convert` hop faces no second Vercel cap but a **9-13× return-path blob** (each chunk's 1024-float embedding inline + full markdown, materialized by `await pyResponse.json()`); and **no `maxDuration` is set anywhere** (grep-empty), so the true user-facing ceiling is an **UNMEASURED duration limit**, not size. Corrections recorded so they are not re-inherited: the 4.5 MB cap does **NOT** mitigate decompression bombs (compressed bytes are bounded, expansion is not); the pre-auth body-buffering concern is **bounded** by the edge cap, so the `getUser`-above-`formData` reorder is a low-priority ride-along (`checkDocumentLimit` **cannot** move — needs `kbId` from the body; `enforceLimit` **should not** — it would burn an upload credit on a rejected request); and **`PIVOT_PLAN.md:269`'s "Size cap (50 MB) already exists" is FALSE** (the effective cap is the platform's ~4.5 MB). | Not a bug and not deferred work — a **decision record** written before implementation so the founder's rulings are captured, not inferred. The split is recorded (B5b-1 request-path magic-byte + declared-size ZIP-ratio pre-check + the auth reorder, doable now; B5b-2 enforced expansion bounds + nested depth + AV/scanning, genuinely gated on **B6**; plus enabler moves (b1) persist chunks from Railway and (b2) signed-URL direct-to-storage). **No code, no route/validation/size-constant change; `PIVOT_PLAN.md` untouched.** | **Seven OPEN founder decisions** (each stated with consequences in the doc, none picked here): (1) max file size — accept ~4.5 MB vs. deliver real 50 MB via (b2); (2) format list — keep all six vs. drop the docx/pptx/xlsx ZIP trio that carries all bomb risk; (3) scanning scope — AV (nothing in the image today), PII, and **prompt-injection in extracted text** (flows unfiltered into Haiku via `` interpolation — currently unmitigated, needs at minimum a documented accepted-risk decision); (4) failure posture — reject-at-upload (415) vs. accept-then-quarantine; (5) enforcement point — Next, Railway, or both (Railway trusts its caller on a single bearer token, so Next-only is bypassed if `INGESTION_TOKEN` leaks); (6) the three ZIP numbers (expansion ratio, absolute expanded cap, nesting depth) — each measured from the spoofable declared central directory (cheap) or actual expansion (accurate, needs B6); (7) the signature table (`%PDF-`; `PK\x03\x04` + inner part-path for the OOXML trio; the txt/md non-signature UTF-8/NUL rule). **OUTSTANDING measurement:** at what file size does the upload time out — plain-`.txt`-on-Preview procedure + failure-signature table recorded in the doc §5 (each upload burns a rate-limit credit and the #22 free tier — throwaway subject, delete rows after). Related: **#45** (the duplicate ingestion service closed while scoping this), **#22** (bounds (b2) and the §5 test budget), **#24** (why `enforceLimit` cannot move), **#17** (the `:52` hardcoded-English class). | +| 52 🔎 **OPEN — standing CI gap, opened 2026-07-26 (PR A0). The check now EXISTS but is deliberately NOT a required status check, and the row stays OPEN until the promotion rule below is either satisfied or formally abandoned.** | **Nothing in this repository inspected a single line of Python.** The two required checks are `tsc` (`typecheck.yml`) and `db-types` (`db-types.yml`); neither reads `services/ingestion/`. Neither carries a `paths:` filter, so both **do run and do report green** on a service-only PR — a **vacuous pass**, a checkmark that vouches for nothing. Compounding it: Railway's **"Wait for CI" is OFF** and **Watch Paths is empty**, so a merge to `main` starts the image build immediately and does not pause for those checks even in principle. The concrete failure this leaves open: `services/ingestion/main.py` imports `acreate_client`/`AsyncClient`/`AsyncClientOptions` from `supabase` at **module scope** (`:13`) and calls `tiktoken.get_encoding` at **module scope** (`:50`), so a bad resolve is a **boot-time crash**, and all four endpoints share one uvicorn process — meaning a boot crash takes **the Ask path down with the upload path**. | Not a bug in shipped behaviour — a **gate gap**, and it is broader than (b1): it predates the (b1) work and outlives it, which is why it is its own row rather than folded into **#51**. Closed *mechanically* by `.github/workflows/ingestion-image.yml` (PR A0), which builds through **Railway's own context** (Root Directory `/services/ingestion`, empty Dockerfile Path — read from the dashboard 2026-07-26), then **imports the module** and **boots the container** inside the built image. It deliberately proves **boot only, never environment**: CI has no `SUPABASE_*` vars, so its `/health` correctly reports `supabase_configured: false`. **FIRST INSTANCE OF THIS ROW'S OWN THESIS, and it was the fix itself:** the workflow's first version omitted `-i` on `docker run`, so `python -` read EOF from an unattached stdin, ran an **empty program**, exited **0**, and the import step reported **green having asserted nothing** (run **30218194135**, zero bytes of stdout). Fixed twice over — `-i`, plus a trailing **`grep -q "IMPORT OK"`** on a `tee`-captured log, because GitHub runs steps with `bash -e` and **no `-o pipefail`**, so an exit code alone can be masked by a pipe. **The rule this row now carries: a check step must assert on its OUTPUT, never on its exit code alone.** That a purpose-built anti-vacuous-green check shipped a vacuous green on its first attempt is the strongest available evidence that this failure mode is easy to miss by inspection and must be closed structurally. | **PROMOTION RULE, binding:** promote to a required status check **only** after **20 consecutive green runs on `main` with zero re-runs**, and record the count and the decision in a §7 block. **Any red caused by a third party's network — Docker Hub, PyPI, or the `tiktoken` fetch (#53) — RESETS the counter to zero** and *weakens* rather than strengthens the case. The reason is **#43**, which is this exact class: `db-types` redded `main` **twice** on `docker: toomanyrequests` with no commit to this repo, and its residual (1) states plainly that Docker Hub is **also** anonymous-rate-limited per IP with GitHub runners sharing NAT pools. #43 quotes the governing principle: *"A gate that reds `main` on a coin flip is worse than no gate"*, and *"a required check that reds on a third party's traffic trains people to re-run it."* This job has **three** third-party network dependencies in its hot path (Docker Hub, PyPI, Azure blob via #53) where `db-types` now has one, digest-pinned. **It is therefore NOT required today, and it may never be — that is an acceptable outcome, not a failure of this row.** Mechanically it also *cannot* be required until a green run on `main` exists, per `typecheck.yml:11-12`. Related: **#43**, **#44**, **#53**. | +| 53 🔎 **OPEN — production cold-start dependency on an unpinned third party, opened 2026-07-26 (PR A0). PRE-EXISTING ON `main`, NOT a (b1) regression: no fix is attempted here.** | **The ingestion service makes an outbound HTTPS call to Azure blob storage at import time, on every cold container start, before it can serve its first request.** `services/ingestion/main.py:50` runs `ENCODER = tiktoken.get_encoding("cl100k_base")` at **module scope**. Verified in the vendored package rather than assumed: `tiktoken_ext/openai_public.py:77` resolves that encoding to `https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken`, and `tiktoken/load.py:36` `read_file_cached` is **cache-first** (`TIKTOKEN_CACHE_DIR` at `:38`, otherwise a temp-dir cache). A **fresh container's cache is empty**, so the fetch is real on every cold start. Nothing in this repo pins, vendors, caches, or even records that dependency — until now. | Same **class** as **#43** and **#44** (an unpinned third party sitting in a hot path where an outage or throttle produces a failure with no commit to this repo), but in the **production service**, not in CI. Surfaced while drafting PR A0's workflow, whose import step exercises exactly this call — which is a point in the check's favour, and also the reason a network failure there is made loudly distinguishable from a real `ImportError` in the job's output. **Blast radius if it fails:** the container never finishes importing, so `/health`, `/embed`, `/convert` and `/ingest` are all down together — the Ask path with the upload path. | **Deliberately NOT fixed in PR A0**, which is CI-only and must not touch `services/ingestion/`. Candidate fixes, none chosen: **(a)** bake the encoder into the image — set `TIKTOKEN_CACHE_DIR` and warm it in a build step, so the fetch happens at **build** time where a failure fails the build instead of the boot; **(b)** vendor the `.tiktoken` file and load from disk. Both change the production image and therefore belong to a **service** PR, not this one. **Also unmeasured:** how long that fetch adds to cold start, and what the service does if Azure is unreachable at boot rather than slow. Related: **#43**, **#44**, **#52**. | **Founder-owned open items (PIVOT_PLAN §10) — not engineering-blocked, tracked for launch:** Next.js hosting decision (Vercel Pro vs self-host, ~Phase 7/10) · **Supabase backend @@ -208,6 +210,10 @@ bodies total 37,095 bytes. **This retires the Option C frozen-tail invariant by existed only to police a boundary inside an unreviewable single line, and the append-only rule above supersedes it. No bespoke hash is needed for future updates: the diff is the proof. +### 2026-07-26 - (b1) PR A0 of 3: Python finally has a CI gate (registers #52, #53) + +(**PR A0 is CI-ONLY and changes no service behaviour whatsoever — zero files under `src/`, zero files under `services/`.** It adds one workflow, `.github/workflows/ingestion-image.yml`, and this file. **THE GAP IT CLOSES (register #52):** the two required checks are `tsc` and `db-types`, and **neither inspects a single line of Python**. Neither carries a `paths:` filter, so both **run and report green** on a service-only PR — a **vacuous pass**, a checkmark that vouches for nothing. Railway's **"Wait for CI" is OFF** and **Watch Paths is empty**, so a merge to `main` starts the image build immediately and would not pause for those checks even if they meant something. Meanwhile `services/ingestion/main.py` imports `supabase` symbols at **module scope** (`:13`) and calls `tiktoken.get_encoding` at **module scope** (`:50`), so a bad resolve is a **boot-time crash**, and all four endpoints share **one uvicorn process** — a boot crash takes the **Ask path** down with the **upload path**. **WHAT THE CHECK PROVES:** it builds `services/ingestion/` through **Railway's own build context** (Root Directory `/services/ingestion`, empty Dockerfile Path — read from the dashboard 2026-07-26, not assumed), then **imports the module inside the built image** and **boots the container and probes `/health`** — which is strictly stronger than an import, because it catches ASGI/lifespan startup failures a bare import does not. **WHAT IT DELIBERATELY DOES NOT PROVE:** that the Railway **environment** is correct. CI has no `SUPABASE_URL`/`SUPABASE_ANON_KEY`, so its `/health` correctly reports **`supabase_configured: false`**, and the job asserts only `"ok":true`. That separation is load-bearing and must not be collapsed: this job is the CI twin of the **live boot probe**, and the **environment probe has no CI twin and cannot have one**. **THE FIRST VERSION OF THIS CHECK SHIPPED THE EXACT DEFECT IT EXISTS TO END, AND ITS OWN FIRST RUN EXPOSED IT — recorded here rather than quietly amended away.** Run **30218194135** on PR **#71** reported the `ingestion-image` job **GREEN in 38 seconds**, and its import step emitted **zero bytes of stdout**: no `ROUTES:` line, no `IMPORT OK`, no `IMPORT FAILED`. Cause, proven by direct experiment rather than inferred: `python -` reads its program from **stdin**, and `docker run` **without `-i`** does not attach stdin to the container, so python read EOF, executed an **empty program**, exited **0**, and the step passed having asserted nothing. **A workflow written to end vacuous green checks shipped a vacuous green check.** **Severity, stated accurately rather than dramatically:** the job was not proving *nothing* about the import — the boot probe transitively proves the module imports, because uvicorn cannot serve `/health` without importing `main:app`. What the dead step actually cost was the **route assertions**, the `ROUTES:` evidence line, and the **network-vs-`ImportError` diagnostic banner**. **TWO fixes, and the second is the one that matters:** (1) add `-i`; (2) pipe the step through `tee` and make a trailing **`grep -q "IMPORT OK"`** the thing that actually fails it. GitHub runs steps with **`bash -e` and NOT `-o pipefail`** (`shell: /usr/bin/bash -e {0}`), so `tee` would otherwise mask a non-zero `docker run`. Both paths were tested locally under `bash -e` before pushing: with `-i` the step exits **0** and prints `IMPORT OK`; with `-i` removed the step now exits **1**, so the regression cannot silently return. **The generalised rule, and the reason register #52 exists: a check step must assert on its own OUTPUT, not merely on its exit code.** **AUDIT OF THE REMAINING STEPS under that same lens, so the fix is not mistaken for a sweep:** `Build the ingestion image` is **not** exposed — its exit code and its effect are the same fact, since `COPY requirements.txt` / `COPY main.py` make a wrong context or a missing file fail the build loudly, and there is nothing further to assert on; `Boot the container and probe /health` already asserts on its **payload** (`grep -q '"ok":true'`), and that grep sits in the `if` **body** where `bash -e` applies, so a wrong payload does red the step; `Check out the repository` is self-verifying downstream; `Container logs (always)` is `|| true` by design and claims nothing. **The import step was the only exposed one. No other fix was invented.** **NOT A REQUIRED CHECK, DELIBERATELY, AND NOT ADDED TO BRANCH PROTECTION.** This job has **three** third-party network dependencies in its hot path — Docker Hub for `python:3.11-slim`, PyPI for `markitdown[all]` + `supabase`, and Azure blob for `tiktoken`'s `cl100k_base` (**#53**) — where `db-types` now has **one**, digest-pinned. **Register #43 is precisely this class:** it redded `main` **twice** on `docker: toomanyrequests` with **no commit to this repo**, and its residual (1) records that Docker Hub is **also** anonymous-rate-limited per IP with GitHub runners sharing NAT pools. #43's governing quote applies unchanged: *"A gate that reds `main` on a coin flip is worse than no gate."* **PROMOTION RULE, binding and recorded in #52:** required-check status is considered **only** after **20 consecutive green runs on `main` with zero re-runs**; **any red caused by a third party's network RESETS the counter to zero** and weakens rather than strengthens the case. Never being promoted is an **acceptable outcome**. **The base image is NOT digest-pinned in the workflow, deliberately:** `services/ingestion/Dockerfile` says `FROM python:3.11-slim`, and pinning in CI but not in the Dockerfile would make CI green on an image production never builds — a **silent false-green**, worse than the flakiness it would avoid. If the base should be pinned, it must be pinned in the Dockerfile so both paths move together; that is a service PR, not this one. **REGISTER CHOICES, argued rather than assumed:** **#52** is its own row rather than folded into **#51** because the vacuous-green gap **predates (b1) and outlives it** — #51 closes when PR C lands, and #52's promotion obligation must not close with it. **#53** is its own row because it is a **production** cold-start dependency, not a CI one, and it is **pre-existing on `main`** — recording it under a (b1) row would misattribute it as a (b1) regression, which it is not. **#51 is deliberately SKIPPED here and reserved for (b1)'s own row, which PR A adds** — PR **#70**'s body and the (b1) branch already name it #51, and renumbering it now would create drift with text that already exists. **RUNTIME IS NOW MEASURED, NOT ESTIMATED: 38 seconds** on run **30218194135** (build ~24s), against a pre-run projection of **5-9 minutes** — **wrong by an order of magnitude**, and the projection is recorded here rather than deleted so the next estimate is made with a known error bar. The reason it was wrong: every dependency in `markitdown[all]` resolves to a **prebuilt manylinux wheel** and GitHub's path to PyPI is very fast, so nothing is compiled from source. **Two riders on that number.** It was measured against **`main`'s `requirements.txt`, which does NOT yet contain `supabase==2.29.0`** — **PR A's run will be slower**, though on this evidence nowhere near the original projection. And it was measured on the **defective** run, whose import step was a no-op; the corrected step adds a real container start, so the true figure is slightly higher again. `timeout-minutes: 20` now carries far more headroom than intended, which is left as-is: a generous ceiling costs nothing and a hung third-party pull still fails loudly instead of burning the 6-hour default. **The first green run on `main` of the CORRECTED workflow is what satisfies the sequence's V11 condition** — the 38s figure above came from a run whose central assertion did not execute, so it pins the runtime and nothing else. **Costs nothing:** this repository is **PUBLIC** (`gh repo view` → `visibility: PUBLIC`), and GitHub bills Actions minutes only for private repositories. **Needs NO secret** — no registry credential, no Supabase key, not `INGESTION_TOKEN`; `permissions: contents: read` is the entire grant, so the job cannot reach, read, or mutate production. **Deliberately NOT in this PR:** `docs/b1-verification-protocol.md` (belongs to PR A), any change to `services/ingestion/`, any branch-protection change, any dashboard change, and the closing of PR **#70**. **Main tip verified this session** via the check-runs API rather than recalled: `tsc` and `db-types` both `success` on `140bc31`.) + ### 2026-07-23 - B5b upload-hardening scoping recorded (register #50, docs/b5b-scoping.md) (**B5b (deep upload content hardening) is now SCOPED, not started — register #50 opened, `docs/b5b-scoping.md` created; §1 B5b stays ⬜ because scoping is not progress on the gate.** The doc records this session's established + verified facts, its corrections, the split, and seven OPEN founder decisions. **Established (each verified against the code this session):** the real upload ceiling is Vercel's **hard ~4.5 MB edge cap** (platform constant, unraisable — `413 FUNCTION_PAYLOAD_TOO_LARGE` before the handler), **not** the 50 MB the app promises in **7 places** (`route.ts:51` check / `:52` hardcoded-English 413 message — register-#17 class, `DropZone.tsx:26`, and four i18n strings in `en.ts`/`ar.ts`); ingestion runs on **Railway** (`INGESTION_SERVICE_URL = knowflow-production.up.railway.app`), so the `route.ts`→`/convert` hop faces no second Vercel cap but a **9-13× return-path blob** — `/convert` returns each chunk's **1024-float embedding inline plus full markdown** (`main.py`: `EMBED_DIM=1024`, `CHUNK_TOKENS=512`, `CHUNK_OVERLAP=64`), and `await pyResponse.json()` materializes it in the Next function's memory; and **no `maxDuration` is configured anywhere** (grep-empty across `vercel.json`/`next.config.ts`/`package.json`/`src`), with Voyage embeds batched 128-at-60s and Supabase inserts batched 50, all sequential in one invocation — so the true user-facing ceiling is an **UNMEASURED duration limit**, not size. **Corrections recorded so they are not re-inherited:** the 4.5 MB cap does **NOT** mitigate decompression bombs (it bounds compressed bytes, never the expansion — 4.5 MB of crafted ZIP still exhausts the service); the pre-auth body-buffering concern is **bounded** by the edge cap, making the `getUser`-above-`formData` reorder a low-priority ride-along (`checkDocumentLimit` **cannot** move — it needs `kbId` from the body; `enforceLimit` **should not** — it would burn an upload credit on a rejected request, register #24); and **`PIVOT_PLAN.md:269`'s "Size cap (50 MB) already exists" is a documented FALSE claim** (the effective cap is the platform's ~4.5 MB), corrected here, `PIVOT_PLAN.md` deliberately untouched. **The split:** B5b-1 (request-path, doable now — magic-byte verification, a declared-size ZIP-ratio pre-check from the central directory, the auth reorder); B5b-2 (gated on **B6** — enforced expansion bounds, nested-archive depth, AV/content scanning, all needing actual expansion or a worker); plus enabler moves **(b1)** move chunk+embedding persistence into the Railway service (kills the 9-13× blob, relocates slow work to the only component without a duration ceiling — the enabler of B6, not B6 itself) and **(b2)** signed-URL direct-to-Supabase-storage upload (the only way past the 4.5 MB inbound cap; gated on the file-size ruling; #22's 5 GB/mo egress limits its payoff to ~100 uploads/mo at 50 MB). **Seven OPEN decisions (stated with consequences, none picked): (1)** max file size — ~4.5 MB vs. real 50 MB via (b2), gating the enforcement point and sync/async posture; **(2)** format list — keep six vs. drop the docx/pptx/xlsx ZIP trio that carries all bomb risk; **(3)** scanning scope — AV (nothing in the image today), PII, and **prompt-injection in extracted text** (flows unfiltered into Haiku via `` interpolation across summarize/quiz/agent — unmitigated, needs at minimum a documented accepted-risk decision); **(4)** failure posture — reject-at-upload (415) vs. accept-then-quarantine; **(5)** enforcement point — Next, Railway, or both (Railway trusts its caller on a single bearer token, so Next-only is bypassed if `INGESTION_TOKEN` leaks); **(6)** the three ZIP numbers (expansion ratio, absolute expanded cap, nesting depth) — each measured from the spoofable declared central directory vs. actual expansion; **(7)** the signature table (`%PDF-`; `PK\x03\x04` + inner part-path for the OOXML trio; the txt/md non-signature UTF-8/NUL rule — txt/md have no magic bytes at all). **OUTSTANDING measurement:** the duration ceiling — plain-`.txt`-on-Preview procedure + a failure-signature table (`200`+chunk_count=pass; `413`+Vercel HTML=edge cap; `413`+app JSON=app check; `500`+"Ingestion failed"; `504`/none=duration), each upload burning a rate-limit credit and the #22 free tier. Related registers: **#45** (the duplicate ingestion service closed while scoping this), **#22**, **#24**, **#17**. **Docs-only: no code, no migration, no DB, no route/validation/size-constant change; only `docs/b5b-scoping.md` (new) and this file changed; `Phase 7` stays ⬜, §1 B5b row untouched, `PIVOT_PLAN.md` untouched, register #17 stays OPEN.**)