diff --git a/docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md b/docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md index 33e7a9e..e5a3d89 100644 --- a/docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md +++ b/docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md @@ -33,21 +33,36 @@ Kopfzeile/Fußzeile/main positions the amendment's A3 template projects — the web *self-labels* what OCR must *infer*. Same regions, two acquisition modes, one IR. -## P1. Access preflight (blocking facts to verify, first spider-enabled session) - -`AdaWorldAPI/spider` was NOT readable from this session (repo-access gate). -Before W3 is scoped in detail, a session with spider access verifies: - -- **PF-1** fork exists, default branch, divergence from upstream - `spider-rs/spider`. -- **PF-2** page output surface (assumed from upstream: `Website` crawl → - per-page HTML string + URL + headers; a headless-chrome feature exists - for rendered layout). Verify names/features before writing the seam. -- **PF-3** whether rendered layout coordinates are obtainable (headless - feature) or whether W3 ships DOM-order pseudo-geometry first (see W3 - fallback — the plan works either way). -- **PF-4** workspace layout: where a harvest crate lands in the fork - (sibling crate, mirroring how `ruff_*_spo` crates live in the ruff fork). +## P1. Access preflight — VERIFIED against source (2026-07-13, this session) + +`AdaWorldAPI/spider` is not readable via GitHub-MCP (repo-access gate) but +clones fine over the local git proxy (same transport as MedCare-rs). Cloned +to `/home/user/spider`; all four PF facts verified against the Rust source +(the fork's own `.claude`/`CLAUDE.md` are stale upstream boilerplate and were +NOT used — the seam is grounded in code): + +- **PF-1 ✓** Fork of `spider-rs/spider`, default branch `main` @ v2.52.9, + exactly ONE AdaWorldAPI commit over upstream: `8df3a1b3` *"Add pluggable + in-process HttpFetchEngine seam (#403)"* — the fork already thinks in + swappable seams (same shape as `DocRenderer` / OGAR producer seams), + lowering the risk of wiring a doc-IR producer. Otherwise virgin. +- **PF-2 ✓** Page output surface: `spider::page::Page` exposes `get_html()` + (String) + `get_url()` + `get_bytes()` + status/headers + optional + `screenshot_bytes`. That is the W3 producer input. +- **PF-3 ✓ (upgraded from "unknown" to "feasible-later")** Per-element + geometry is NOT exposed as page output, BUT the CDP primitive is used + throughout the `chrome` feature path — `bounding_box()` on element handles + (`features/solvers.rs`) and `getBoundingClientRect()` in injected JS + (`features/webdriver.rs`, `chrome_common.rs`). So rendered-coordinate rails + are a KNOWN-FEASIBLE later increment (a CDP/JS pass over the region + landmarks reusing that in-repo primitive), not speculative. W3 v1 ships + DOM-order pseudo-geometry (below); rendered rails are the increment. +- **PF-4 ✓** Harvest crate `spider_doc_ir` lands as a workspace sibling next + to **`spider_agent_html`** — which is NOT an empty field: it already ships + `clean_html_with_profile` / `smart_clean_html` / `CleaningIntent` / + `HtmlCleaningProfile{Raw,Default,Aggressive,Slim,Minimal,Auto}` with + `from_content_analysis` — a content-relevance layer the region harvest + builds ON, not around. ## P2. Wave plan @@ -86,14 +101,36 @@ gains a dep on `ogar-doc-ir`; `ogar-doc-ir` never depends back. ### W3 — spider-fork producer (`spider_doc_ir` harvest crate, in the fork) -DOM → doc IR: HTML5 landmarks → region kinds; DOM order → reading order; -`` → cell grids; microdata/schema.org → typed fields; rendered -layout coords → rails when PF-3 confirms the headless path, else -**DOM-order pseudo-geometry** (top-to-bottom rail assignment) as the -honest v1 — provenance lane marks which geometry mode produced the rails. -Fixture-driven tests on committed HTML files; no live crawling in CI. -Lives in the spider fork (PF-4), mirroring `ruff_*_spo`-in-ruff; depends -on `ogar-doc-ir` via git dep floating on main (D-NEVER-PIN-BUMP). +**Grounded in the verified surface (PF-2/3/4).** `spider_agent_html` parses +with **`lol_html`** (Cloudflare's streaming CSS-selector rewriter — no +random-access DOM tree), and that is a *gift* for region harvesting, not a +constraint: + +- **HTML5 landmarks are CSS selectors → lol_html element handlers.** + `header`→Header, `main`/`article`→Main, `footer`→Footer, `nav`→furniture, + `table`→Table (with nested `tr`/`td`/`th` handlers → cell grids), + `figure`/`img`→Figure. The harvest is a rewriter with one handler per + landmark selector, emitting `ogar-doc-ir` region nodes as the stream + passes. +- **Reading order is free.** lol_html is forward-only streaming = document + order = the reading-order the IR needs (and the temporal stream DeepNSM + consumes). No sort, no tree walk. +- **DOM-order pseudo-geometry is free.** The streaming index → u8 rail + (top-to-bottom) IS the v1 spatial rail — lol_html gives no layout, but + document order is exactly the honest v1 the amendment specified. Rendered + rects (PF-3, the `bounding_box()` path) are the later increment; the + provenance lane marks which geometry mode produced the rails. +- **Furniture/boilerplate discrimination is half-done.** Reuse + `HtmlCleaningProfile::from_content_analysis` / `CleaningIntent` to mark + which regions are content vs chrome — the "detect page furniture" concern + (header/footer/nav) the OCR side infers, the web side already scores. +- **Typed fields:** microdata / schema.org / `` → typed-field nodes + (the DOM analogue of OCR's `harvest_profile`). + +Crate: `spider_doc_ir`, sibling to `spider_agent_html`, deps = `lol_html` +(already in the workspace) + `ogar-doc-ir` (git dep floating on main, +D-NEVER-PIN-BUMP). Fixture-driven tests on committed HTML files; no live +crawling in CI (P4). Mirrors `ruff_*_spo`-in-ruff. ### W4 — `ogar-doc` (persistence + template + renderer trait) @@ -152,3 +189,113 @@ producers exist and BEFORE W4 persists anything** — persisting a tesseract-shaped subtree and discovering the divergence later would bake the bias into stored data. Merge-order discipline per house rule: OGAR-side crates first, fork-side producers second, in lockstep. + +## P6. The `ogar-doc-ir` type surface (the concrete contract W0 verifies) + +Per the operator protocol (spec detailed enough that the council VERIFIES, +not redesigns), the W1 crate's surface in concrete form. Types only; no +minting, no canon change — the council ratifies THIS shape, then W1 codes it. +serde-only crate. **Canon-critical:** the spatial rail is `u8:u8` — **two +separate bytes, never widened to u16** (P0 GUID canon) — and IS the `X:Y` +reading of the 4+12 facet payload when a region persists as a node (W4). + +```rust +/// Perceptual-IR version marker. `from_json` refuses any other value +/// (I-LEGACY-API-FEATURE-GATED: a v2 reshape is a NEW marker, never silent). +pub const DOC_IR_VERSION: &str = "doc.v1"; + +/// Which retina produced this document. Confidence semantics differ per +/// source, so they never share a float (OCR recognition ≠ crawl trust). +#[non_exhaustive] +pub enum Provenance { Ocr, Dom } + +/// One spatial coordinate on the unit-square tile — ONE byte per axis. +/// A page/viewport is a 256×256 tile (P0 "256×256 centroid tile"); this +/// is the "2D focus of attention" address. NEVER a u16. +pub struct Rail { pub x: u8, pub y: u8 } + +/// A region's box: two rails (top-left, bottom-right) = 4 bytes = two of +/// the six `u8:u8` pairs of the 12-byte facet register. +pub struct BBoxRail { pub tl: Rail, pub br: Rail } + +/// How the rails were derived — kept in the provenance lane so a consumer +/// never mistakes document-order placement for measured geometry. +#[non_exhaustive] +pub enum Geometry { + /// Measured layout (OCR pixel bbox / DOM `getBoundingClientRect`). + Rendered, + /// Reading-order placement (lol_html stream index / OCR line order). + DomOrder, +} + +/// The CLOSED region-kind vocabulary. `from_json` HARD-FAILS on any kind +/// outside this set (mirrors `ruff_spo_triplet::from_ndjson`'s closed +/// predicate gate) — the single rule that keeps the IR source-agnostic +/// instead of drifting into a tesseract-shaped or DOM-shaped grab-bag. +#[non_exhaustive] +pub enum RegionKind { + Header, // OCR page-furniture top / HTML
+ Footer, // OCR page-furniture bottom / HTML
+ Figure, // OCR halftone region / HTML
| + Text, // a text block that is none of the above landmarks +} + +/// One harvested typed field (invoice netto/ust/IBAN, a schema.org prop…). +/// Value stays a string here; typed validation (arithmetic_ok, iban_mod97) +/// is the executor/consumer's — the IR carries facts, not policy. +pub struct TypedField { + pub key: String, + pub value: String, + pub bbox: BBoxRail, + pub confidence: u8, // per-source semantics (see Provenance) +} + +/// One cell of a Table region, addressed by (row, col) — itself a `u8:u8` +/// pair on the facet register when persisted. +pub struct TableCell { pub row: u8, pub col: u8, pub text: String, pub bbox: BBoxRail } + +/// A region node — the recursive awareness unit. +pub struct Region { + pub kind: RegionKind, + pub bbox: BBoxRail, + pub reading_order: u16, // stream position within the page + pub text: Option, // for text-bearing kinds + pub cells: Vec, // for Table + pub children: Vec, // nested regions (a Main containing Tables…) +} + +/// One page of a document. +pub struct DocPage { pub number: u16, pub width: u32, pub height: u32, pub regions: Vec } + +/// The document root — the `document 0x080B` subtree before persistence. +pub struct DocIr { + pub version: &'static str, // == DOC_IR_VERSION + pub source: Provenance, + pub geometry: Geometry, + /// Content identity: sha256 of the ORIGINAL bytes (scan or HTML) — + /// the convergence key. Same document via both retinas ⇒ same id ⇒ + /// one subtree (P-XRETINA's identity assertion). + pub content_sha256: [u8; 32], + pub mime: String, + pub pages: Vec, + pub fields: Vec, +} + +/// The load gate — closed-vocabulary + version enforcement at the boundary. +/// Fails loud on unknown region kind or wrong version (the source-agnostic +/// guard; a producer that emits an off-vocab kind is caught at the seam, +/// not silently dropped downstream). +pub fn from_json(s: &str) -> Result; +``` + +**What the council checks against this surface:** (1) every field a +*tesseract* producer needs and every field a *spider* producer needs are +both expressible with no source-specific escape hatch (source-agnostic +test); (2) `Rail`/`TableCell` stay `u8:u8`, never widened (canon); +(3) the closed `RegionKind` gate + `DOC_IR_VERSION` are the only load-time +authority; (4) `content_sha256` is the identity, not a producer-assigned id; +(5) `confidence` never conflates the two `Provenance` semantics. A numbered +factual error here is a council finding; taste is not.