Ingest: jsonb field-metadata merge never removes fields that disappeared

[jfilter]

Found by a multi-agent sweep on 2026-07-21. Reported by hunter agents but never adjudicated — the verifier pass hit a rate limit, so these are unverified, not confirmed. Check each against the code before acting.

• apps/web/lib/services/schema-builder/schema-builder.ts:395 — buildManualSchema throws a TypeError on any nested or array-of-object field, so the getSchema() error fallback itself crashes.
• apps/web/lib/services/schema-detection/utilities/geo.ts:132 — enrichTagField classifies a column as a tag field when even one value is an array, making the whole column's scalar values unfilterable.
• apps/web/lib/ingest/fetch-remote-data.ts:406 — the native-CSV excludeFields path uses a bare Papa.unparse and drops columns for ragged CSVs. The web app has an unparseRowsToCsv helper that handles this; this path does not use it.
• apps/web/lib/jobs/handlers/create-schema-version-job.ts:195 — the jsonb merge of dataset fieldMetadata/fieldTypes never removes fields that disappeared from the source schema.
• apps/web/lib/jobs/handlers/create-schema-version-job.ts:193 — datasets are written via raw drizzle, bypassing updatedAt and the versions table.

[jfilter]

Four of five findings fixed in 8129714 (merged as 899211a); this issue stays open for the fifth, which is real but needs a design decision.

Fixed:

• buildManualSchema crashed on nested and array-of-object fields. Field stats record both the container path (user) and its children (user.name); the container was visited first and produced a leaf with no properties, so descending dereferenced undefined. Two distinct TypeErrors reproduced. This killed getSchemaSync() and the quicktype-failure fallback.
• One array value flipped a whole column to a tag field. 99 scalar rows plus 1 array row set isTagField and replaced enumValues with the stray row's elements, making every scalar value unfilterable. Now gated on an array majority, reusing FIELD_TYPE_MAJORITY_THRESHOLD.
• Bare Papa.unparse derives its header from row 0, so ragged CSV lost every column absent from the first row for the whole file. Now uses unparseRowsToCsv.
• The raw drizzle UPDATE left updated_at stale despite datasets_updated_at_idx. The versions half of that finding is much weaker than stated — datasets use versions.drafts but every programmatic write is published, so the only real effect is absence from version history. Hand-writing into _datasets_v would be riskier than the gap it closes, so that was left alone.

Still open — create-schema-version-job.ts:195, jsonb merge never removes disappeared fields. The finding is accurate: || unions keys, so a dropped source column keeps a stale fieldMetadata entry and a filter matching nothing. But the merge exists precisely because a single sheet only knows its own subset of a dataset's fields. Pruning from this per-sheet job re-creates the last-writer-wins bug the merge was written to fix — sibling sheets' fields would vanish. A correct fix needs a dataset-level reconciliation pass that can see every sheet and source. The trade-off is documented in the code.