Skip to content

fix(data_contract): enrich in bounded units so wide schemas and wide tables don't truncate#8

Merged
mischa76 merged 1 commit into
mainfrom
fix-data-contract-truncation
Jul 14, 2026
Merged

fix(data_contract): enrich in bounded units so wide schemas and wide tables don't truncate#8
mischa76 merged 1 commit into
mainfrom
fix-data-contract-truncation

Conversation

@mischa76

Copy link
Copy Markdown
Owner

Summary

Fixes the pre-existing data_contract truncation blocker flagged in the WP9 milestone: the enricher drafted all assets in one ForcedToolCaller call capped at max_tokens, so a wide schema's combined enrichment exceeded the cap and WP3's max_tokens guard raised LLMCallError — the full messy_insurance run could never complete end-to-end.

DataContractAgent.run now enriches in bounded units, scaling in both dimensions:

  • one asset per call → a wide schema (many tables) never overflows;
  • a table wider than _FIELDS_PER_CALL (40) is split by field into ceil(cols/40) calls → a wide table (many columns — routine in legacy insurance/banking sources, e.g. 256 attributes) never overflows either. _merge_enrichment folds an asset's chunks back together (merging fields, keeping the first doc).

The system prompt (carrying the full declared schema) is byte-identical across calls, so WP3 prompt caching makes the extra calls cheap on input tokens; the per-call output budget is raised to 8192 (output tokens are billed per generation, so headroom is free).

Verification

Live (real Anthropic API):

  • 5-table / 38-column messy_insurance — 5/5 contracts, all 38 fields typed, 0 undetermined-type flags.
  • A real 256-column table7 bounded calls (ceil(256/40)), all 256 fields contracted, no truncation.

Both cases previously raised LLMCallError.

Keyless (CI, no API key):

  • test_enrichment_is_batched_one_asset_per_call — one bounded call per asset.
  • test_wide_table_is_chunked_and_fully_enriched — 256 cols → chunked, every field covered.

331 tests green, ruff check . clean, uv run mypy clean (32 files).

Notes

  • A 256-column table is now 7 sequential calls (correct and bounded). If very wide tables ever matter for latency, a future optimization could parallelize the chunks at some prompt-cache-warmup cost — not needed now.
  • The data_contract agent was the specific blocker to a full messy_insurance run; this closes it. A full end-to-end vault-agent run on messy (all LLM agents, token-heavy) was not run here.

🤖 Generated with Claude Code

…tables don't truncate

The enricher drafted ALL assets in a single ForcedToolCaller call capped at max_tokens,
so a wide schema's combined enrichment exceeded the cap and WP3's max_tokens guard raised
LLMCallError — the full messy_insurance `run` could never complete end-to-end.

DataContractAgent.run now enriches in BOUNDED units, scaling in both dimensions:

  * one asset per call — a wide *schema* (many tables) never overflows; and
  * a table wider than _FIELDS_PER_CALL (40) is split by field into ceil(cols/40) calls —
    a wide *table* (many columns, routine in legacy insurance/banking sources, e.g. 256
    attributes) never overflows either. _merge_enrichment folds an asset's chunks back
    together (merging fields, keeping the first doc).

The system prompt (carrying the full declared schema) is byte-identical across calls, so
WP3 prompt caching makes the extra calls cheap on input tokens; the per-call output budget
is raised to 8192 (output tokens billed per generation, so headroom is free).

Verified live: (a) 5-table / 38-column messy_insurance — 5/5 contracts, all 38 fields
typed, 0 undetermined-type flags; (b) a real 256-column table — 7 bounded calls
(ceil(256/40)), all 256 fields contracted, no truncation. Both previously LLMCallError.

Keyless tests: test_enrichment_is_batched_one_asset_per_call (one call per asset),
test_wide_table_is_chunked_and_fully_enriched (256 cols → chunked, every field covered).
331 tests green, ruff clean, mypy strict clean (32 files).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mischa76 mischa76 merged commit 2296cf8 into main Jul 14, 2026
4 checks passed
@mischa76 mischa76 deleted the fix-data-contract-truncation branch July 14, 2026 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant