fix: upsert scoped Norgate metadata instead of replacing the whole table#25
Merged
Conversation
A scoped `cotdata-update --metadata --symbols DC ZO KE EMD NKD` run REPLACED the entire metadata/contract_specs.parquet with only those rows, dropping the other 42 markets' contract specs (Norgate_Symbol, tick/point value, margin, …) and breaking position-sizing / cost lookups across the deploy book. Contract specs live in ONE Symbol-keyed table (unlike prices/COT, which are one parquet per symbol), so a subset write must upsert, not overwrite. - store.upsert_metadata: read existing → replace/add requested Symbols → keep the rest → write the union (manifest reflects the union count). - norgate.update_metadata: scoped run (`symbols` given) upserts by Symbol; no-symbols run still regenerates the full registry and replaces the table. - Tests: scoped update preserves unlisted markets; full update replaces; upsert on empty store behaves like a plain write. Other metadata writers (prices, COT) are per-symbol files, so they have no replace-vs-merge exposure — only contract_specs shares one table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Jul 20, 2026
#28) * docs: fix dangling doc refs and backfill CHANGELOG The docs/plan_promote_reconstructed_volume.md spec was removed in #15 but was still referenced from config.py, store.py (a user-facing RuntimeError), and README.md — each a dead pointer. Drop the references; the surrounding text and the README reconstructed-volume section already explain the schema-v2 behavior. Backfill the Unreleased CHANGELOG, which had only #23: add the Yahoo provider + MSCI held-out markets (#24), fold #26 into the propadj entry, and record the metadata-upsert (#25) and Yahoo-only-skip (#27) fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: commit uv.lock for reproducible installs uv.lock was neither tracked nor ignored, so it perpetually showed as untracked. The README promotes the uv workflow (`uv pip install -e .`), so track the lock for reproducible resolution. Verified current with `uv lock --check`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: cover dormant databento provider and COT read API The largest provider (databento, 280 LOC) had no tests and cot.py's consumer read path was untested. Add: - test_databento_provider.py — smoke tests mocking the databento SDK (an optional extra) that lock the hard-won statistics parsing: Open Interest from stat_type 9, and settlement (stat_type 3) overriding Close dated by ts_ref. - test_cot.py — get_cot report dispatch, symbol/CFTC-code lookup, and the predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on overlapping dates, sort). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(prices): fail fast on Norgate outage; drop null metadata rows Two producer reliability fixes for the Norgate path. 1. Preflight NDU reachability. norgatedata retries each data call 10x then calls bare sys.exit() — which exits 0 (a scheduled --prices run looks "successful" while writing nothing and never triggers scheduler retry) and raises SystemExit past the per-symbol `except Exception`, killing the run on the first symbol. update()/update_metadata() now probe norgatedata.status() (a safe, non-exiting check) first and raise a clear RuntimeError → non-zero exit if NDU is down. 2. Guard all-null metadata rows. get_symbol_metadata catches each field error to None, so a transient Norgate failure on a covered symbol wrote an all-null spec row (and on a scoped upsert would overwrite good specs with nulls). Skip such rows with a warning. Tests: preflight aborts update()/update_metadata() when status() is False; the all-null row is skipped on a full run. Full suite 81 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes a data-loss bug:
cotdata-update --metadata --symbols …replaced the wholecontract_specstable with just the requested rows, dropping every other market's specs (hit in practice — a 5-symbol run wiped the other 42).Adds
store.upsert_metadata(keep rows whoseSymbolisn't in the incoming frame, add/replace the rest).update_metadataroutes a scoped run through it; a full run (no--symbols) still replaces the table (registry regeneration).Tests
test_upsert_metadata_preserves_unlisted_symbolsreproduces the exact 5-vs-42 regression; plus empty-store fallback and manifest-reflects-union. 21 passed.🤖 Generated with Claude Code