fix(prices): skip Yahoo-only markets (MME/MFS) in the Norgate producer#27
Merged
Conversation
Expand CONTRIBUTING.md with detailed uv setup, including: - uv installation steps and alternatives - Platform-specific venv activation (Mac/Linux/Windows) - Optional Norgate extras on Windows - Fallback standard pip setup Add Development section to README pointing to setup guide. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
MME and MFS are MSCI international indices Norgate doesn't carry — they are priced off ETF proxies (EEM/EFA) via the yfinance provider. But the registry defaulted every symbol's Norgate ticker to &<internal>, so they became &MME/&MFS and the Norgate producer tried to fetch nonexistent &MME_CCB / &MFS_CCB. --prices failed those per-symbol; --metadata caught each field error to None and silently wrote two all-null spec rows to contract_specs. Mirror how the yfinance provider filters by s.yahoo: mark these markets `norgate: null` in the registry (norgate is now Optional[str]) and add _norgate_covered(), applied in both update() and update_metadata(), to skip symbols with no Norgate coverage (with a note) instead of erroring. Tests: registry norgate=None for MME/MFS, provider skip (unit + full-run store round-trip). Full suite passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Jul 20, 2026
#28) * docs: fix dangling doc refs and backfill CHANGELOG The docs/plan_promote_reconstructed_volume.md spec was removed in #15 but was still referenced from config.py, store.py (a user-facing RuntimeError), and README.md — each a dead pointer. Drop the references; the surrounding text and the README reconstructed-volume section already explain the schema-v2 behavior. Backfill the Unreleased CHANGELOG, which had only #23: add the Yahoo provider + MSCI held-out markets (#24), fold #26 into the propadj entry, and record the metadata-upsert (#25) and Yahoo-only-skip (#27) fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: commit uv.lock for reproducible installs uv.lock was neither tracked nor ignored, so it perpetually showed as untracked. The README promotes the uv workflow (`uv pip install -e .`), so track the lock for reproducible resolution. Verified current with `uv lock --check`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: cover dormant databento provider and COT read API The largest provider (databento, 280 LOC) had no tests and cot.py's consumer read path was untested. Add: - test_databento_provider.py — smoke tests mocking the databento SDK (an optional extra) that lock the hard-won statistics parsing: Open Interest from stat_type 9, and settlement (stat_type 3) overriding Close dated by ts_ref. - test_cot.py — get_cot report dispatch, symbol/CFTC-code lookup, and the predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on overlapping dates, sort). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(prices): fail fast on Norgate outage; drop null metadata rows Two producer reliability fixes for the Norgate path. 1. Preflight NDU reachability. norgatedata retries each data call 10x then calls bare sys.exit() — which exits 0 (a scheduled --prices run looks "successful" while writing nothing and never triggers scheduler retry) and raises SystemExit past the per-symbol `except Exception`, killing the run on the first symbol. update()/update_metadata() now probe norgatedata.status() (a safe, non-exiting check) first and raise a clear RuntimeError → non-zero exit if NDU is down. 2. Guard all-null metadata rows. get_symbol_metadata catches each field error to None, so a transient Norgate failure on a covered symbol wrote an all-null spec row (and on a scoped upsert would overwrite good specs with nulls). Skip such rows with a warning. Tests: preflight aborts update()/update_metadata() when status() is False; the all-null row is skipped on a full run. Full suite 81 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running
cotdata-update --metadata(Windows producer) errored on two symbols:Root cause:
MMEandMFSare MSCI international indices Norgate doesn't carry — they're priced off ETF proxies (EEM/EFA) via the yfinance provider. Butregistry.pydefaulted every symbol's Norgate ticker to&<internal>, so they silently became&MME/&MFS. The Norgate producer iterated all symbols and tried to fetch nonexistent&MME_CCB/&MFS_CCB.--pricesfailed those cleanly per-symbol, but--metadatacatches each field error toNoneand silently wrote two all-null spec rows tocontract_specs— the real harm.Fix
Mirrors how the yfinance provider already filters by
s.yahoo:norgate: nullinregistry.yaml(Symbol.norgateis nowOptional[str])._norgate_covered()in the Norgate provider, applied in bothupdate()(prices) andupdate_metadata(), to skip symbols with no Norgate coverage — with a one-line note instead of erroring.Now the producer prints
skipping 2 symbol(s) with no Norgate coverage (priced elsewhere): MME, MFSand processes only the 47 real Norgate markets.Store cleanup
The two all-null rows already written to the store were pruned directly via the store API (49 → 47 rows, atomic write + manifest update). Not part of this diff — it's a data change, outside version control — but already applied, so no producer re-run is required to correct the data.
Tests
test_registry.py: MME/MFSnorgate is None; ES still defaults to&ES.test_norgate_provider.py:_norgate_coveredunit filter + full-runupdate_metadatastore round-trip proving MME is never fetched and no null row is written.🤖 Generated with Claude Code