Skip to content

fix(prices): skip Yahoo-only markets (MME/MFS) in the Norgate producer#27

Merged
mspinola merged 2 commits into
mainfrom
claude/norgate-data-connection-05507a
Jul 20, 2026
Merged

fix(prices): skip Yahoo-only markets (MME/MFS) in the Norgate producer#27
mspinola merged 2 commits into
mainfrom
claude/norgate-data-connection-05507a

Conversation

@mspinola

Copy link
Copy Markdown
Owner

Problem

Running cotdata-update --metadata (Windows producer) errored on two symbols:

ERROR: Norgate Data: security_name: &MME_CCB not found
ERROR: Norgate Data: security_name: &MFS_CCB not found
... (every spec field, for both symbols)
Successfully wrote metadata for 49 symbols -> store

Root cause: MME and MFS are MSCI international indices Norgate doesn't carry — they're priced off ETF proxies (EEM/EFA) via the yfinance provider. But registry.py defaulted every symbol's Norgate ticker to &<internal>, so they silently became &MME/&MFS. The Norgate producer iterated all symbols and tried to fetch nonexistent &MME_CCB/&MFS_CCB. --prices failed those cleanly per-symbol, but --metadata catches each field error to None and silently wrote two all-null spec rows to contract_specs — the real harm.

Fix

Mirrors how the yfinance provider already filters by s.yahoo:

  • Mark MME/MFS norgate: null in registry.yaml (Symbol.norgate is now Optional[str]).
  • Add _norgate_covered() in the Norgate provider, applied in both update() (prices) and update_metadata(), to skip symbols with no Norgate coverage — with a one-line note instead of erroring.

Now the producer prints skipping 2 symbol(s) with no Norgate coverage (priced elsewhere): MME, MFS and processes only the 47 real Norgate markets.

Store cleanup

The two all-null rows already written to the store were pruned directly via the store API (49 → 47 rows, atomic write + manifest update). Not part of this diff — it's a data change, outside version control — but already applied, so no producer re-run is required to correct the data.

Tests

  • test_registry.py: MME/MFS norgate is None; ES still defaults to &ES.
  • test_norgate_provider.py: _norgate_covered unit filter + full-run update_metadata store round-trip proving MME is never fetched and no null row is written.
  • Full suite: 70 passed.

🤖 Generated with Claude Code

mspinola and others added 2 commits July 20, 2026 09:51
Expand CONTRIBUTING.md with detailed uv setup, including:
- uv installation steps and alternatives
- Platform-specific venv activation (Mac/Linux/Windows)
- Optional Norgate extras on Windows
- Fallback standard pip setup

Add Development section to README pointing to setup guide.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
MME and MFS are MSCI international indices Norgate doesn't carry — they are
priced off ETF proxies (EEM/EFA) via the yfinance provider. But the registry
defaulted every symbol's Norgate ticker to &<internal>, so they became
&MME/&MFS and the Norgate producer tried to fetch nonexistent &MME_CCB /
&MFS_CCB. --prices failed those per-symbol; --metadata caught each field error
to None and silently wrote two all-null spec rows to contract_specs.

Mirror how the yfinance provider filters by s.yahoo: mark these markets
`norgate: null` in the registry (norgate is now Optional[str]) and add
_norgate_covered(), applied in both update() and update_metadata(), to skip
symbols with no Norgate coverage (with a note) instead of erroring.

Tests: registry norgate=None for MME/MFS, provider skip (unit + full-run store
round-trip). Full suite passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mspinola
mspinola merged commit e9ff38d into main Jul 20, 2026
5 checks passed
@mspinola
mspinola deleted the claude/norgate-data-connection-05507a branch July 20, 2026 16:13
mspinola added a commit that referenced this pull request Jul 20, 2026
#28)

* docs: fix dangling doc refs and backfill CHANGELOG

The docs/plan_promote_reconstructed_volume.md spec was removed in #15 but was
still referenced from config.py, store.py (a user-facing RuntimeError), and
README.md — each a dead pointer. Drop the references; the surrounding text and
the README reconstructed-volume section already explain the schema-v2 behavior.

Backfill the Unreleased CHANGELOG, which had only #23: add the Yahoo provider +
MSCI held-out markets (#24), fold #26 into the propadj entry, and record the
metadata-upsert (#25) and Yahoo-only-skip (#27) fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: commit uv.lock for reproducible installs

uv.lock was neither tracked nor ignored, so it perpetually showed as untracked.
The README promotes the uv workflow (`uv pip install -e .`), so track the lock
for reproducible resolution. Verified current with `uv lock --check`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: cover dormant databento provider and COT read API

The largest provider (databento, 280 LOC) had no tests and cot.py's consumer
read path was untested. Add:

- test_databento_provider.py — smoke tests mocking the databento SDK (an optional
  extra) that lock the hard-won statistics parsing: Open Interest from stat_type 9,
  and settlement (stat_type 3) overriding Close dated by ts_ref.
- test_cot.py — get_cot report dispatch, symbol/CFTC-code lookup, and the
  predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on
  overlapping dates, sort).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(prices): fail fast on Norgate outage; drop null metadata rows

Two producer reliability fixes for the Norgate path.

1. Preflight NDU reachability. norgatedata retries each data call 10x then calls
   bare sys.exit() — which exits 0 (a scheduled --prices run looks "successful"
   while writing nothing and never triggers scheduler retry) and raises SystemExit
   past the per-symbol `except Exception`, killing the run on the first symbol.
   update()/update_metadata() now probe norgatedata.status() (a safe, non-exiting
   check) first and raise a clear RuntimeError → non-zero exit if NDU is down.

2. Guard all-null metadata rows. get_symbol_metadata catches each field error to
   None, so a transient Norgate failure on a covered symbol wrote an all-null spec
   row (and on a scoped upsert would overwrite good specs with nulls). Skip such
   rows with a warning.

Tests: preflight aborts update()/update_metadata() when status() is False; the
all-null row is skipped on a full run. Full suite 81 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant