providers: yfinance price source + MSCI EM/EAFE held-out markets#24
Merged
Conversation
A third price provider (after Norgate/databento) for markets those don't cover. Registry Symbol gains an optional `yahoo` ticker; `providers/yfinance.py` fetches Yahoo OHLCV and writes the store's standard Open/High/Low/Close/Volume + Date-index shape (ETF/spot proxies have no roll, so backadj == unadj). CLI: --prices-yahoo. Research-grade (Yahoo is free/unofficial — gaps/revisions/API breakage possible). Adds MSCI Emerging Markets (MME → EEM) and MSCI EAFE (MFS → EFA): the liquid, out-of-selection equity indices needed for a clean generalization test — COT is the real ICE futures positioning, price is the ETF proxy (no roll gaps). ~15y COT + deep ETF history each. Hermetic provider tests (mock download + write_prices); golden registry tests bumped 47→49 / Equities 6→8 / checksum. Full suite 58 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The provider tests imported the real yfinance, which isn't installed in CI (it's an optional dep). Inject a fake yfinance module into sys.modules instead, so the tests run without the package. Declare yfinance as the [yahoo] optional extra (matching norgate/databento). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Jul 20, 2026
#28) * docs: fix dangling doc refs and backfill CHANGELOG The docs/plan_promote_reconstructed_volume.md spec was removed in #15 but was still referenced from config.py, store.py (a user-facing RuntimeError), and README.md — each a dead pointer. Drop the references; the surrounding text and the README reconstructed-volume section already explain the schema-v2 behavior. Backfill the Unreleased CHANGELOG, which had only #23: add the Yahoo provider + MSCI held-out markets (#24), fold #26 into the propadj entry, and record the metadata-upsert (#25) and Yahoo-only-skip (#27) fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: commit uv.lock for reproducible installs uv.lock was neither tracked nor ignored, so it perpetually showed as untracked. The README promotes the uv workflow (`uv pip install -e .`), so track the lock for reproducible resolution. Verified current with `uv lock --check`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: cover dormant databento provider and COT read API The largest provider (databento, 280 LOC) had no tests and cot.py's consumer read path was untested. Add: - test_databento_provider.py — smoke tests mocking the databento SDK (an optional extra) that lock the hard-won statistics parsing: Open Interest from stat_type 9, and settlement (stat_type 3) overriding Close dated by ts_ref. - test_cot.py — get_cot report dispatch, symbol/CFTC-code lookup, and the predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on overlapping dates, sort). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(prices): fail fast on Norgate outage; drop null metadata rows Two producer reliability fixes for the Norgate path. 1. Preflight NDU reachability. norgatedata retries each data call 10x then calls bare sys.exit() — which exits 0 (a scheduled --prices run looks "successful" while writing nothing and never triggers scheduler retry) and raises SystemExit past the per-symbol `except Exception`, killing the run on the first symbol. update()/update_metadata() now probe norgatedata.status() (a safe, non-exiting check) first and raise a clear RuntimeError → non-zero exit if NDU is down. 2. Guard all-null metadata rows. get_symbol_metadata catches each field error to None, so a transient Norgate failure on a covered symbol wrote an all-null spec row (and on a scoped upsert would overwrite good specs with nulls). Skip such rows with a warning. Tests: preflight aborts update()/update_metadata() when status() is False; the all-null row is skipped on a full run. Full suite 81 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A third cotdata price provider (after Norgate/databento) — Yahoo Finance — for markets those don't cover, plus the MSCI EM / EAFE held-out markets it unlocks.
Symbolgains an optionalyahooticker;providers/yfinance.pyfetches Yahoo OHLCV and writes the store's standardOpen/High/Low/Close/Volume+Date-index shape. ETF/spot proxies have no roll, sobackadj == unadj. CLI:--prices-yahoo.Why
These are the only liquid, out-of-selection, CFTC-reported markets available, and Norgate/databento don't cover them. They flipped the generalization read from "confounded no-edge" (illiquid markets) to +0.66R / PF 1.52, both markets positive — suggestive that the edge travels, just underpowered at n=36.
Caveat (by design)
Research-grade. Yahoo is free/unofficial — gaps, silent revisions, and the scraper breaks when Yahoo changes their site. Not a production replacement for Norgate. ETF-as-index-proxy is an approximation (COT on futures, price on the tracking ETF).
Tests
Hermetic provider tests (mock
download+write_prices— no network, no store writes): normalization of Yahoo's MultiIndex frame, both adjustments written, skip-without-ticker, empty→failure. Golden registry tests bumped 47→49 / Equities 6→8 / checksum. 58 passed.🤖 Generated with Claude Code