Docs hygiene, uv.lock, test coverage, and Norgate producer reliability#28
Merged
Conversation
The docs/plan_promote_reconstructed_volume.md spec was removed in #15 but was still referenced from config.py, store.py (a user-facing RuntimeError), and README.md — each a dead pointer. Drop the references; the surrounding text and the README reconstructed-volume section already explain the schema-v2 behavior. Backfill the Unreleased CHANGELOG, which had only #23: add the Yahoo provider + MSCI held-out markets (#24), fold #26 into the propadj entry, and record the metadata-upsert (#25) and Yahoo-only-skip (#27) fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
uv.lock was neither tracked nor ignored, so it perpetually showed as untracked. The README promotes the uv workflow (`uv pip install -e .`), so track the lock for reproducible resolution. Verified current with `uv lock --check`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The largest provider (databento, 280 LOC) had no tests and cot.py's consumer read path was untested. Add: - test_databento_provider.py — smoke tests mocking the databento SDK (an optional extra) that lock the hard-won statistics parsing: Open Interest from stat_type 9, and settlement (stat_type 3) overriding Close dated by ts_ref. - test_cot.py — get_cot report dispatch, symbol/CFTC-code lookup, and the predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on overlapping dates, sort). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two producer reliability fixes for the Norgate path. 1. Preflight NDU reachability. norgatedata retries each data call 10x then calls bare sys.exit() — which exits 0 (a scheduled --prices run looks "successful" while writing nothing and never triggers scheduler retry) and raises SystemExit past the per-symbol `except Exception`, killing the run on the first symbol. update()/update_metadata() now probe norgatedata.status() (a safe, non-exiting check) first and raise a clear RuntimeError → non-zero exit if NDU is down. 2. Guard all-null metadata rows. get_symbol_metadata catches each field error to None, so a transient Norgate failure on a covered symbol wrote an all-null spec row (and on a scoped upsert would overwrite good specs with nulls). Skip such rows with a warning. Tests: preflight aborts update()/update_metadata() when status() is False; the all-null row is skipped on a full run. Full suite 81 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch of maintenance improvements surfaced during a repo review. Four independent commits.
📝 docs: fix dangling doc refs + backfill CHANGELOG
docs/plan_promote_reconstructed_volume.mdwas removed in chore: remove completed todo and spec docs #15 but still referenced fromconfig.py,store.py(a user-facingRuntimeError), andREADME.md— all dead pointers. Removed.[Unreleased]CHANGELOG, which only had feat(prices): add propadj (ratio) adjustment; salvage DC Class III Milk #23: added the Yahoo provider + MSCI markets (providers: yfinance price source + MSCI EM/EAFE held-out markets #24), folded in feat(prices): propadj (ratio) adjustment — salvage low-priced contracts (DC) #26, and recorded the metadata-upsert (fix: upsert scoped Norgate metadata instead of replacing the whole table #25) and Yahoo-only-skip (fix(prices): skip Yahoo-only markets (MME/MFS) in the Norgate producer #27) fixes.📦 chore: commit
uv.lockWas neither tracked nor ignored (perpetual untracked noise). README promotes the uv workflow, so track it for reproducible installs. Verified with
uv lock --check.✅ test: cover the dormant databento provider and COT read API
test_databento_provider.py— the largest provider (280 LOC) had zero tests. Smoke tests mock the SDK (optional extra) and lock the hard-won statistics logic: Open Interest (stat_type 9) and settlement (stat_type 3) overriding Close dated byts_ref.test_cot.py—get_cotreport dispatch, symbol/CFTC-code lookup, and predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on overlaps, sort).🛠 fix(prices): Norgate producer reliability
sys.exit()— which exits 0 (a scheduled--pricesrun looks "successful" while writing nothing and never retries) and raisesSystemExitpast the per-symbol handler (kills the run on symbol Norgate integration: cotdata data layer (Norgate prices, CFTC ETL, Databento dormant, code-stitching) #1). The producer now probesnorgatedata.status()(a safe, non-exiting check) first and raises a clearRuntimeError→ non-zero exit if NDU is down.get_symbol_metadatacatches each field error toNone, so a transient Norgate failure on a covered symbol wrote an all-null row (and on a scoped upsert would overwrite good specs with nulls). Such rows are now skipped with a warning.Tests
Full suite: 81 passed (was 70; +11). No behavior change to existing green paths — existing Norgate tests just stub the new
status()preflight.🤖 Generated with Claude Code