feat(prices): add propadj (ratio) adjustment; salvage DC Class III Milk#23
Merged
Conversation
Norgate publishes continuous futures only unadjusted and ADDITIVELY back-adjusted (_CCB) — no native ratio-adjusted series. For a low-priced, ~29-year contract like DC (Class III Milk, ~$15-20/cwt), additive accumulation of roll gaps drives 46.7% of DC_backadj closes <= 0 (range -9.83..23.09), which makes price-based stops, R-multiples, and percentage returns meaningless — so CMR cannot use DC's backadj at all. Add a `propadj` view derived on read from the already-stored unadj + backadj series (no producer re-run, no schema bump, cross-platform): the offset O=B-U steps only at rolls, so each roll's calendar spread is recoverable (s = O[r-1]-O[r]) and convertible to a multiplicative ratio k = (U[r-1]+s)/U[r-1]. Scaling each segment by the cumulative product of k, anchored to the most-recent segment, yields a strictly-positive series (DC 4.68..25.01 over full 1997-2026 history) that preserves within-segment percentage returns exactly and is sign-identical to backadj on every day including rolls. Recommendation: CMR reads DC (and any similarly low-priced, long-history contract) with adjustment="propadj". Restricting DC to its positive-price era (~2011+) or dropping it were the fallbacks; neither is needed. - prices.get_prices: accept adjustment="propadj"; _ratio_adjust() derivation - tests/test_propadj.py: positivity, anchor, %-return + sign fidelity, hand-computed factors, no-Delivery-Month fallback, missing-series safety, reconstructed-volume composition - README: document propadj + the DC rationale Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Jul 20, 2026
#28) * docs: fix dangling doc refs and backfill CHANGELOG The docs/plan_promote_reconstructed_volume.md spec was removed in #15 but was still referenced from config.py, store.py (a user-facing RuntimeError), and README.md — each a dead pointer. Drop the references; the surrounding text and the README reconstructed-volume section already explain the schema-v2 behavior. Backfill the Unreleased CHANGELOG, which had only #23: add the Yahoo provider + MSCI held-out markets (#24), fold #26 into the propadj entry, and record the metadata-upsert (#25) and Yahoo-only-skip (#27) fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: commit uv.lock for reproducible installs uv.lock was neither tracked nor ignored, so it perpetually showed as untracked. The README promotes the uv workflow (`uv pip install -e .`), so track the lock for reproducible resolution. Verified current with `uv lock --check`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: cover dormant databento provider and COT read API The largest provider (databento, 280 LOC) had no tests and cot.py's consumer read path was untested. Add: - test_databento_provider.py — smoke tests mocking the databento SDK (an optional extra) that lock the hard-won statistics parsing: Open Interest from stat_type 9, and settlement (stat_type 3) overriding Close dated by ts_ref. - test_cot.py — get_cot report dispatch, symbol/CFTC-code lookup, and the predecessor-code stitching (scale numeric cols, re-stamp code, keep primary on overlapping dates, sort). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(prices): fail fast on Norgate outage; drop null metadata rows Two producer reliability fixes for the Norgate path. 1. Preflight NDU reachability. norgatedata retries each data call 10x then calls bare sys.exit() — which exits 0 (a scheduled --prices run looks "successful" while writing nothing and never triggers scheduler retry) and raises SystemExit past the per-symbol `except Exception`, killing the run on the first symbol. update()/update_metadata() now probe norgatedata.status() (a safe, non-exiting check) first and raise a clear RuntimeError → non-zero exit if NDU is down. 2. Guard all-null metadata rows. get_symbol_metadata catches each field error to None, so a transient Norgate failure on a covered symbol wrote an all-null spec row (and on a scoped upsert would overwrite good specs with nulls). Skip such rows with a warning. Tests: preflight aborts update()/update_metadata() when status() is False; the all-null row is skipped on a full run. Full suite 81 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DC (Class III Milk, internal symbol
DC/ Norgate&DC_CCB) is the flagship new-asset-class (Dairy) held-out generalization market for CMR, but its stored back-adjusted series is unusable: 46.7% ofDC_backadjcloses are ≤ 0 (range −9.83 to 23.09). Milk is a low-priced (~$15–20/cwt), seasonal, cash-settled contract; over ~29 years, Norgate's additive back-adjustment accumulates roll gaps past zero. CMR's price-based stops, R-multiples, and % returns are meaningless on a non-positive series, so DC can't be used at all.Investigation
_CCB). No native ratio series, no alternate suffix.Band unadjustedUdiffer by an offsetO = B − Uthat steps only at rolls (verified: every step >$0.0001 lands on a Delivery-Month change; the rest is sub-cent float dust). Each roll's calendar spread is recoverable and convertible to a multiplicative ratio.Change
Adds a
propadj(proportional / ratio) view derived on read from the already-storedunadj+backadjseries — a pure function of two stored series, so no producer re-run, no schema bump, cross-platform (matches the existing reconstructed-volume derivation pattern). Available today viaget_prices("DC", adjustment="propadj").Method: the offset
O = B − Uis piecewise-constant, stepping only at rolls. At rollrthe recovered spreads = O[r−1] − O[r]gives a roll ratiok = (U[r−1] + s)/U[r−1] = F_new/F_old. Each historical segment is scaled by the cumulative product ofkfor rolls at/after it, anchoring the most-recent segment to actual prices. O/H/L/C are scaled by the per-row segment factor; Volume / Open Interest / Delivery Month pass through.Validation
unadjexactlybackadj100% including on roll days (corr 0.93 — residual is the intended additive→proportional rescaling)Recommendation
CMR reads DC (and any similarly low-priced, long-history contract) with
adjustment="propadj". DC survives as the held-out Dairy market with clean positive prices. Independent of the other 4 held-out markets (ZO, KE, EMD, NKD), which proceed onbackadjunchanged.Contents
prices.get_prices: acceptadjustment="propadj"; new_ratio_adjust()derivationtests/test_propadj.py: 10 tests (positivity, anchor, %-return + sign fidelity, hand-computed factors, no-Delivery-Month fallback, missing-series safety, reconstructed-volume composition)README: documentspropadj+ the DC rationaleNotes
test_norgate_provider.pyfailures (need Windows-onlynorgatedata) are unrelated — they fail identically onmain._check_roll_gaps"looks UNADJUSTED" heuristic untouched (its DC false-positive is a separate concern, out of scope here).🤖 Generated with Claude Code