fix: reconstruct volume from the two highest-volume contracts, not nearest expiry#16
Merged
Merged
Conversation
…arest expiry _reconstruct_volume identified First/Second as the two earliest-expiry contracts with data, then summed them as "true market volume." For products with serial months around a bi-monthly liquid cycle (GC, SI, ...), the nearest serial carries almost no volume, so the pick summed near-empty contracts and badly understated volume — GC's Volume_Reconstructed was ~6% of the front-month continuous Volume (proof: FirstVolume 558 < SecondVolume 141,592 on the same day, impossible if ranked by volume). Rank contracts by descending volume per day instead; ties break by nearest expiry (columns pre-sorted by expiry, stable sort), NaN sorts last. First/Second now mean the two dominant contracts, matching the README's stated semantics. Surfaced by the cot-analyzer flow-engine regression: switching that engine to reconstructed volume moved GC's weekly commercial-flow estimate ~79% toward the understated number. ES/6E/BTC (reconstructed ~= front) were unaffected. Adds test_reconstruction_picks_by_volume_not_expiry (nearest-expiry contract near-empty, later contract dominant → dominant must be First). NOTE: the store must be re-migrated by a full producer pass (Windows/norgatedata) before the on-disk data reflects this fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_reconstruct_volumeidentifiedFirst/Secondas the two earliest-expiry contracts with data on each day, then summed them as "true market volume." For products with serial months around a bi-monthly liquid cycle (GC, SI, …), the nearest serial month carries almost no volume, so the pick summed near-empty contracts and badly understated true volume.Concretely, in the current store:
FirstVolume(558) <SecondVolume(141,592) on the same day — impossible if the selection ranked by volume. GC'sVolume_Reconstructedcame out at ~6% of the front-month continuousVolume(historical median).How it surfaced
The cot-analyzer flow-engine regression (Step 3.5) for the reconstructed-volume promotion. Switching that engine to reconstructed volume moved GC's weekly commercial-flow estimate by ~79% — toward the understated number. Symbols where reconstructed ≈ front were unaffected:
Fix
Rank contracts by descending volume per day; ties break by nearest expiry (columns are pre-sorted by expiry and the sort is stable); NaN sorts last.
First/Secondnow mean the two dominant contracts — matching the README's stated semantics.Adds
test_reconstruction_picks_by_volume_not_expiry(nearest-expiry contract near-empty, later contract dominant → dominant must beFirst), which fails on the old logic and passes on the new. Full suite: 32 passed.The on-disk store still holds the wrong GC/CL reconstruction until a full producer pass re-runs (
norgate.update(), Windows +norgatedata). Until then cot-analyzer still consumes the degraded values for those symbols. Sequence: merge → re-run producer → re-run the regression (expect GC rec/front → ≥1.0, flow Δ collapses).Suggested post-migration gate: flag any symbol whose median rec/front ratio is outside ~[0.5, 3.0].
🤖 Generated with Claude Code