feat: producer run summary, --check, status.json, and --reconcile#18
Merged
Conversation
Two visibility features for the producer: - Each prices run now prints a per-symbol date-advance (e.g. "ES: 9946 backadj … [2026-07-13 -> 2026-07-14]") and a summary footer (OK/failed counts, rows written, elapsed, newest date). run_summary() is a pure function so it can be emailed later without touching the producer. - `cotdata-update --check`: read-only store status from manifest.json — per-domain entry/row counts, newest data date, last write, and any entries lagging behind their domain's newest (a partial-run signal). No network, no Windows/norgatedata, so it runs anywhere the store is visible. status.py holds the pure summarize/format/run_summary helpers; test_status.py covers them. Full suite: 39 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Downstream tools can trigger on fresh data by polling one small file instead of
scanning the store. Each producer run writes $COTDATA_STORE/status.json
(atomically) with:
- newest_data.<domain> — newest daily-data date per domain (prices,
cot_legacy, cot_disagg, cot_tff). Advances ONLY on genuinely new data →
the field to key "there is new data" on.
- generated_at — refreshed every run (detects "a run happened").
- last_run — kinds, ok/failed, per-symbol failures for alerting.
norgate.update now returns a structured result; the CLI aggregates it and writes
status.json after any run. build_status_doc / write_status_file are pure/tested.
Full suite: 41 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The COT tables on disk are already named {symbol}_{code} (e.g. RTY_23977A), and
get_cot reads that convention. But the manifest still carried ghosts from older
schemes — bare CFTC codes (001602, 23977A) written before the prefix convention,
plus an entire retired `cot` domain — none of which have parquet files. They made
--check / status.json look inconsistently named.
Add store.reconcile_manifest(): drop every manifest entry whose parquet file is
missing, and remove domains left empty. Provably safe — only removes bookkeeping
for files that don't exist; never renames or deletes data. Exposed as
`cotdata-update --reconcile` (prints what it pruned, refreshes status.json).
On the current store this prunes 44 cot ghosts (whole domain) + 44/24/20 bare
cot_legacy/disagg/tff ghosts, leaving the real prefixed entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A symbol's predecessor CFTC codes (hist_codes, e.g. RTY_23977A, LBR_058643) are frozen at old dates by design, so flagging them as "lagging" in --check / status.json is noise. summarize() now takes an ignore_lag set; --check and status.json build it from the registry via hist_code_names(). A symbol's CURRENT code is still flagged if it goes stale — only predecessors are exempt. Verified on the real store: RTY_23977A (2018) / LBR_058643 (2023) remain present but --check now reads "all entries current". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Producer visibility, in three parts.
1. Per-run console summary
Each
--pricesrun prints a per-symbol date advance and a footer:2.
cotdata-update --check— read-only statusPer-domain row counts, newest data date, last write, and any entries lagging behind their peers (a partial-run signal). Manifest-only — no network, no Windows/norgatedata — so it runs anywhere the store is visible.
3.
status.json— structured new-data signal for downstream toolsEvery run writes
$COTDATA_STORE/status.json(atomically, beside the data) so tools that trigger on fresh data poll one small file:{ "generated_at": "2026-07-15T10:15:24Z", "schema_version": 2, "newest_data": { "prices": "2026-07-14", "cot_legacy": "2026-07-07", "cot_disagg": "2026-07-07", "cot_tff": "2026-07-07" }, "domains": { "prices": { "newest_data": "2026-07-14", "last_write": "...", "entries": 84, "rows": 829096, "lagging": 0 } }, "last_run": { "kinds": ["prices"], "ok": ["ES","..."], "symbols_failed": [], "rows": 1658000, "seconds": 88, "at": "..." } }Polling contract:
newest_data.<domain>to your last-seen value; it advances only on genuinely new daily data. Prices and each COT report are separate domains, so price- and COT-triggered tools each watch their own key.generated_at.last_run(kinds, per-symbol failures).Notes
summarize/format_report/build_status_doc/write_status_file/run_summary) live instatus.py;test_status.pycovers them. Full suite: 41 passed.run_summaryreturns a string andstatus.jsonis structured, so it's a small follow-up.🤖 Generated with Claude Code
4.
cotdata-update --reconcile— manifest hygiene / COT naming consistencyCOT parquet files are already named
{symbol}_{code}(e.g.RTY_23977A), andget_cotreads that convention. But the manifest carried ghosts from older schemes — bare CFTC codes (001602,23977A) predating the prefix convention, plus an entire retiredcotdomain — none of which have files. They made--check/status.jsonlook inconsistently named.--reconciledrops every manifest entry whose parquet file is missing (and empties dead domains). Provably safe: only removes bookkeeping for files that don't exist. On the current store it prunes 44cot(whole domain) + 44/24/20 barecot_legacy/disagg/tffghosts, leaving only the real prefixed entries.