Skip to content

feat: producer run summary, --check, status.json, and --reconcile#18

Merged
mspinola merged 4 commits into
mainfrom
feat/producer-status-and-run-summary
Jul 15, 2026
Merged

feat: producer run summary, --check, status.json, and --reconcile#18
mspinola merged 4 commits into
mainfrom
feat/producer-status-and-run-summary

Conversation

@mspinola

@mspinola mspinola commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Producer visibility, in three parts.

1. Per-run console summary

Each --prices run prints a per-symbol date advance and a footer:

ES: 9946 backadj, 9946 unadj  [2026-07-13 -> 2026-07-14]
...
prices update: 41/42 OK · 1 failed · 1,658,000 rows · 88s · newest 2026-07-14
  ✗ GC: <error>

2. cotdata-update --check — read-only status

Per-domain row counts, newest data date, last write, and any entries lagging behind their peers (a partial-run signal). Manifest-only — no network, no Windows/norgatedata — so it runs anywhere the store is visible.

3. status.json — structured new-data signal for downstream tools

Every run writes $COTDATA_STORE/status.json (atomically, beside the data) so tools that trigger on fresh data poll one small file:

{
  "generated_at": "2026-07-15T10:15:24Z",
  "schema_version": 2,
  "newest_data": { "prices": "2026-07-14", "cot_legacy": "2026-07-07", "cot_disagg": "2026-07-07", "cot_tff": "2026-07-07" },
  "domains":     { "prices": { "newest_data": "2026-07-14", "last_write": "...", "entries": 84, "rows": 829096, "lagging": 0 } },
  "last_run":    { "kinds": ["prices"], "ok": ["ES","..."], "symbols_failed": [], "rows": 1658000, "seconds": 88, "at": "..." }
}

Polling contract:

  • New data → compare newest_data.<domain> to your last-seen value; it advances only on genuinely new daily data. Prices and each COT report are separate domains, so price- and COT-triggered tools each watch their own key.
  • A run happenedgenerated_at.
  • Alertinglast_run (kinds, per-symbol failures).

Notes

  • Pure helpers (summarize / format_report / build_status_doc / write_status_file / run_summary) live in status.py; test_status.py covers them. Full suite: 41 passed.
  • Email delivery intentionally not built (needs SMTP config/secrets); run_summary returns a string and status.json is structured, so it's a small follow-up.

🤖 Generated with Claude Code

4. cotdata-update --reconcile — manifest hygiene / COT naming consistency

COT parquet files are already named {symbol}_{code} (e.g. RTY_23977A), and get_cot reads that convention. But the manifest carried ghosts from older schemes — bare CFTC codes (001602, 23977A) predating the prefix convention, plus an entire retired cot domain — none of which have files. They made --check/status.json look inconsistently named.

--reconcile drops every manifest entry whose parquet file is missing (and empties dead domains). Provably safe: only removes bookkeeping for files that don't exist. On the current store it prunes 44 cot (whole domain) + 44/24/20 bare cot_legacy/disagg/tff ghosts, leaving only the real prefixed entries.

mspinola and others added 2 commits July 15, 2026 06:23
Two visibility features for the producer:

- Each prices run now prints a per-symbol date-advance (e.g.
  "ES: 9946 backadj … [2026-07-13 -> 2026-07-14]") and a summary footer
  (OK/failed counts, rows written, elapsed, newest date). run_summary() is a
  pure function so it can be emailed later without touching the producer.
- `cotdata-update --check`: read-only store status from manifest.json —
  per-domain entry/row counts, newest data date, last write, and any entries
  lagging behind their domain's newest (a partial-run signal). No network, no
  Windows/norgatedata, so it runs anywhere the store is visible.

status.py holds the pure summarize/format/run_summary helpers; test_status.py
covers them. Full suite: 39 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Downstream tools can trigger on fresh data by polling one small file instead of
scanning the store. Each producer run writes $COTDATA_STORE/status.json
(atomically) with:

  - newest_data.<domain>  — newest daily-data date per domain (prices,
    cot_legacy, cot_disagg, cot_tff). Advances ONLY on genuinely new data →
    the field to key "there is new data" on.
  - generated_at          — refreshed every run (detects "a run happened").
  - last_run              — kinds, ok/failed, per-symbol failures for alerting.

norgate.update now returns a structured result; the CLI aggregates it and writes
status.json after any run. build_status_doc / write_status_file are pure/tested.
Full suite: 41 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mspinola mspinola changed the title feat: producer run summary + cotdata-update --check store status feat: producer run summary, --check status, and status.json new-data signal Jul 15, 2026
The COT tables on disk are already named {symbol}_{code} (e.g. RTY_23977A), and
get_cot reads that convention. But the manifest still carried ghosts from older
schemes — bare CFTC codes (001602, 23977A) written before the prefix convention,
plus an entire retired `cot` domain — none of which have parquet files. They made
--check / status.json look inconsistently named.

Add store.reconcile_manifest(): drop every manifest entry whose parquet file is
missing, and remove domains left empty. Provably safe — only removes bookkeeping
for files that don't exist; never renames or deletes data. Exposed as
`cotdata-update --reconcile` (prints what it pruned, refreshes status.json).

On the current store this prunes 44 cot ghosts (whole domain) + 44/24/20 bare
cot_legacy/disagg/tff ghosts, leaving the real prefixed entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mspinola mspinola changed the title feat: producer run summary, --check status, and status.json new-data signal feat: producer run summary, --check, status.json, and --reconcile Jul 15, 2026
A symbol's predecessor CFTC codes (hist_codes, e.g. RTY_23977A, LBR_058643)
are frozen at old dates by design, so flagging them as "lagging" in --check /
status.json is noise. summarize() now takes an ignore_lag set; --check and
status.json build it from the registry via hist_code_names(). A symbol's
CURRENT code is still flagged if it goes stale — only predecessors are exempt.

Verified on the real store: RTY_23977A (2018) / LBR_058643 (2023) remain present
but --check now reads "all entries current".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mspinola
mspinola merged commit 645c80a into main Jul 15, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant