feat: exit codes for scheduler retry + README scheduling/diagram updates#19
Merged
Conversation
…es updates Also brings the README open-source overhaul forward — the #18 merge predated that commit, so main still had the old README (old intro + "COT analyzer" diagram). This restores the overhaul and layers the requested updates on top. Code: - The three CFTC providers' update() now return {"kind","ok","wrote"}; ok=False only when the *current* year's download hard-fails (source unreachable) — not when there's simply no new data yet. - cotdata-update exits non-zero if any run hard-fails (prices symbols_failed or a COT ok=False), and records failed domains in status.json last_run.failed, so a scheduler can retry real outages. Tests in test_cli_exit.py. Docs: - Producer examples: add `cotdata-update --prices` (all registry symbols). - Regenerate the architecture diagram — provably aligned (box lines equal width), fully generic consumer labels (no private package names). - New "Scheduling on Windows (Task Scheduler)" section: nightly prices, a Friday 3:25-4:00pm ET COT polling window to catch the ~3:30 release fast, a daily morning catch-up for holiday slips, and restart-on-failure via the new exit codes. Idempotency makes the over-polling harmless. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ning Per Norgate's update schedule, Final futures prices land ~8:40pm ET (Futures) and ~8:55pm ET (Futures Continuous). cotdata reads both databases (continuous series + individual contracts for volume reconstruction), so the nightly price task must run after both Finals. Move the example from 18:30 to 21:15 and add a timing note; earlier runs would capture non-final interim prices. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nals Norgate publishes Final futures prices ~8:40pm ET (Futures) / ~8:55pm ET (Continuous Futures), but the local Data Updater pulls them on its own poll — so a fixed-time schedule risks capturing interim (non-final) bars. norgate.finals_ready() checks norgatedata.last_database_update_time() for the 'Futures' and 'Continuous Futures' databases (exact names from norgatedata.databases()) against a local cutoff. `cotdata-update --prices --require-final [--final-cutoff HH:MM]` fetches only when both are refreshed at/after the cutoff; otherwise it defers with a non-zero exit so Task Scheduler's restart-on-failure waits out the gap and fires the moment NDU has the Finals — event-driven instead of a fixed-time guess. Pure _finals_ready() is unit-tested; CLI defer/run paths covered in test_cli_exit.py. README scheduling section updated to use it (prices task at 20:55 + restart-every-10-min). Full suite: 50 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…PowerShell /ET and /DU are MINUTE/HOURLY-only in schtasks, so the weekly Friday repeating window can't be created with schtasks. Replace with a PowerShell Register-ScheduledTask snippet (and a GUI equivalent); keep schtasks for the two plain daily tasks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
norgatedata.last_database_update_time() returns tz-AWARE local datetimes (e.g. ...-04:00); comparing them against a naive cutoff raised TypeError. Normalize to naive local (_to_naive_local) before comparison. Regression test with tz-aware inputs added. Caught by running the live check on the Windows producer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add PowerShell New-ScheduledTaskSettingsSet snippet for restart-on-failure (schtasks can't set it), alongside the GUI equivalent. - Friday COT window polls every 2 minutes (was 5) to catch the ~3:30 release faster. - Explicitly note the jobs can be viewed/managed in the Task Scheduler GUI (Win+R -> taskschd.msc, or Start menu -> Task Scheduler). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace literal-looking C:\path\... placeholders with <STORE>/<VENV>/<DIR> and a prominent 'replace these in both the .cmd files AND the task commands' callout (easy to paste C:\path verbatim and miss). - Show run-prices.cmd and run-cot.cmd as two separate files with a note that run-cot.cmd uses a DIFFERENT command (--cot-all), not the prices command — they were easy to conflate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the requested producer/README updates, plus exit-code support so Windows Task Scheduler can retry.
Docs
cotdata-update --prices(all registry symbols) added to the producer examples, alongside the--symbolssubset form.your signal research/your backtest / dashboards)..cmd,schtaskscommands, timezone note, and a note that idempotency makes the over-polling harmless.Code — exit codes for retry
The three CFTC providers'
update()now return{"kind","ok","wrote"}.okisFalseonly when the current year's download hard-fails (source unreachable) — not when there's simply no new data yet.cotdata-updateexits non-zero if any run hard-fails (pricessymbols_failed, or a COTok=False) and recordslast_run.failedinstatus.json.This gives the scheduler a clean signal: retry genuine outages, ignore ordinary "nothing new" runs (the schedule handles release timing).
Tests:
test_cli_exit.py(non-zero on COT failure / prices failure, zero on success). Full suite: 47 passed.🤖 Generated with Claude Code