Fix Scheduler freeze when SQL Server has downtime#159
Open
jduust wants to merge 1 commit into
Open
Conversation
Author
|
Also if you were in doubt, yes this is mostly generated with AI. However we've been running a slightly more barebones "nonlinted" version of the code on one of our machines, and it seems to work as intended so far. Use it if it makes sense, it's a huge gamechanger for us in terms of uptime and catching processes stuck on running. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the scheduler freeze that occurs when SQL Server has downtime or restarts (issues #152 and #106).
Root cause
The Tk
after-chain inrun_tab.loop()died on any unhandled DB exception, leaving the scheduler frozen with no further polls scheduled. SQLAlchemy's pooled connections also went stale across server restarts (nopool_pre_ping).Changes
db_util.connect(): enablespool_pre_ping=Trueandpool_recycle=1800. Engine is kept alive on operational probe failure so it can self-heal viapool_pre_pingon the next session checkout. A 5 s ODBC login timeout is set only formssql+pyodbcconnection strings (driver-specific via a helper); other drivers use their defaults.run_tab.loop(): body wrapped intry/except/finallyso theapp.afterreschedule always runs. Exponential backoff12s -> 24s -> ... -> 600s (cap), counter resets on the next successful tick. Tracebacks appended to a desktop log file so failures that happened overnight are visible afterwards.run_tab.check_heartbeats(): per-job DB calls wrapped so one job's failure doesn't abort the whole sweep; first DB error is re-raised soloop()enters the backoff path.runner+ newscheduler/inflight.py: trigger IDs this scheduler owns are persisted to%APPDATA%/OpenOrchestrator/inflight.json. OnRunclick,reconcile_orphans()reads the file and marks any still-RUNNINGtriggers asFAILEDwith a log entry. Closes the gap where a crash betweenbegin_*_trigger()and the job ending leaves a trigger stuck inRUNNING.How I tested
pylint --rcfile=.pylintrc OpenOrchestrator --good-names=OpenOrchestrator: 10.00/10flake8 --extend-ignore=E501,E251 OpenOrchestrator: cleanpython -m unittest discoveragainst sqlite: passesLOST DATABASE CONNECTION, backoff progression, thenDatabase connection restored after N failed attempt(s). No freeze, no manual restart required.Things worth discussing in review
~/Desktop/OpenOrchestrator_scheduler_errors.log) is opinionated. Happy to switch to Python'sloggingmodule / a configurable path if preferred.pool_recycle=1800is a default that should be fine for most setups; can be made configurable if needed.started_bycolumn on theTriggerstable so reconciliation can happen DB-side without local state — out of scope for this PR.