Skip to content

Fix Scheduler freeze when SQL Server has downtime#159

Open
jduust wants to merge 1 commit into
itk-dev-rpa:developfrom
jduust:fix/sql-resilience
Open

Fix Scheduler freeze when SQL Server has downtime#159
jduust wants to merge 1 commit into
itk-dev-rpa:developfrom
jduust:fix/sql-resilience

Conversation

@jduust

@jduust jduust commented May 20, 2026

Copy link
Copy Markdown

Summary

Fixes the scheduler freeze that occurs when SQL Server has downtime or restarts (issues #152 and #106).

Root cause

The Tk after-chain in run_tab.loop() died on any unhandled DB exception, leaving the scheduler frozen with no further polls scheduled. SQLAlchemy's pooled connections also went stale across server restarts (no pool_pre_ping).

Changes

  • db_util.connect(): enables pool_pre_ping=True and pool_recycle=1800. Engine is kept alive on operational probe failure so it can self-heal via pool_pre_ping on the next session checkout. A 5 s ODBC login timeout is set only for mssql+pyodbc connection strings (driver-specific via a helper); other drivers use their defaults.
  • run_tab.loop(): body wrapped in try/except/finally so the app.after reschedule always runs. Exponential backoff 12s -> 24s -> ... -> 600s (cap), counter resets on the next successful tick. Tracebacks appended to a desktop log file so failures that happened overnight are visible afterwards.
  • run_tab.check_heartbeats(): per-job DB calls wrapped so one job's failure doesn't abort the whole sweep; first DB error is re-raised so loop() enters the backoff path.
  • runner + new scheduler/inflight.py: trigger IDs this scheduler owns are persisted to %APPDATA%/OpenOrchestrator/inflight.json. On Run click, reconcile_orphans() reads the file and marks any still-RUNNING triggers as FAILED with a log entry. Closes the gap where a crash between begin_*_trigger() and the job ending leaves a trigger stuck in RUNNING.

How I tested

  • pylint --rcfile=.pylintrc OpenOrchestrator --good-names=OpenOrchestrator: 10.00/10
  • flake8 --extend-ignore=E501,E251 OpenOrchestrator: clean
  • python -m unittest discover against sqlite: passes
  • Manual chaos test on a robot machine: Windows Firewall blocked outbound TCP/1433 to the SQL Server for 15s/60s/90s. Scheduler logged LOST DATABASE CONNECTION, backoff progression, then Database connection restored after N failed attempt(s). No freeze, no manual restart required.

Things worth discussing in review

  • The desktop crash log path (~/Desktop/OpenOrchestrator_scheduler_errors.log) is opinionated. Happy to switch to Python's logging module / a configurable path if preferred.
  • pool_recycle=1800 is a default that should be fine for most setups; can be made configurable if needed.
  • The orphan reconciliation is per-machine via a local JSON file. A future cleaner approach would be a started_by column on the Triggers table so reconciliation can happen DB-side without local state — out of scope for this PR.

@jduust

jduust commented May 21, 2026

Copy link
Copy Markdown
Author

Also if you were in doubt, yes this is mostly generated with AI. However we've been running a slightly more barebones "nonlinted" version of the code on one of our machines, and it seems to work as intended so far. Use it if it makes sense, it's a huge gamechanger for us in terms of uptime and catching processes stuck on running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant