Add supervised background-task helpers to FleetConnector#83
Merged
Conversation
Periodic work placed in `_execution_loop` is already supervised by the framework: `__run_loop` wraps it in try/except, logs the traceback, and retries on the next tick. Work scheduled with a bare `asyncio.create_task` is not: if it raises, the exception is stored on a task nobody awaits (so it is never logged), the task is never restarted, and whatever it fed (e.g. pose / key-values) freezes until the process restarts -- while independent datasources keep working, which masks the failure. Add to FleetConnector: - `_create_supervised_task(name, coro_factory, restart_delay)`: runs a long-lived loop that is logged-with-traceback and restarted if it ever exits or crashes (CancelledError propagates for clean shutdown). - `_spawn_logged_task(coro)`: one-shot tasks whose failures are surfaced via a done-callback instead of dying silently. - Track these tasks and cancel/await them in `__disconnect`. Tests: 3 new cases in tests/test_connector.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
edge-sdk 3.1.0 stops a callback exception from killing the paho network loop thread (guarded callbacks + suppress_exceptions) and surfaces it via enable_logger. Connectors built on this framework get that fix transitively. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ckoff - examples/simple-connector: move odometry polling into a supervised _speed_poll_loop scheduled via _create_supervised_task, showing a loop on a faster cadence than _execution_loop that logs+restarts instead of dying. - __supervise: TODO(metrics) to emit a background_task_errors counter, and TODO(backoff) for capped exponential backoff on repeated crashes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- examples/simple-connector: add a one-shot _announce_firmware startup task scheduled via _spawn_logged_task, alongside the supervised speed loop, to show the fire-and-forget-but-logged (not restarted) pattern. - _spawn_logged_task now passes name through to the task so __log_task_exception identifies which task failed (the name arg was previously unused). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both now lead with a required `name`: _spawn_logged_task(name, coro) to match _create_supervised_task(name, coro_factory, ...). Updates the example and test call sites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b-Tomas
requested changes
Jun 9, 2026
b-Tomas
left a comment
b-Tomas
left a comment
Member
There was a problem hiding this comment.
Neat feature
Could you update the documentation as well?
Member
Author
|
Added
…On Tue, Jun 9, 2026 at 7:44 PM Tomás Badenes ***@***.***> wrote:
***@***.**** requested changes on this pull request.
Neat feature
Could you update the documentation as well?
—
Reply to this email directly, view it on GitHub
<#83?email_source=notifications&email_token=AB5XVRDLMNPKF6CFB7V6U3T47CHMPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINBWGMYDCNJWG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4463015675>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB5XVRGG4FVUNTHBIMRNNOL47CHMPAVCNFSM6AAAAAC2BF76V2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHM2DINRTGAYTKNRXGU>
.
Triage notifications, keep track of coding agent tasks and review pull
requests on the go with GitHub Mobile for iOS
<https://github.com/notifications/mobile/ios/AB5XVRA75RP7LTO5OFBVFVT47CHMPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINBWGMYDCNJWG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG>
and Android
<https://github.com/notifications/mobile/android/AB5XVRGJC7LXPBTCI4Z6CIL47CHMPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINBWGMYDCNJWG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>.
Download it today!
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
Leandro Pineda
Engineer, InOrbit.ai <http://inorbit.ai/>
|
b-Tomas
approved these changes
Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Periodic work placed in
_execution_loopis already supervised by the framework:__run_loopwraps each call in try/except, logs the traceback, and retries on the next tick. Work a connector schedules itself with a bareasyncio.create_taskis not supervised — if it raises, the exception is stored on a task nobody awaits (never logged), the task is never restarted, and whatever it fed (pose, key-values, etc.) freezes until the process restarts, while independent datasources keep working (which masks the failure).This adds first-class, reusable helpers so connectors that need off-cadence loops (e.g. a fast pose loop + a slower key-value loop) get supervision instead of hand-rolling fire-and-forget tasks.
Changes
FleetConnector._create_supervised_task(name, coro_factory, restart_delay=5.0)— runs a long-lived loop that is logged-with-traceback and restarted if it ever exits/crashes.asyncio.CancelledErrorpropagates for clean shutdown.FleetConnector._spawn_logged_task(name, coro)— one-shot tasks whose failure is logged (with traceback) via a done-callback instead of dying silently; not restarted. Samename-first signature as_create_supervised_taskfor consistency.__log_task_exceptiondone-callback (used by both) surfaces a task's exception, tagged with the task name.__disconnect.inorbit-edge>=3.1.0(pyproject.toml) so connectors built on this framework transitively get the edge-sdk MQTT callback-resilience fix.examples/simple-connectornow demonstrates both helpers: a supervised_speed_poll_loop(faster cadence than_execution_loop) and a one-shot_announce_firmwarescheduled via_spawn_logged_task.__supervisefor follow-ups: abackground_task_errorsmetric and capped exponential backoff on repeated crashes.New (additive) framework API; the only dependency change is the
inorbit-edgelower-bound bump.Test plan
tests/test_connector.py::TestSupervisedBackgroundTasks(restart-on-crash, cancellation-on-__disconnect, one-shot failure logged + named).examples/simple-connector(session pool mocked, faults injected): supervised loop crashes → logs → restarts → recovers; one-shot fails → logs once → not restarted.flake8clean,black --checkclean.🤖 Generated with Claude Code