fix(conductor): stop bridge.py from spawning duplicate conductor sessions#1609
fix(conductor): stop bridge.py from spawning duplicate conductor sessions#1609youling66 wants to merge 5 commits into
Conversation
…ions
ensure_conductor_running looked up the conductor session by a fixed
title ("conductor-<name>"), but the session was created without
--title-lock, so agent-deck's title-sync overwrote that title with the
agent's own session name on its first hook event. Every later restart
then failed the exact-title lookup (including the asheshgoplani#1380 dedupe check,
which matches the same way) and created a brand new conductor session
instead of reusing the existing one.
Combined with a second gap - main() awaited every platform task via a
single asyncio.gather() with no exception handling, so one transient
network error (e.g. a proxy timeout on Telegram's getMe call) crashed
the whole bridge process, which the OS service manager immediately
respawned - a single flaky network connection turned into a duplicate
conductor session roughly every 15-30 minutes, indefinitely.
Fixes both:
- pass --title-lock when creating a conductor session, so its title
can never drift away from what the bridge looks it up by
- wrap each platform task (Telegram/Slack/Discord) in
_run_platform_task, which retries with exponential backoff instead
of letting the exception escape gather() and kill the process
📝 WalkthroughWalkthroughThe conductor bridge now uses fail-closed, path-based session deduplication with title locking and supervises Telegram, Slack, and Discord tasks with cancellation-aware exponential retry backoff. Regression tests cover session identity, retries, cancellation, overlap, and capped delays. ChangesConductor bridge resilience
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant main
participant asyncio.gather
participant _run_platform_task
participant PlatformIntegration
main->>asyncio.gather: await supervised platform tasks
asyncio.gather->>_run_platform_task: start task supervisor
_run_platform_task->>PlatformIntegration: execute integration
PlatformIntegration-->>_run_platform_task: failure or success
_run_platform_task->>_run_platform_task: calculate capped backoff
_run_platform_task->>PlatformIntegration: retry after sleep
🚥 Pre-merge checks | ✅ 6 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (6 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
internal/session/conductor_bridge.py (1)
2887-2887: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd a type hint for
coro_factory.The
coro_factoryparameter lacks a type annotation. AddingCallable[[], Awaitable[None]](orCallable[[], Coroutine[Any, Any, None]]) would improve readability and IDE support.♻️ Optional type hint
-async def _run_platform_task(name: str, coro_factory, max_backoff: int = 300) -> None: +from typing import Awaitable, Callable + +async def _run_platform_task( + name: str, coro_factory: Callable[[], Awaitable[None]], max_backoff: int = 300 +) -> None:As per path instructions,
internal/session/**requires extra attention to signal handling — the explicitexcept asyncio.CancelledError: raisebefore the broadExceptioncatch correctly ensures SIGINT/SIGTERM cancellation propagates for clean teardown.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/session/conductor_bridge.py` at line 2887, Update the _run_platform_task function signature by annotating coro_factory as a zero-argument callable returning an awaitable of None, using the appropriate typing imports. Preserve the existing cancellation handling, including explicit propagation of asyncio.CancelledError before the broad Exception catch.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@internal/session/conductor_bridge.py`:
- Line 2887: Update the _run_platform_task function signature by annotating
coro_factory as a zero-argument callable returning an awaitable of None, using
the appropriate typing imports. Preserve the existing cancellation handling,
including explicit propagation of asyncio.CancelledError before the broad
Exception catch.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: de0ee0a6-2a97-463f-87f8-cd2c4a8ee9d4
📒 Files selected for processing (3)
conductor/tests/test_bridge_task_resilience.pyconductor/tests/test_conductor_title_lock.pyinternal/session/conductor_bridge.py
Addresses CodeRabbit nitpick on PR asheshgoplani#1609.
Reuse the unique profile-scoped session at the canonical conductor path by stable ID. Restore and lock drifted titles, fail closed on ambiguous or unverifiable identity, and extend retry lifecycle coverage for issue asheshgoplani#1608. Committed by Ashesh Goplani
Treat valid JSON with a non-list sessions field as unverifiable in fail-closed identity checks, preventing accidental conductor creation from malformed CLI output. Committed by Ashesh Goplani
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/session/conductor_bridge.py`:
- Around line 976-1014: Update the session selection logic around
_find_session_by_title and the path_match or exact_match assignment to detect
when path_match and exact_match both exist but refer to different sessions. Fail
migration closed with an error before invoking the rename through run_cli, while
preserving the existing behavior when only one match exists or both identify the
same session.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 30c2b993-f2ff-4c48-b55a-c09e8f3c05b9
📒 Files selected for processing (4)
conductor/tests/test_bridge_task_resilience.pyconductor/tests/test_conductor_title_lock.pyconductor/tests/test_issue1351_conductor_dedupe.pyinternal/session/conductor_bridge.py
Fail closed when the canonical conductor path and expected title resolve to different session IDs, preventing a drift repair from renaming onto an existing title. Committed by Ashesh Goplani
|
Pulled these commits and ran them on the machine that originally hit #1608 — full test suite passes (73/73, aside from 6 pre-existing venv-only failures unrelated to this change), and the deployed process has now run 19+ hours straight with zero restarts and the conductor session count holding steady at 1, including surviving a real Telegram "Bad Gateway" blip along the way without crashing. Looks solid to me. |
Fixes #1608.
Original implementation and authorship by @youling66. Maintainer integration commits add migration safety and regression coverage without squashing the contributor commits.
Root cause
The bridge identified conductor sessions only by the expected
conductor-<name>title. Existing sessions whose title had already drifted were invisible to that lookup, so an upgrade could still create another row. Separately, an uncaught Telegram, Slack, or Discord polling exception escapedasyncio.gather()and caused the service manager to restart the bridge repeatedly.Fix
--title-lockon every newly created conductor session.list --jsonrecord and canonical conductor project path for drifted sessions.CancelledErrorpropagates, normal return ends the wrapper, failures log loudly, and backoff grows from 5 seconds to a 300 second cap.Surfaces
conductor/tests/test_conductor_title_lock.py,conductor/tests/test_bridge_task_resilience.py, and the updated Restarting the conductor bridge registers a duplicate conductor instance (phantom) instead of reusing the existing one #1351 compatibility tests.id,title, andpathfields fromlist --jsonand existingsession setbehavior.Verification
go build ./...: passed.govulncheck ./...: zero reachable vulnerabilities locally and passed in GitHub Actions.main:TestForbidigo_BansRawOSEnvironInSpawnPathandTestGetJSONLPathChecked_SameIDDifferentProjectIsNotCollision.git diff --check: passed.Do not merge as part of this integration task.