fix(server): keep provider sessions alive during background agent work#4199
fix(server): keep provider sessions alive during background agent work#4199ChamaruAmasara wants to merge 3 commits into
Conversation
The provider session reaper stops sessions after 30 min of lastSeenAt inactivity. Background agent work (dynamic workflows, subagents) keeps running after the foreground turn settles, but the adapter session goes "ready" with no active turn and nothing refreshes lastSeenAt — so the reaper tears the session down mid-workflow, and the in-flight background orchestration is lost. Refresh lastSeenAt from runtime activity: ProviderService now bumps the binding's last_seen_at when the adapter emits runtime events (e.g. task.progress from background subagents), throttled per thread so an event burst is at most one lightweight write per window. A new targeted touchLastSeen on the runtime repository / session directory updates only last_seen_at, and only for non-stopped rows so a reaped session is never resurrected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8bd44f2. Configure here.
ApprovabilityVerdict: Needs human review 1 blocking correctness issue found. This PR introduces new runtime behavior for session lifecycle management (keeping sessions alive during background work). There is an unresolved review comment about a potential memory leak in the throttle map that should be addressed before merge. You can customize Macroscope's approvability policy. Learn more. |
Address review feedback on the runtime-activity lastSeenAt refresh: - Arm the per-thread throttle window only after touchLastSeen succeeds, not before. Previously a failed (silently caught) touch still armed the window, suppressing the next refresh and leaving last_seen_at stale — which could let the reaper reap a live session. Failed touches now let the next event retry immediately. - Catch only the error channel (Effect.catch) instead of the full cause, so fiber interrupts propagate and don't delay subscription teardown. Add a regression test: a touch that fails once is retried on the next event (not throttled away), and the window arms once the touch succeeds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| // reaper never mistakes an active background session for an idle one. One | ||
| // write per thread per window is plenty and keeps the DB churn negligible. | ||
| const lastSeenTouchMs = options?.runtimeActivityTouchThrottleMs ?? 60_000; | ||
| const lastSeenTouchByThread = new Map<ThreadId, number>(); |
There was a problem hiding this comment.
🟡 Medium Layers/ProviderService.ts:228
The lastSeenTouchByThread map in refreshLastSeenForActivity retains an entry for every threadId that ever emits a runtime event, and entries are never removed when sessions stop or are replaced. On a long-lived server serving an unbounded sequence of threads, this map grows for the process lifetime, leaking memory. Consider evicting entries in stopSession and runStopAll (e.g., via a small clearLastSeenTouch(threadId) helper) so the map stays bounded by the number of active threads.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Layers/ProviderService.ts around line 228:
The `lastSeenTouchByThread` map in `refreshLastSeenForActivity` retains an entry for every `threadId` that ever emits a runtime event, and entries are never removed when sessions stop or are replaced. On a long-lived server serving an unbounded sequence of threads, this map grows for the process lifetime, leaking memory. Consider evicting entries in `stopSession` and `runStopAll` (e.g., via a small `clearLastSeenTouch(threadId)` helper) so the map stays bounded by the number of active threads.

Closes #4198
What Changed
touchLastSeento the provider session runtime repository and session directory: a lightweight update that bumps onlylast_seen_at, and only for non-stoppedrows (so a reaped session is never resurrected).ProviderService, refresh a session'slast_seen_atfrom runtime activity:processRuntimeEventnow callstouchLastSeenwhen the adapter emits runtime events, throttled per thread (default 60s) so an event burst is at most one lightweight write per window. The touch is best-effort — a failed write never breaks event processing.touchLastSeen(bumps a live row; leaves a stopped row untouched); aProviderServicetest proving a runtime event triggers a touch and that a rapid burst is throttled.Why
ProviderSessionReaperstops any provider session after 30 min oflast_seen_atinactivity. Background agent work — dynamic workflows and subagents — keeps running after the foreground turn settles, but at that point the adapter session has gonereadywith no active turn, and nothing refresheslast_seen_at. So a long-running background workflow with no foreground turn is torn down mid-run, and the in-flight background orchestration is lost.Runtime events are the reliable signal for background activity: while a workflow runs, the adapter keeps emitting events (e.g.
task.progressfrom subagents) even with no active foreground turn. Refreshinglast_seen_atfrom these events keeps the session warm exactly when the reaper would otherwise reap it. Throttling keeps DB churn negligible (one write per thread per window instead of one per event).Validation
vp test run apps/server/src/provider/Layers/ProviderService.test.ts— includes a new test that a runtime event toucheslastSeenAtand that a burst is throttled (verified it fails without the fix).vp test run apps/server/src/provider/Layers/ProviderSessionDirectory.test.ts—touchLastSeenbumps a live binding; leaves astoppedbinding untouched.last_seen_atkept advancing fromtask.progressactivity and the reaper did not reap the session past its idle threshold.UI Changes
None.
Checklist
Note
Medium Risk
Changes session lifecycle timing used by the reaper; incorrect throttling or stopped-row handling could cause premature stops or stale sessions, but scope is narrow and heavily tested.
Overview
Adds
touchLastSeenfrom persistence throughProviderSessionDirectory: a lightweight SQL update that only bumpslast_seen_atand skips rows withstatus = 'stopped', so reaped sessions are not resurrected.ProviderServiceLivenow callstouchLastSeenwhile handling canonical runtime events (e.g.task.progress), with per-thread throttling (default 60s, overridable viaruntimeActivityTouchThrottleMs). Touches are best-effort (errors swallowed) and the throttle window is armed only after a successful write so transient failures do not block retries.Tests cover directory/repository behavior, throttled touches from runtime events, and retry when the first touch fails.
Reviewed by Cursor Bugbot for commit 4f64170. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Keep provider sessions alive by touching
last_seen_atduring background agent activitytouchLastSeen(threadId)toProviderSessionDirectoryand its underlying repository, issuing a targeted SQLUPDATEonlast_seen_atthat skips rows withstatus = 'stopped'.ProviderServicecallstouchLastSeenon eachProviderRuntimeEvent, throttled to at most one successful write per thread per 60s (configurable viaruntimeActivityTouchThrottleMs).Macroscope summarized 4f64170.