Skip to content

fix(server): keep provider sessions alive during background agent work#4199

Open
ChamaruAmasara wants to merge 3 commits into
pingdotgg:mainfrom
ChamaruAmasara:fix/workflows-session-restart
Open

fix(server): keep provider sessions alive during background agent work#4199
ChamaruAmasara wants to merge 3 commits into
pingdotgg:mainfrom
ChamaruAmasara:fix/workflows-session-restart

Conversation

@ChamaruAmasara

@ChamaruAmasara ChamaruAmasara commented Jul 20, 2026

Copy link
Copy Markdown

Closes #4198

What Changed

  • Add a targeted touchLastSeen to the provider session runtime repository and session directory: a lightweight update that bumps only last_seen_at, and only for non-stopped rows (so a reaped session is never resurrected).
  • In ProviderService, refresh a session's last_seen_at from runtime activity: processRuntimeEvent now calls touchLastSeen when the adapter emits runtime events, throttled per thread (default 60s) so an event burst is at most one lightweight write per window. The touch is best-effort — a failed write never breaks event processing.
  • Tests: persistence/directory tests for touchLastSeen (bumps a live row; leaves a stopped row untouched); a ProviderService test proving a runtime event triggers a touch and that a rapid burst is throttled.

Why

ProviderSessionReaper stops any provider session after 30 min of last_seen_at inactivity. Background agent work — dynamic workflows and subagents — keeps running after the foreground turn settles, but at that point the adapter session has gone ready with no active turn, and nothing refreshes last_seen_at. So a long-running background workflow with no foreground turn is torn down mid-run, and the in-flight background orchestration is lost.

Runtime events are the reliable signal for background activity: while a workflow runs, the adapter keeps emitting events (e.g. task.progress from subagents) even with no active foreground turn. Refreshing last_seen_at from these events keeps the session warm exactly when the reaper would otherwise reap it. Throttling keeps DB churn negligible (one write per thread per window instead of one per event).

Validation

  • vp test run apps/server/src/provider/Layers/ProviderService.test.ts — includes a new test that a runtime event touches lastSeenAt and that a burst is throttled (verified it fails without the fix).
  • vp test run apps/server/src/provider/Layers/ProviderSessionDirectory.test.tstouchLastSeen bumps a live binding; leaves a stopped binding untouched.
  • All affected provider test files pass (83 tests).
  • End-to-end: with a live server and a running background workflow, last_seen_at kept advancing from task.progress activity and the reaper did not reap the session past its idle threshold.
  • Typecheck: 0 errors. Lint: clean.

UI Changes

None.

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • No UI changes

Note

Medium Risk
Changes session lifecycle timing used by the reaper; incorrect throttling or stopped-row handling could cause premature stops or stale sessions, but scope is narrow and heavily tested.

Overview
Adds touchLastSeen from persistence through ProviderSessionDirectory: a lightweight SQL update that only bumps last_seen_at and skips rows with status = 'stopped', so reaped sessions are not resurrected.

ProviderServiceLive now calls touchLastSeen while handling canonical runtime events (e.g. task.progress), with per-thread throttling (default 60s, overridable via runtimeActivityTouchThrottleMs). Touches are best-effort (errors swallowed) and the throttle window is armed only after a successful write so transient failures do not block retries.

Tests cover directory/repository behavior, throttled touches from runtime events, and retry when the first touch fails.

Reviewed by Cursor Bugbot for commit 4f64170. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Keep provider sessions alive by touching last_seen_at during background agent activity

  • Adds touchLastSeen(threadId) to ProviderSessionDirectory and its underlying repository, issuing a targeted SQL UPDATE on last_seen_at that skips rows with status = 'stopped'.
  • ProviderService calls touchLastSeen on each ProviderRuntimeEvent, throttled to at most one successful write per thread per 60s (configurable via runtimeActivityTouchThrottleMs).
  • Failed writes are silently ignored and do not arm the throttle, so the next event will retry immediately.

Macroscope summarized 4f64170.

The provider session reaper stops sessions after 30 min of lastSeenAt
inactivity. Background agent work (dynamic workflows, subagents) keeps
running after the foreground turn settles, but the adapter session goes
"ready" with no active turn and nothing refreshes lastSeenAt — so the
reaper tears the session down mid-workflow, and the in-flight background
orchestration is lost.

Refresh lastSeenAt from runtime activity: ProviderService now bumps the
binding's last_seen_at when the adapter emits runtime events (e.g.
task.progress from background subagents), throttled per thread so an
event burst is at most one lightweight write per window. A new targeted
touchLastSeen on the runtime repository / session directory updates only
last_seen_at, and only for non-stopped rows so a reaped session is never
resurrected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 8973ec75-6d69-4408-bcba-6bc8e771dd5f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Jul 20, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8bd44f2. Configure here.

Comment thread apps/server/src/provider/Layers/ProviderService.ts Outdated
Comment thread apps/server/src/provider/Layers/ProviderService.ts
@macroscopeapp

macroscopeapp Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

1 blocking correctness issue found. This PR introduces new runtime behavior for session lifecycle management (keeping sessions alive during background work). There is an unresolved review comment about a potential memory leak in the throttle map that should be addressed before merge.

You can customize Macroscope's approvability policy. Learn more.

ChamaruAmasara and others added 2 commits July 21, 2026 11:02
Address review feedback on the runtime-activity lastSeenAt refresh:

- Arm the per-thread throttle window only after touchLastSeen succeeds,
  not before. Previously a failed (silently caught) touch still armed the
  window, suppressing the next refresh and leaving last_seen_at stale —
  which could let the reaper reap a live session. Failed touches now let
  the next event retry immediately.
- Catch only the error channel (Effect.catch) instead of the full cause,
  so fiber interrupts propagate and don't delay subscription teardown.

Add a regression test: a touch that fails once is retried on the next
event (not throttled away), and the window arms once the touch succeeds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
// reaper never mistakes an active background session for an idle one. One
// write per thread per window is plenty and keeps the DB churn negligible.
const lastSeenTouchMs = options?.runtimeActivityTouchThrottleMs ?? 60_000;
const lastSeenTouchByThread = new Map<ThreadId, number>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Layers/ProviderService.ts:228

The lastSeenTouchByThread map in refreshLastSeenForActivity retains an entry for every threadId that ever emits a runtime event, and entries are never removed when sessions stop or are replaced. On a long-lived server serving an unbounded sequence of threads, this map grows for the process lifetime, leaking memory. Consider evicting entries in stopSession and runStopAll (e.g., via a small clearLastSeenTouch(threadId) helper) so the map stays bounded by the number of active threads.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Layers/ProviderService.ts around line 228:

The `lastSeenTouchByThread` map in `refreshLastSeenForActivity` retains an entry for every `threadId` that ever emits a runtime event, and entries are never removed when sessions stop or are replaced. On a long-lived server serving an unbounded sequence of threads, this map grows for the process lifetime, leaking memory. Consider evicting entries in `stopSession` and `runStopAll` (e.g., via a small `clearLastSeenTouch(threadId)` helper) so the map stays bounded by the number of active threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Idle session reaper kills in-flight background agent work (dynamic workflows / subagents)

1 participant