chore(engine): add force shutdown timeout

[NathanFlurry]

No description provided.

[NathanFlurry]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix(rivetkit/next-js): fix request handler #4001
• fix(rivetkit): handle undefined options with scheduling #4000
• chore(engine): add force shutdown timeout #3999 👈 (View in Graphite)
• fix(rivetkit): fix fetch handler in nextjs serverless #3998
• fix(rivetkit): add back dev hot reloading in next #3997
• fix(rivetkit): fix routing of /websocket #3995
• fix(rivetkit): add config parameters to spawned serverless engine for local dev #3994
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: Add Force Shutdown Timeout

Summary

This PR adds a configurable force_shutdown_duration timeout to ensure the engine process doesn't hang indefinitely during shutdown. The implementation includes validation to ensure the force shutdown timeout is greater than both worker and guard shutdown durations.

---

✅ Strengths

1. Good safety mechanism: The force shutdown timeout prevents hung processes during shutdown, which is important for reliability
2. Proper validation: Config validation ensures force_shutdown_duration is greater than both worker_shutdown_duration and guard_shutdown_duration (engine/packages/config/src/config/mod.rs:216-226)
3. Sensible defaults: Defaults to guard_shutdown_duration + 30 seconds, which provides a reasonable buffer
4. Structured logging: Uses tracing's structured logging correctly with ?force_shutdown_duration
5. Clear documentation: Comments explain the purpose and requirements

---

🔍 Issues & Suggestions

1. Potential race condition with force exit task (engine/packages/service-manager/src/lib.rs:342-349)

The spawned force exit task is not cancelled if graceful shutdown completes successfully. This means:

• If shutdown completes normally before the timeout, the spawned task will still fire and exit(1) unnecessarily
• This could cause false positives in logs and monitoring

Recommendation: Store the task handle and abort it when graceful shutdown completes:

let force_shutdown_handle = tokio::spawn(async move {
    tracing::info!(?force_shutdown_duration, "force shutdown timer started");
    tokio::time::sleep(force_shutdown_duration).await;
    tracing::warn!("force shutdown timeout reached, exiting process, this indicates a bug");
    std::process::exit(1);
});

// ... later, after graceful shutdown completes ...
force_shutdown_handle.abort();

2. Exit code should be more distinctive

Using exit(1) for a force shutdown timeout is ambiguous. Consider using a specific exit code (e.g., exit(124) like timeout command, or exit(143) for SIGTERM-related).

3. Missing structured logging field

The warning message at line 347 could benefit from including which services/tasks were still running when the timeout fired. This would help debug why shutdown hung.

4. No test coverage

There are no tests validating:

• The config validation logic (that force > max(worker, guard))
• The force shutdown behavior
• Edge cases (e.g., what happens if force_shutdown_duration is set but worker/guard are not)

Consider adding unit tests for the validation logic in engine/packages/config/src/config/mod.rs.

5. Documentation comment style inconsistency (engine/packages/config/src/config/runtime.rs:36)

The doc comment says "Returns the force shutdown duration" but doesn't mention it also computes the default. Consider:

/// Returns the force shutdown duration.
/// 
/// Defaults to guard_shutdown_duration + 30 seconds if not explicitly configured.
pub fn force_shutdown_duration(&self) -> Duration {

---

🔒 Security Considerations

No security concerns identified. The use of std::process::exit(1) is appropriate for this use case.

---

⚡ Performance Considerations

• Spawning the tokio task has negligible overhead
• No performance concerns

---

📝 Style & Convention Notes

1. Good adherence to conventions: Structured logging is used correctly per CLAUDE.md
2. Timestamp naming: Consider if this should be named with _at suffix per CLAUDE.md conventions (though it's a duration, not a timestamp, so current naming is correct)

---

🎯 Recommendation

Approve with suggested changes. The PR is functionally sound but would benefit from:

1. Fixing the race condition by aborting the force shutdown task on graceful completion
2. Adding test coverage for config validation
3. Using a more distinctive exit code

The core functionality is solid and addresses a real operational concern. The validation logic is well-implemented.

[pkg-pr-new]

More templates

• actor-actions
• example-ai-agent
• example-ai-and-user-generated-actors-freestyle
• chat-room
• example-cloudflare-workers
• example-cloudflare-workers-hono
• cross-actor-actions
• example-cursors
• example-cursors-raw-websocket
• custom-serverless
• example-sqlite
• example-elysia
• example-experimental-durable-streams-ai-agent
• hello-world
• example-hono
• hono-react
• example-kitchen-sink
• multi-region
• native-websockets
• next-js
• example-raw-fetch-handler
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-react
• scheduling
• state
• example-stream
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3999

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3999

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3999

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3999

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3999

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3999

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3999

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@3999

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3999

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3999

commit: 52751a1

[graphite-app]

Merge activity

• Jan 22, 6:58 AM UTC: NathanFlurry added this pull request to the Graphite merge queue.
• Jan 22, 6:59 AM UTC: CI is running for this pull request on a draft pull request (#4007) due to your merge queue CI optimization settings.
• Jan 22, 7:00 AM UTC: Merged by the Graphite merge queue via draft PR: #4007.