feat(commits-processor): drain pending intents before shutdown

[0xzrf]

Summary

Implements graceful shutdown for the commit pipeline so in-flight commit intents finish, results are broadcast and processed, and workers exit cleanly instead of aborting mid-flight. Closes #1262.

Shutdown is wired end-to-end across the stack:

• Intent execution engine/manager — drain mode on cancel; joins in-flight executors before signaling idle; rejects new schedules with ShuttingDown.
• CommittorProcessor — shutdown() drains the execution manager; rejects new and recovered schedules during shutdown.
• CommittorActor / CommittorService — two-phase shutdown: reject queued/mutating messages, then processor.shutdown().await; stopped() waits for actor drain completion.
• CommittorServiceExt — dispatcher stays in drain mode after stopped() until the results channel closes, delivering in-flight waiter results.
• ScheduledCommitsProcessor — result listener drains after cancel; shutdown() waits until intents_meta_map is empty; process() no-ops once stopped.
• MagicValidator::stop — reordered shutdown: committor stop() + stopped().await, then SCP shutdown().await, then global token cancel and remaining teardown.

Breaking Changes

• None

Test Plan

• cargo test -p magicblock-committor-service shutdown — 16 tests
• cargo test -p magicblock-accounts shutdown — 2 tests

Summary by CodeRabbit

• New Features

  • Added graceful shutdown mechanism that drains in-flight pending work before service termination, ensuring no pending results are lost during shutdown.
  • Services now reject new operations with a "shutting down" error when shutdown is initiated.

• Tests

  • Added comprehensive test coverage for shutdown draining behavior across multiple service components.

[coderabbitai]

Warning

Review limit reached

@0xzrf, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 44 minutes and 25 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: be432901-9e2e-4d32-9d2a-983992bab452

📥 Commits

Reviewing files that changed from the base of the PR and between 728725e and 614d65c.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (12)

• magicblock-accounts/Cargo.toml
• magicblock-accounts/src/scheduled_commits_processor.rs
• magicblock-accounts/src/traits.rs
• magicblock-api/src/magic_validator.rs
• magicblock-committor-service/src/committor_processor.rs
• magicblock-committor-service/src/error.rs
• magicblock-committor-service/src/intent_execution_manager.rs
• magicblock-committor-service/src/intent_execution_manager/intent_execution_engine.rs
• magicblock-committor-service/src/intent_execution_manager/intent_scheduler.rs
• magicblock-committor-service/src/intent_execution_manager/test_support.rs
• magicblock-committor-service/src/service.rs
• magicblock-committor-service/src/service_ext.rs

📝 Walkthrough

Walkthrough

This PR implements coordinated graceful shutdown across the entire committor service pipeline. Starting at the lowest layer, IntentExecutionEngine gains drain-mode logic that stops accepting new work after a CancellationToken is cancelled but continues executing in-flight intents, signalling idle via a watch channel. IntentExecutionManager and CommittorProcessor wire this token/idle channel pattern up their respective layers and guard scheduling entry points with ShuttingDown early-returns. CommittorActor is redesigned with a dual-token pattern (shutdown_token/actor_done_token) and drains/rejects queued messages before calling processor.shutdown(). CommittorServiceExt gains a drain-mode dispatcher. ScheduledCommitsProcessorImpl gains a polling-drain shutdown() method. MagicValidator::stop is reordered to await each layer's drain sequentially before cancelling the validator token.

Assessment against linked issues

Objective	Addressed	Explanation
Wait until all in-flight intent results arrive before exiting worker/committing shutdown [#1262]	✅
Finalize ongoing intents on shutdown instead of aborting mid-way [#1262]	✅
CommittorProcessor graceful shutdown support [#1262]	✅

Suggested reviewers

• GabrielePicco
• taco-paco
• snawaz

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

magicblock-committor-service/src/intent_execution_manager/intent_execution_engine.rs (1)

196-201: 🧹 Nitpick | 🔵 Trivial | 💤 Low value

Existing .expect() on semaphore acquire.

Per coding guidelines, .expect() in production Rust code under magicblock-*/** should use proper error handling or have explicit invariant justification. While the semaphore is owned by the engine and never closed (making this practically unreachable), consider adding a comment documenting this invariant or converting to a match that handles the impossible case.

The semaphore is created in new() with MAX_EXECUTORS permits and is never explicitly closed, so AcquireError is effectively unreachable during normal operation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@magicblock-committor-service/src/intent_execution_manager/intent_execution_engine.rs`
around lines 196 - 201, The `.expect()` call on the semaphore acquire operation
lacks proper error handling or invariant justification required by the coding
guidelines. Either add a comment explaining the invariant (that the semaphore is
created in the constructor with MAX_EXECUTORS permits and is never explicitly
closed, making AcquireError effectively unreachable), or refactor the code to
use a match statement that explicitly handles the error case, even if it's
unreachable. The change should be made at the point where
`executors_semaphore.clone().acquire_owned().await.expect(SEMAPHORE_CLOSED_MSG)`
is called.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@magicblock-committor-service/src/service_ext.rs`:
- Around line 118-130: The `.expect()` calls on mutex lock operations lack
explicit inline comments justifying the panic, as required by coding guidelines.
Add a comment before each `.expect(POISONED_MUTEX_MSG)` call explaining that
mutex poisoning indicates a prior panic occurred while holding the lock, which
means continuing in that state would be unsafe and therefore justifying the
panic. This pattern of mutex lock followed by expect appears multiple times in
the CommittorServiceExt code and should have this justification documented at
each occurrence.

---

Outside diff comments:
In
`@magicblock-committor-service/src/intent_execution_manager/intent_execution_engine.rs`:
- Around line 196-201: The `.expect()` call on the semaphore acquire operation
lacks proper error handling or invariant justification required by the coding
guidelines. Either add a comment explaining the invariant (that the semaphore is
created in the constructor with MAX_EXECUTORS permits and is never explicitly
closed, making AcquireError effectively unreachable), or refactor the code to
use a match statement that explicitly handles the error case, even if it's
unreachable. The change should be made at the point where
`executors_semaphore.clone().acquire_owned().await.expect(SEMAPHORE_CLOSED_MSG)`
is called.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 76be0bb6-49fe-41a9-af15-34e01afd4992

📥 Commits

Reviewing files that changed from the base of the PR and between 34480c1 and 728725e.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (12)

• magicblock-accounts/Cargo.toml
• magicblock-accounts/src/scheduled_commits_processor.rs
• magicblock-accounts/src/traits.rs
• magicblock-api/src/magic_validator.rs
• magicblock-committor-service/src/committor_processor.rs
• magicblock-committor-service/src/error.rs
• magicblock-committor-service/src/intent_execution_manager.rs
• magicblock-committor-service/src/intent_execution_manager/intent_execution_engine.rs
• magicblock-committor-service/src/intent_execution_manager/intent_scheduler.rs
• magicblock-committor-service/src/intent_execution_manager/test_support.rs
• magicblock-committor-service/src/service.rs
• magicblock-committor-service/src/service_ext.rs