Skip to content

fix(api): retain commands after ambiguous Queue sends#299

Merged
Yevanchen merged 3 commits into
langgenius:mainfrom
AsperforMias:fix/api-command-outbox-recovery
Jul 14, 2026
Merged

fix(api): retain commands after ambiguous Queue sends#299
Yevanchen merged 3 commits into
langgenius:mainfrom
AsperforMias:fix/api-command-outbox-recovery

Conversation

@AsperforMias

@AsperforMias AsperforMias commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Persist each API command with a queue_delivery_pending marker before attempting Queue delivery.
  • Treat a successful durable write as accepted work: Queue send failures are recorded for scheduled redrive instead of failing the caller and changing the owning Run to terminal failure.
  • Redrive both commands whose first send never began and commands whose producer response failed; duplicate deliveries remain safe through durable consumer claiming.
  • Fixes fix(api): retain commands when Queue send outcome is ambiguous #284.

Why

A producer-side timeout can occur after Cloudflare Queue has accepted a message. The original path deleted the command record. The first revision retained that record but still threw to App deployment, which marked the DeploymentRun failed before a delivered or redriven command could execute. The durable outbox must be the acceptance boundary, not the synchronous Queue response.

Verification

  • Commands: just test-file apps/api/tests/api-command-queue.test.ts (6 passing); just test-file apps/api/tests/app-deployment-service.test.ts (19 passing); just tc-package @mosoo/api; just commit-check; just check (942 passing tests, plus format, docs, lint, type, and GraphQL checks).
  • Manual steps: Used local SQLite D1 fixtures and Queue fault injection. Verified a retained-but-timeout message remains claimable, a definite Queue failure is redriven, a command persisted before its first send is redriven, and an App deployment remains queued instead of becoming failed when Queue delivery is deferred.
  • Not run: A remote Cloudflare Queue end-to-end test. No Cloudflare account resources were available for an isolated remote deployment, so no remote resources were created and no account usage or billing was incurred. The local fault-injection regression covers the relevant outbox/consumer consistency boundary, but does not claim a remote Cloudflare incident was observed.

Impact

  • User/API/contract changes: An operation with a successfully persisted command now succeeds even when the immediate Queue send response fails; delivery becomes asynchronous and observable through the durable command record.
  • Generated files / GraphQL / DB / lockfile: N/A.
  • Env or config changes: N/A.
  • Risk and rollback: A caller can receive success before Queue delivery completes, and a recovery send can duplicate an accepted message. This is intentional at-least-once behavior; consumer claim CAS prevents duplicate command execution. Revert the two PR commits to restore the prior synchronous failure behavior.

Review

  • Closest review areas: API command ledger, scheduled maintenance redrive, App deployment dispatch.
  • Known trade-offs: Scheduled redrive bounds recovery latency. The retained outbox marker prioritizes non-loss over immediate failure reporting.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

All contributors on this pull request have signed the CLA.
Posted by the CLA Assistant Lite bot.

@AsperforMias

Copy link
Copy Markdown
Collaborator Author

I have read the CLA Document and I hereby sign the CLA

@AsperforMias AsperforMias marked this pull request as draft July 13, 2026 18:41
@AsperforMias AsperforMias marked this pull request as ready for review July 13, 2026 19:14

@Yevanchen Yevanchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with full codebase context. Verified the outbox design end to end: consumer claim CAS + ack-on-claim-miss makes duplicate redrive sends safe; the marker-clear update is guarded by status='queued' plus a marker whitelist so it cannot clobber real retry errors written by releaseApiCommandForRetry; the only production reader of api_command.lastErrorCode (deployment recoverMissingDispatch) also requires attemptCount >= max, so the new pending marker cannot fail runs prematurely. Session-run latency is unaffected thanks to the inline waitUntil dispatch; with the 1-minute cron, redrive latency is bounded and the LIMIT 100 scan is covered by the (status, updated_at) index. Tests exercise the real scheduled handler and dispatch path. LGTM.

@Yevanchen Yevanchen merged commit bbfc9e5 into langgenius:main Jul 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(api): retain commands when Queue send outcome is ambiguous

2 participants