Skip to content

[webhooks] Add dead-letter queue and alerting hook#36

Closed
extremecoder-rgb wants to merge 2 commits into
inthhq:mainfrom
extremecoder-rgb:main
Closed

[webhooks] Add dead-letter queue and alerting hook#36
extremecoder-rgb wants to merge 2 commits into
inthhq:mainfrom
extremecoder-rgb:main

Conversation

@extremecoder-rgb
Copy link
Copy Markdown

  • Add 'dead' status to NotificationDeliveryStatus after max retry attempts
  • Add listByStatus() to NotificationDeliveryAttemptsRepository for DLQ queries
  • Add GET /webhooks/dispatches endpoint with status filter
  • Add onDeadDispatchAlert callback for tenant-level alerting
  • Add CLI commands: webhooks dlq list, webhooks dlq replay

- Add 'dead' status to NotificationDeliveryStatus after max retry attempts
- Add listByStatus() to NotificationDeliveryAttemptsRepository for DLQ queries
- Add GET /webhooks/dispatches endpoint with status filter
- Add onDeadDispatchAlert callback for tenant-level alerting
- Add CLI commands: webhooks dlq list, webhooks dlq replay
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 10, 2026

Review Change Stack

Warning

Rate limit exceeded

@extremecoder-rgb has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 39 minutes and 42 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3390bb5d-28b9-4e02-a60c-140d6da7afbc

📥 Commits

Reviewing files that changed from the base of the PR and between c213827 and d8d1470.

📒 Files selected for processing (8)
  • packages/backend/src/http-api/groups/webhooks.ts
  • packages/backend/src/http-api/request-schemas.ts
  • packages/backend/src/routes/webhooks/dispatches.ts
  • packages/backend/src/services/notifications/service.ts
  • packages/cli/src/parity/route-map.ts
  • packages/internals/persistence/src/services/persistence.ts
  • packages/internals/persistence/src/types/domain.ts
  • packages/node-sdk/src/endpoints/types/requests.ts
📝 Walkthrough

Walkthrough

This PR introduces a dead-letter queue system for webhook and email dispatch tracking. It adds a new "dead" terminal status for delivery attempts that exhaust retries, implements a GET /webhooks/dispatches endpoint to query dead dispatches, extends persistence layer with status-filtered querying, adds a configurable runtime alert hook, and provides CLI parity mappings for DLQ operations.

Changes

Dead-Letter Queue Feature Implementation

Layer / File(s) Summary
Type Definitions & Contracts
packages/backend/src/events/contracts.ts, packages/internals/persistence/src/types/domain.ts, packages/backend/src/adapters/contract.ts, packages/node-sdk/src/endpoints/types/requests.ts, packages/backend/src/http-api/request-schemas.ts
Introduces DeadDispatchAlertEvent interface and extends NotificationDeliveryStatus, NotificationAttemptPayload.status, and HTTP schemas to include the "dead" terminal status.
Persistence Layer
packages/internals/persistence/src/types/domain.ts, packages/internals/persistence/src/services/persistence.ts, packages/internals/persistence/src/services/persistence/shared.ts
Adds listByStatus(status, limit?) method to repository for status-filtered querying; implements tenant-scoped query logic ordered by creation timestamp descending; updates status parser to accept "dead" as valid.
Delivery Core Logic
packages/backend/src/services/notifications/service.ts, packages/backend/src/routes/requests/shared.ts
Adds notifyDeadDispatch helper to trigger runtime alert callback; marks delivery attempts as "dead" on retry exhaustion; updates resolveNotificationStatus aggregation to include "dead" precedence after "delivered".
Runtime Configuration Hook
packages/backend/src/types/runtime.ts
Extends RuntimeConfig with optional onDeadDispatchAlert callback for host applications to subscribe to dead-dispatch events.
HTTP API Schema & Endpoint
packages/backend/src/http-api/groups/webhooks.ts
Defines WebhookDispatchSchema and declares protected GET /webhooks/dispatches endpoint with optional status and limit query parameters.
Route Handler
packages/backend/src/routes/webhooks/dispatches.ts
Implements listWebhooksDispatchesRoute handler parsing query parameters, fetching tenant-scoped delivery attempts, and mapping to response schema.
Route Registration
packages/backend/src/routes/webhooks.ts
Registers listWebhooksDispatchesRoute in exported webhookRoutes array.
CLI & Parity
packages/cli/src/commands/webhooks.ts, packages/cli/src/parity/route-map.ts
Adds webhooks_dispatches_list and webhooks_dispatches_replay CLI command identifiers and maps them to backend HTTP endpoints.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

  • [webhooks] Dead-letter queue and alerting hook #8: PR implements the core dead-letter queue feature, dead dispatch status tracking, alerting hook, and DLQ endpoint that directly address the retrieved issue's requirements for webhook dead-letter queuing and configurable alert notification.

Possibly related PRs

  • inthhq/dsar#2: Both PRs modify the notification delivery flow in packages/backend/src/services/notifications/service.ts; this PR adds dead-dispatch handling and alerting, while the related PR refactors email transport integration.

Poem

🐰 A rabbit hops through dead letters today,

With queues and alerts now tidily displayed,

Each retry exhausted marks dispatches "dead,"

The DLQ's born—no more falling thread! 📬✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title '[webhooks] Add dead-letter queue and alerting hook' clearly and specifically summarizes the main changes: introducing DLQ functionality and an alerting mechanism for webhook dispatches.
Description check ✅ Passed The description is directly related to the changeset, listing specific features added: dead status, listByStatus method, new endpoint, alerting callback, and CLI commands, all of which are present in the changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/backend/src/http-api/groups/webhooks.ts`:
- Around line 30-41: The WebhookDispatchSchema currently fixes status to
Schema.Literal("dead") but the endpoint supports multiple statuses; update the
status definition in WebhookDispatchSchema to allow the full set
["dead","failed","delivered","pending","skipped"] (e.g. replace the single
Literal with a Schema.Union of Schema.Literal(...) entries or a Schema.Enum) so
the response objects validate against the same statuses the endpoint accepts and
adjust any related usages of WebhookDispatchSchema if necessary.

In `@packages/backend/src/http-api/request-schemas.ts`:
- Line 137: NotificationEventSummarySchema.status currently omits the "dead"
literal while the attempt/accept schemas allow it, causing inconsistent response
validation; update the NotificationEventSummarySchema (the status property and
any related enums/literals) to include "dead" alongside "pending", "delivered",
"failed", and "skipped" so event-level status matches attempt-level status and
typing/validation are consistent across schemas.

In `@packages/backend/src/routes/webhooks/dispatches.ts`:
- Around line 20-23: The code uses parseInt(limitParam ?? String(DEFAULT_LIMIT),
10) to compute limit but doesn't handle parseInt returning NaN; update the logic
around limit (the parseInt usage) to first parse into a variable like
parsedLimit, check Number.isNaN(parsedLimit) and if NaN fall back to
DEFAULT_LIMIT (or 1), then apply Math.max(...)/Math.min(..., MAX_LIMIT) to clamp
it—ensure you reference the existing symbols limitParam, DEFAULT_LIMIT,
MAX_LIMIT and the variable limit when implementing the NaN check and fallback.
- Around line 30-32: The code unsafely asserts the query string statusParam to
"dead" before calling notificationDeliveryAttempts.listByStatus; validate
statusParam first (e.g., check it equals the allowed literal "dead" or one of
the permitted statuses) and only call listByStatus with a correctly-typed value,
otherwise return a 400/validation error or map unknown values to a safe default;
update the call site that references statusParam and listByStatus to use the
validated/normalized status so no type assertion is required.
- Around line 25-28: Call requireRequestTenantId as a function (use
requireRequestTenantId()) instead of referencing it, and replace the dynamic
import of RuntimeServicesTag with a static import at the top of the file, then
use Effect.service(RuntimeServicesTag) to obtain services; update the handler
code that currently uses yield* requireRequestTenantId and yield*
Effect.service(import(...).then(...)) to use yield* requireRequestTenantId() and
yield* Effect.service(RuntimeServicesTag) respectively so the dependency is
resolved correctly.

In `@packages/backend/src/services/notifications/service.ts`:
- Around line 304-314: The code is inserting a phantom delivery attempt record
by calling appendDeliveryAttempt with attempt: attempt + 1 and status "dead";
either document this choice or change it to update the last real attempt
instead: either add a concise inline comment next to the appendDeliveryAttempt
call explaining that a synthetic attempt record (attempt+1, status "dead") is
intentionally created to represent terminal state, or replace that call with
logic to update the previous attempt record (the existing appendDeliveryAttempt
call that created the last actual attempt) to set its status to "dead" and
error/responseCode accordingly so attempt numbers reflect real deliveries;
reference appendDeliveryAttempt and the last actual appendDeliveryAttempt
invocation when making the change.
- Line 42: Remove the redundant Promise.resolve wrapper around the callback
invocation: call services.config.onDeadDispatchAlert(alertEvent) directly
(instead of Promise.resolve(...)) where it's passed into Effect.tryPromise so
the raw promise/value returned by onDeadDispatchAlert is used; locate the
invocation of services.config.onDeadDispatchAlert and remove the Promise.resolve
wrapper.
- Line 308: Extract the literal "Max retry attempts exhausted" into a named
constant (e.g., MAX_RETRY_ATTEMPTS_EXHAUSTED or DEFAULT_RETRY_ERROR) and replace
the inline string in the expression result.error ?? "Max retry attempts
exhausted" with that constant; declare the constant near the top of the module
(or exported from this module if used elsewhere) so it follows project
naming/placement conventions and update any tests or imports accordingly.

In `@packages/cli/src/parity/route-map.ts`:
- Around line 107-113: The command definition for the route with id
"webhooks_dispatches_replay" mismatches the API path: the command array
currently only captures ":eventId" but the path requires both ":requestId" and
":eventId"; update the command spec in route-map (the object with id
"webhooks_dispatches_replay", method "POST", path
"/requests/:requestId/notifications/:eventId/replay") so its command parameter
list includes ":requestId" (e.g., change command from
["webhooks","dlq","replay",":eventId"] to include ":requestId" before
":eventId") ensuring the CLI can accept both requestId and eventId.

In `@packages/internals/persistence/src/services/persistence.ts`:
- Around line 940-959: listByStatus currently interpolates the caller-provided
limit directly into the SQL; clamp the limit to a safe range before using it:
add a descriptive constant (e.g., MAX_NOTIFICATION_DELIVERY_ATTEMPTS = 100 or
similar) and ensure the effectiveLimit = Math.max(1, Math.min(limit,
MAX_NOTIFICATION_DELIVERY_ATTEMPTS)); then use effectiveLimit in the SQL
interpolation in listByStatus so negative or huge values cannot be injected into
the query and the code follows the project guideline to avoid magic numbers.

In `@packages/internals/persistence/src/types/domain.ts`:
- Around line 1399-1407: The JSDoc for listByStatus is wrong: it describes
`status` as optional but the signature `readonly listByStatus: (status:
NotificationDeliveryStatus, limit?: number)` requires it; update the comment to
mark `status` as required (remove "Optional" wording and any "(e.g. "dead" for
DLQ)" placement if needed) and keep `limit` documented as optional. Locate the
comment block above the listByStatus declaration in domain.ts and make the param
descriptions consistent with the function signature (referencing listByStatus
and NotificationDeliveryStatus).

In `@packages/node-sdk/src/endpoints/types/requests.ts`:
- Line 177: The NotificationEventPayload.status union is missing the "dead"
variant so it conflicts with the attempt status; update the
NotificationEventPayload type (the status property on the
NotificationEventPayload interface/type in this file) to include "dead"
alongside "pending" | "delivered" | "failed" | "skipped" so SDK consumers accept
backend responses containing "dead" and remain consistent with the attempt
status union.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 01e2c69f-ce99-43c6-8181-42f01dae622c

📥 Commits

Reviewing files that changed from the base of the PR and between eb6e11d and c213827.

📒 Files selected for processing (15)
  • packages/backend/src/adapters/contract.ts
  • packages/backend/src/events/contracts.ts
  • packages/backend/src/http-api/groups/webhooks.ts
  • packages/backend/src/http-api/request-schemas.ts
  • packages/backend/src/routes/requests/shared.ts
  • packages/backend/src/routes/webhooks.ts
  • packages/backend/src/routes/webhooks/dispatches.ts
  • packages/backend/src/services/notifications/service.ts
  • packages/backend/src/types/runtime.ts
  • packages/cli/src/commands/webhooks.ts
  • packages/cli/src/parity/route-map.ts
  • packages/internals/persistence/src/services/persistence.ts
  • packages/internals/persistence/src/services/persistence/shared.ts
  • packages/internals/persistence/src/types/domain.ts
  • packages/node-sdk/src/endpoints/types/requests.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{ts,tsx}: Use explicit types for function parameters and return values when they enhance clarity
Prefer unknown over any when the type is genuinely unknown
Use const assertions (as const) for immutable values and literal types
Leverage TypeScript's type narrowing instead of type assertions

Files:

  • packages/cli/src/commands/webhooks.ts
  • packages/backend/src/routes/webhooks.ts
  • packages/node-sdk/src/endpoints/types/requests.ts
  • packages/backend/src/types/runtime.ts
  • packages/backend/src/events/contracts.ts
  • packages/backend/src/http-api/request-schemas.ts
  • packages/internals/persistence/src/services/persistence/shared.ts
  • packages/cli/src/parity/route-map.ts
  • packages/backend/src/adapters/contract.ts
  • packages/backend/src/routes/webhooks/dispatches.ts
  • packages/backend/src/routes/requests/shared.ts
  • packages/internals/persistence/src/services/persistence.ts
  • packages/backend/src/http-api/groups/webhooks.ts
  • packages/backend/src/services/notifications/service.ts
  • packages/internals/persistence/src/types/domain.ts
**/*.{js,jsx,ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{js,jsx,ts,tsx}: Use meaningful variable names instead of magic numbers - extract constants with descriptive names
Use arrow functions for callbacks and short functions
Prefer for...of loops over .forEach() and indexed for loops
Use optional chaining (?.) and nullish coalescing (??) for safer property access
Prefer template literals over string concatenation
Use destructuring for object and array assignments
Use const by default, let only when reassignment is needed, never var
Always await promises in async functions - don't forget to use the return value
Use async/await syntax instead of promise chains for better readability
Handle errors appropriately in async code with try-catch blocks
Don't use async functions as Promise executors
Remove console.log, debugger, and alert statements from production code
Throw Error objects with descriptive messages, not strings or other values
Use try-catch blocks meaningfully - don't catch errors just to rethrow them
Prefer early returns over nested conditionals for error cases
Extract complex conditions into well-named boolean variables
Use early returns to reduce nesting
Prefer simple conditionals over nested ternary operators
Don't use eval() or assign directly to document.cookie
Avoid spread syntax in accumulators within loops
Use top-level regex literals instead of creating them in loops
Prefer specific imports over namespace imports
Use descriptive names for functions, variables, and types instead of relying on Oxlint + Oxfmt to catch naming issues
Add comments for complex logic, but prefer self-documenting code

Files:

  • packages/cli/src/commands/webhooks.ts
  • packages/backend/src/routes/webhooks.ts
  • packages/node-sdk/src/endpoints/types/requests.ts
  • packages/backend/src/types/runtime.ts
  • packages/backend/src/events/contracts.ts
  • packages/backend/src/http-api/request-schemas.ts
  • packages/internals/persistence/src/services/persistence/shared.ts
  • packages/cli/src/parity/route-map.ts
  • packages/backend/src/adapters/contract.ts
  • packages/backend/src/routes/webhooks/dispatches.ts
  • packages/backend/src/routes/requests/shared.ts
  • packages/internals/persistence/src/services/persistence.ts
  • packages/backend/src/http-api/groups/webhooks.ts
  • packages/backend/src/services/notifications/service.ts
  • packages/internals/persistence/src/types/domain.ts
🔇 Additional comments (3)
packages/backend/src/routes/requests/shared.ts (1)

376-392: LGTM!

The status aggregation logic correctly prioritizes "delivered" > "dead" > "failed" > "skipped" > "generated", which appropriately treats "dead" as a terminal failure state after retry exhaustion.

packages/backend/src/routes/webhooks.ts (1)

5-5: LGTM!

The new dispatch listing route is properly imported and registered in the webhook routes array.

Also applies to: 15-15

packages/cli/src/commands/webhooks.ts (1)

11-12: LGTM!

The CLI command definitions for DLQ list and replay operations are correctly added.

Comment thread packages/backend/src/http-api/groups/webhooks.ts
Comment thread packages/backend/src/http-api/request-schemas.ts
Comment thread packages/backend/src/routes/webhooks/dispatches.ts Outdated
Comment thread packages/backend/src/routes/webhooks/dispatches.ts Outdated
Comment thread packages/backend/src/routes/webhooks/dispatches.ts
Comment thread packages/backend/src/services/notifications/service.ts Outdated
Comment thread packages/cli/src/parity/route-map.ts
Comment thread packages/internals/persistence/src/services/persistence.ts
Comment thread packages/internals/persistence/src/types/domain.ts Outdated
Comment thread packages/node-sdk/src/endpoints/types/requests.ts
- Fix WebhookDispatchSchema to allow all statuses
- Add 'dead' to NotificationEventSummarySchema.status
- Add NaN validation for parseInt in dispatches route
- Fix requireRequestTenantId() invocation and remove dynamic import
- Add status validation to prevent unsafe type assertion
- Remove redundant Promise.resolve wrapper
- Add RETRY_EXHAUSTED_ERROR_MESSAGE constant
- Add comment explaining phantom dead attempt record
- Fix CLI command parameters to include requestId
- Add limit clamping in listByStatus
- Fix JSDoc for listByStatus (status is required, not optional)
- Add 'dead' to NotificationEventPayload.status
@extremecoder-rgb extremecoder-rgb closed this by deleting the head repository May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant