Skip to content

feat(grpc): per-method timeout budgets and inbound deadline propagation#658

Open
Bolajiomo99 wants to merge 5 commits into
CredenceOrg:mainfrom
Bolajiomo99:main
Open

feat(grpc): per-method timeout budgets and inbound deadline propagation#658
Bolajiomo99 wants to merge 5 commits into
CredenceOrg:mainfrom
Bolajiomo99:main

Conversation

@Bolajiomo99

Copy link
Copy Markdown
Contributor

Composes caller-remaining and configured budgets in sdk/grpc/client.ts and emits DEADLINE_EXCEEDED metrics.

Closes #632

Description
This pull request adds configurable per-method timeout budgets and propagates inbound deadline information to internal gRPC calls, preventing cascading stalls from hung upstreams. The deadline composition follows a monotonic chain from the edge inward.
Previously, createCredenceGrpcClient issued calls without an explicit per-method deadline, so a hung upstream could block a caller indefinitely and there was no incoming-deadline propagation when a gRPC call was made on behalf of an HTTP request.

Changes Made

  • Extended CredenceGrpcConfig with a timeoutsMs map (per-method, with wildcard "*" default) and timeoutMetricsCollector.
  • Created createDeadlineInterceptor in src/sdk/grpc/interceptors.ts that:
  • Looks up per-method budgets by fully-qualified method name ("service.typeName/method.name"), with fallback to "*" then a built-in 10s default.
  • Reads caller-remaining deadline from the async tracingContext via GRPC_DEADLINE_REMAINING_KEY.
  • Composes budgets using effective = min(configuredBudget, callerRemaining).
  • Fast-fails when the effective timeout is <= 0 or the parent signal is already aborted.
  • Wraps timeout errors as SdkRequestTimeoutCredenceError (consistent with src/sdk/errors.generated.ts).
  • Emits a GRPC_TIMEOUT metric via TimeoutMetricsCollector from src/observability/timeoutMetrics.ts.
  • Added 'grpc' to ServiceType and 'GRPC_TIMEOUT' to TimeoutReasonCode in src/lib/timeouts.ts with conservative default budgets.
  • Exported createDeadlineInterceptor, GRPC_DEADLINE_REMAINING_KEY, GRPC_DEFAULT_TIMEOUT_MS, and isDeadlineExceededError from the SDK barrel.
  • Added 17 unit tests covering:
  • Per-method / wildcard / default budget selection.
  • Caller-remaining deadline propagation (min composition, NaN/negative/absent edge cases).
  • Fast-fail on expired budget (zero, negative, parent signal aborted).
  • Timeout firing, metric emission, no metric on success.
  • Server-side DEADLINE_EXCEEDED wrapping, passthrough for other errors.
  • Signal composition and cleanup (parent abort propagation, timer/listener cleanup).

Closes #632

Type of Change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that would cause existing functionality to not work as expected)
  • Refactoring / Performance (clean-up or performance optimization without behavioral changes)
  • Documentation / CI (changes to docs, workflows, config files)

Verification & Test Plan
Automated Tests
npm test
npm run lint
npm run build

Manual Verification

  1. Instantiate createCredenceGrpcClient with timeoutsMs including per-method entries and a "*" wildcard; confirm per-method budgets are applied.
  2. Set GRPC_DEADLINE_REMAINING_KEY in tracingContext before a gRPC call; verify the effective deadline is min(configured, callerRemaining).
  3. Configure a very short budget (e.g. 1ms) and verify SdkRequestTimeoutCredenceError is thrown with a GRPC_TIMEOUT metric recorded.
  4. Verify existing callers without timeoutsMs continue to use the legacy timeoutMs default (10s).

Notes

  • The GRPC_DEADLINE_REMAINING_KEY async context key enables inbound deadline propagation from HTTP request middleware without requiring changes to individual call sites.
  • The deadline interceptor runs first in the chain so budget enforcement happens before authentication/header injection.
  • timeoutMetricsCollector defaults to createDefaultMetricsCollector() when not provided, which is a no-op in production and uses ConsoleTimeoutMetrics in development.
  • The legacy timeoutMs config property is deprecated in favor of timeoutsMs with a "*" key.

Checklist

  • Code follows the project's style guidelines.
  • Added comprehensive unit tests.
  • Performed a self-review.
  • Added inline documentation (TSDoc).
  • No new warnings introduced.
  • All existing tests continue to pass.

Composes caller-remaining and configured budgets in sdk/grpc/client.ts and
emits DEADLINE_EXCEEDED metrics.
@drips-wave

drips-wave Bot commented Jun 29, 2026

Copy link
Copy Markdown

@Bolajiomo99 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

…eoutCredenceError constructor call

- Replaced forever-pending mock next with signal-aware version in timeout tests
- Fixed SdkRequestTimeoutCredenceError cause passed as status instead of options
- Removed unused afterEach import
- Add missing @connectrpc packages to dependencies (package.json)
- Add AuditAction mock export in replayService.test.ts
- Expand openapi-drift.js route list to match generate-openapi.ts
- Wrap MockConnectError in vi.hoisted() for vitest v4 compatibility
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add gRPC client deadline-propagation and per-method timeout budgets in sdk/grpc/client.ts

1 participant