Feat/add webhook support pool events#50
Open
mariocodecr wants to merge 10 commits into
Open
Conversation
Introduce the foundational primitives the webhook subsystem will build on: the catalogue of pool/price event types (with wildcard support) and a timing-safe HMAC-SHA256 sign/verify helper plus secret generator.
Model the future PostgreSQL schema for webhooks and webhook_deliveries behind a repository abstraction backed by Redis today. The SQL DDL is documented inline so migrating to Postgres only requires swapping the repository implementation. Includes a reusable in-memory cache mock so the existing Jest pattern keeps working across the new test files.
The dispatcher fans an event out to every active webhook subscribed to its type, persists a delivery record per webhook, signs the body with HMAC-SHA256, and retries failed deliveries up to WEBHOOK_MAX_ATTEMPTS. Retry decisions follow standard webhook semantics: network errors and 5xx/408/429 responses are retried with exponential backoff, while other 4xx responses are marked failed immediately so a misconfigured consumer cannot be retried into the ground. Retries are queued in a Redis ZSET so they survive process restarts. All webhook subsystem config knobs are introduced here so callers in later commits can rely on them.
Poll the Redis ZSET of due delivery retries on a configurable interval and re-invoke the dispatcher for each one. A simple running flag avoids overlapping ticks if a previous batch is still in flight.
Generic per-IP limiter built on INCR + EXPIRE so it can be reused by any route by tagging it with a keyPrefix. The middleware fails open if Redis is unreachable so a cache outage cannot lock users out of management endpoints, and surfaces standard X-RateLimit-* headers on every response.
Add CRUD endpoints for webhook registration, a dedicated test-delivery endpoint (rate-limited tighter than mgmt to prevent abuse as an outbound HTTP cannon) and a deliveries listing endpoint that powers the admin dashboard view of webhook health. The secret is returned in plaintext only on creation and never echoed back from list/get; subsequent reads only expose a short preview.
Wire the webhook subsystem into the Express bootstrap and shut the retry worker down cleanly on SIGTERM/SIGINT alongside the existing price job.
…semantics Add a full Webhooks section to the README: supported event types, the public REST API surface, the outgoing request shape (headers + body), a Node.js HMAC verification example, retry/backoff behavior, the repository-pattern note explaining the PG migration path, rate-limit defaults and the new env vars.
Contributor
|
resolve merge conflicts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request feat: webhook subsystem for pool events (HMAC, retries, rate limiting)
Summary
Add a complete webhook subsystem so projects can register HTTP endpoints that receive notifications when farming/pool
events are indexed. Includes HMAC-SHA256 signing, exponential-backoff retries with a persistent queue, delivery logs for
an admin dashboard, event-type subscriptions, a synthetic test endpoint, and per-IP rate limiting.
Closes #2
Architectural decision: Postgres vs. Redis
The issue spec calls for a
webhookstable in PostgreSQL. The project currently runs entirely on Redis (alerts, cache,jobs). Adding Postgres in this PR would mean introducing the
pgdriver, a migration tool, Docker/CI updates, and aschema-bootstrap path — a platform-level change that warrants its own decision and its own PR.
What this PR does instead:
src/repositories/) with the exact PostgreSQL schema the spec asks fordocumented as DDL in each repo's header.
services/alerts.js.src/repositories/change — the dispatcher,routes, worker, tests, and callers remain untouched.
If reviewers prefer to land Postgres now, I'm happy to follow up with a second PR that swaps the implementation; the
contract is already defined.
What's included (mapped to acceptance criteria)
webhookstable (id, url, events[], secret, active, …)src/repositories/webhookRepository.js(DDL in header)POST /api/v1/webhookssrc/routes/webhooks.jsGET/PATCH/DELETE /api/v1/webhooks/:idsrc/routes/webhooks.jssrc/services/webhookDispatcher.js→dispatch({event_type, event_id, data})src/services/webhookSignature.js(timing-safe verify)src/services/webhookDispatcher.js+ Redis ZSET queuesrc/jobs/webhookRetryWorker.jssrc/repositories/deliveryRepository.js(DDL insrc/services/webhookEvents.js+listActiveForEventsrc/middleware/rateLimit.js(Redis fixed window, fail-open)POST /api/v1/webhooks/:id/testGET /api/v1/webhooks/:id/deliveriesREADME.md(Webhooks section)Retry semantics
5xx,408,429.4xx(consumer rejected — retrying won't help, marksfailedimmediately).WEBHOOK_RETRY_BASE_MS * WEBHOOK_RETRY_FACTOR ^ (attempts-1)(defaults: 30s → 60s → 120s).WEBHOOK_MAX_ATTEMPTStotal attempts (default 3). Survives process restarts via Redis ZSET.Security
secret_previewonly.
crypto.timingSafeEqualand length-checks before comparing./testendpoint is independently rate-limited (5 req/min/IP by default) to prevent abuse as an outbound HTTP cannon.New env vars (all optional, sane defaults)
WEBHOOK_MAX_ATTEMPTS3WEBHOOK_RETRY_BASE_MS30000WEBHOOK_RETRY_FACTOR2WEBHOOK_TIMEOUT_MS5000WEBHOOK_RETRY_POLL_MS5000WEBHOOK_RETRY_BATCH25WEBHOOK_RATELIMIT_WINDOW/_MAX60s/60WEBHOOK_TEST_RATELIMIT_WINDOW/_MAX60s/5/testrate limitTest plan
Automated (
npm test): 104 tests / 9 suites passing, including the pre-existingalerts,cors, andresiliencesuites (no regressions).
npm install && npm test— all 9 suites passnpm run devboots without errors and logs the retry worker startingcurl -X POST http://localhost:3000/api/v1/webhooks -H 'Content-Type: application/json' -d '{"url":"https://webhook.site/<id>","events":["pool.assets_locked"]}'returns 201 withsecretandidcurl http://localhost:3000/api/v1/webhookslists the webhook withsecret_preview(notsecret)curl -X POST http://localhost:3000/api/v1/webhooks/<id>/testproduces a delivery on webhook.site withX-SmartDrop-Signature: sha256=…/test— confirm delivery row goes topendingwithnext_retry_atset, and that the retry worker re-processes itcurl http://localhost:3000/api/v1/webhooks/<id>/deliveriesshows the delivery historyPOST /api/v1/webhooks/<id>/testmore than 5x/min — confirm429withX-RateLimit-*headersPATCHthe webhook to{"active": false}and confirmdispatchno longer routes to it