Skip to content

feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071)#230

Open
nijeesh-stream wants to merge 15 commits into
masterfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads
Open

feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071)#230
nijeesh-stream wants to merge 15 commits into
masterfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads

Conversation

@nijeesh-stream
Copy link
Copy Markdown
Contributor

@nijeesh-stream nijeesh-stream commented May 7, 2026

Summary

Adds first-class support for gzip-compressed webhook payloads (HTTP webhooks, SQS, SNS) and exposes a stable verify_and_parse_* API that mirrors the cross-SDK contract published in Webhooks Overview.

New public API (stream_chat/webhook.py)

Module-level primitives:

  • gunzip_payload(body) -> bytes — gzip-magic-byte detection, no-op when not compressed
  • decode_sqs_payload(body) -> bytes — base64-decode then gunzip-if-magic
  • decode_sns_payload(notification_body) -> bytes — JSON-parse the SNS HTTP notification envelope, extract the inner Message, then run the SQS pipeline. Falls through to a pre-extracted Message string when the input is not a JSON envelope
  • verify_signature(body, signature, secret) -> bool — HMAC-SHA256 over the uncompressed body, with a constant-time comparison (matters for the HTTP webhook path where the X-Signature header is exposed publicly; SQS / SNS deliveries arrive over AWS-internal transports where timing-attack resistance is not strictly required)
  • parse_event(payload) -> dict — JSON → dict

Module-level composites (return dict):

  • verify_and_parse_webhook(body, signature, secret) -> dict
  • verify_and_parse_sqs(body, signature, secret) -> dict
  • verify_and_parse_sns(body, signature, secret) -> dict

StreamChat/StreamChatAsync instances expose verify_and_parse_webhook / verify_and_parse_sqs / verify_and_parse_sns that use the configured api_secret automatically.

Typed Event objects will land in Python in a follow-up release. Until then the helpers return the parsed JSON as a dict.

Backwards compatibility

StreamChat#verify_webhook is preserved for plain (uncompressed) bodies. The experimental decompress_webhook_body and verify_and_decode_webhook surfaces are removed (they were never released).

Unified error handling

Per cross-SDK coordination (mogita's review across the 6 sibling SDK PRs), every webhook failure path now terminates at a single exception class — InvalidWebhookError so customers only need one except arm. The message text identifies which failure mode fired so callers that want to differentiate (security logging, retry policy) can filter on substring:

Failure mode Message
Signature mismatch signature mismatch
Base64 decode invalid base64 encoding
Gzip decompression gzip decompression failed
JSON parse invalid JSON payload

Module-level constants (INVALID_WEBHOOK_SIGNATURE_MISMATCH / INVALID_WEBHOOK_INVALID_BASE64 / INVALID_WEBHOOK_GZIP_FAILED / INVALID_WEBHOOK_INVALID_JSON) are exported for callers that prefer exact-match filtering. The legacy StreamChat#verify_webhook (bool return) is untouched.

Tests

stream_chat/tests/test_webhook_compression.py covers plain / gzip / base64 / base64+gzip payloads, signature mismatches, malformed bytes, JSON parsing into dict, and the StreamChat / StreamChatAsync client-bound paths. Linked Linear ticket: CHA-3071.

Golden test fixtures (Tommaso)

Added shared reference fixtures to the test suite so future SDKs can sanity-check decoders against the same payloads:

aGVsbG93b3JsZA==                          -> helloworld   (base64)
H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64 + gzip)

Test plan

  • pytest stream_chat/tests/test_webhook_compression.py — 59 passed (was 57)
  • black / isort — clean

… (CHA-3071)

Stream Chat backend can now compress outbound webhook payloads with gzip
and, for SQS / SNS firehose delivery, base64-wrap the compressed bytes so
they remain valid UTF-8 over the queue. Add two new client methods that
let customers decompress + verify in a single call:

- decompress_webhook_body(body, content_encoding=None, payload_encoding=None)
  primitive decode that handles gzip and/or base64
- verify_and_decode_webhook(body, x_signature, content_encoding=None,
  payload_encoding=None) decode + HMAC-SHA256 verify

Both are exposed on the sync StreamChat and async StreamChatAsync clients
through the shared StreamChatInterface base, mirroring the existing
verify_webhook helper. The existing verify_webhook signature and behavior
are unchanged for backward compatibility.

A new WebhookSignatureError (extends StreamAPIException) is raised on
signature mismatch, malformed gzip, or malformed base64. Unsupported
encoding values raise ValueError with a message that points at the
supported algorithm (gzip).

The decoding logic lives in stream_chat/webhook.py so it can be tested
without instantiating an HTTP client. The new tests cover the cross-SDK
contract: passthrough, gzip round-trip, base64 round-trip, base64 + gzip
(SQS / SNS shape), case-insensitive aliases, every unsupported
content_encoding (br / brotli / zstd / deflate / compress / lz4),
unsupported payload_encoding (hex / url / binary), invalid gzip / base64
input, and three signature-mismatch variants (wrong signature, signature
over compressed bytes, signature over wrapped bytes).

Docs: webhooks_overview.md gets a "Compressed webhook bodies" section
with Django, Flask, and SQS / SNS usage examples.

Co-authored-by: Cursor <cursoragent@cursor.com>
nijeesh-stream and others added 2 commits May 7, 2026 12:37
Co-authored-by: Cursor <cursoragent@cursor.com>
Replaces the earlier verify_and_decode_webhook surface with the
cross-SDK contract documented at
https://getstream.io/chat/docs/node/webhooks_overview/.

Module-level helpers in stream_chat.webhook:

  Primitives:
    ungzip_payload       - gzip magic-byte detection + inflate
    decode_sqs_payload   - base64 then ungzip-if-magic
    decode_sns_payload   - alias for decode_sqs_payload
    verify_signature     - constant-time HMAC-SHA256 comparison
    parse_event          - JSON -> dict (typed event lands later)

  Composite (return parsed event dict):
    verify_and_parse_webhook
    verify_and_parse_sqs
    verify_and_parse_sns

The composite functions auto-detect compression from body bytes, so
the same handler stays correct whether or not Stream is currently
compressing payloads, and behind middleware that auto-decompresses.

Client instance methods (StreamChat / StreamChatAsync) mirror the
three composite helpers with api_secret pulled from the client.

The legacy verify_webhook(body, x_signature) -> bool boolean helper
is unchanged for backward compatibility.

Co-authored-by: Cursor <cursoragent@cursor.com>
@nijeesh-stream nijeesh-stream changed the title feat(webhooks): add verify_and_decode_webhook for compressed payloads feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071) May 8, 2026
nijeesh-stream and others added 2 commits May 8, 2026 16:24
binascii.Error is a subclass of ValueError, so listing both in the
except clause triggers flake8-bugbear B014. Catching ValueError alone
covers both cases.

Co-authored-by: Cursor <cursoragent@cursor.com>
RFC 1952 defines the gzip magic number as the two-byte sequence
1F 8B; the third byte (CM) is informational and not part of the
identifier. Trim the magic check from three bytes to two to match
the spec and stay consistent with the reference implementations
in the public docs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread docs/webhooks/webhooks_overview/webhooks_overview.md Outdated
Comment thread docs/webhooks/webhooks_overview/webhooks_overview.md Outdated
nijeesh-stream and others added 10 commits May 11, 2026 11:16
Previously, passing a signature with non-ASCII bytes (e.g. b"\xff..."),
a non-ASCII unicode string, or a non-string type would raise
UnicodeDecodeError / TypeError from inside verify_signature, leaking
through verify_and_parse_webhook / _sqs / _sns and breaking the
documented contract that says malformed inputs must surface as
WebhookSignatureError.

The boolean primitive now returns False for those inputs (an
invalid-format signature can by definition never match), so the
composite helpers raise WebhookSignatureError("invalid webhook
signature") as expected. The constant-time HMAC comparison path is
unchanged for well-formed inputs.

Adds regression tests for non-ASCII bytes, non-ASCII str, and
non-string signature inputs at both the primitive and composite
layers.

Co-authored-by: Cursor <cursoragent@cursor.com>
The previous draft referenced helpers that were renamed during the
refactor to the verify_and_parse_* contract (CHA-3071):

  - client.verify_and_decode_webhook(...)  -> verify_and_parse_webhook
  - decompress_webhook_body(...)           -> removed (no public form)
  - content_encoding / payload_encoding    -> removed (magic-byte detect)

Following the old snippets would hit AttributeError immediately. The
section is rewritten to document the real surface area:

  - client.verify_and_parse_webhook(body, signature)
  - client.verify_and_parse_sqs(message_body, signature)
  - client.verify_and_parse_sns(message, signature)
  - module-level webhook.verify_and_parse_* helpers for stateless use
  - WebhookSignatureError as the single error class

It also clarifies the return type (parsed dict, not raw bytes) and
notes that the legacy verify_webhook bool helper stays unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
decode_sns_payload now JSON-parses the SNS HTTP notification envelope
({"Type":"Notification","Message":"..."}) and extracts the inner
Message field before running the SQS pipeline. Falls through to the
pre-extracted Message string when the input is not a JSON envelope so
existing call sites keep working.

Test adds a realistic SNS HTTP notification body fixture and exercises
both the new envelope path and the existing pre-extracted Message path.
Docs updated to show the typical "pass the raw HTTP body" call site.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…den fixtures (CHA-3071)

Per Tommaso's suggestion, align the gzip helper with the GNU `gunzip`
command name. The function was added in this PR and not yet released,
so this is a straight rename with no back-compat alias.

Adds Tommaso's reference fixtures to the test suite as named cases so
future SDKs can sanity-check against the same payloads:

  aGVsbG93b3JsZA==                          -> helloworld   (base64)
  H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64+gzip)

Co-authored-by: Cursor <cursoragent@cursor.com>
…HA-3071)

Per cross-SDK coordination (mogita's review on the 6 sibling SDK PRs),
every webhook failure path now terminates at a single exception class.
Customers only need one except arm and can filter by message text for
mode-specific behaviour (signature mismatch vs invalid base64 etc.).

Renames the previously-unreleased WebhookSignatureError to
InvalidWebhookError and threads it through every primitive:

  verify_signature      -> 'signature mismatch'
  gunzip_payload        -> 'gzip decompression failed'
  decode_sqs_payload    -> 'invalid base64 encoding'
  parse_event           -> 'invalid JSON payload'

StreamChat#verify_webhook (the legacy bool helper) is untouched. The
message constants are exported so callers can exact-match if they
prefer that over substring matching.

Co-authored-by: Cursor <cursoragent@cursor.com>
…CHA-3071)

Stream does not ship an X-Signature on SQS or SNS deliveries — those
transports ride AWS-internal infrastructure (IAM-authenticated queues
and AWS-signed SNS notifications), so HMAC verification on top is
theatre. signature + secret are now optional on both module helpers
and on the StreamChat / StreamChatAsync instance methods.

- verify_and_parse_sqs(body)                -> decode + parse
- verify_and_parse_sqs(body, sig, secret)   -> decode + verify + parse
- verify_and_parse_sns(envelope_body)       -> unwrap + decode + parse
- verify_and_parse_sns(envelope_body, sig, secret) -> + verify

Passing only one of (signature, secret) raises InvalidWebhookError.
The HTTP-webhook path (verify_and_parse_webhook) is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ndParseWebhook; docs + tests

Co-authored-by: Cursor <cursoragent@cursor.com>
…bhookSignatureError + parse_*; Python WebhookSignatureError; guard test init without STREAM_*
…ip_payload

- Rename WebhookSignatureError → InvalidWebhookError
- Rename ungzip_payload → gunzip_payload
- Align error messages to documented strings: signature mismatch /
  invalid base64 encoding / gzip decompression failed / invalid JSON payload
- parse_event now wraps json.JSONDecodeError as InvalidWebhookError
- Export INVALID_WEBHOOK_* constants for exact-match filtering
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants