ci(spec-sdk-tests): add workflow that runs the contract suite against a local Outpost by leggetter · Pull Request #925 · hookdeck/outpost

leggetter · 2026-05-28T18:59:55Z

Closes #921.

What this workflow validates

"Does this PR's server code implement what this PR's spec says?"

Regenerates the TS SDK from this PR's docs/apis/openapi.yaml, builds outpost-server from this PR's source, runs the spec-sdk-tests contract suite against them. Catches drift introduced by the PR.

This is the workflow that would have caught the discriminator-union bug in #920 before it landed — the kind of "spec says X, server does Y" mismatch that's silent until production.

What it does not validate

	Where it lives
Drift between managed/deployed Outpost and the spec	not covered — separate concern, may be a follow-up
Newly-regen'd SDK against the latest released Outpost	#926 (PR: #927)
Older SDK releases against current server (version compat matrix)	not covered
Post-deploy smoke against managed	not covered

Conflating these into one workflow was tempting but creates problems — "always test against managed" puts "merge through red" pressure on every breaking spec change (Alex's framing), and PR-time validation against your own code answers a fundamentally different question from "has prod drifted." Separate workflows answering separate questions.

Triggers

on:
  workflow_dispatch:
  pull_request:
    paths:
      - "docs/apis/openapi.yaml"
      - "sdks/outpost-typescript/**"
      - "spec-sdk-tests/**"
      - ".speakeasy/**"
      - "sdks/schemas/**"
      - "internal/apirouter/**"
      - "internal/destregistry/**"
      - "internal/models/**"
      - "cmd/outpost/**"
      - ".github/workflows/spec-sdk-tests.yml"

Job shape

Service containers: postgres:16-alpine, redis/redis-stack-server:latest (RediSearch — tenants.list needs it), rabbitmq:3-management.
Build outpost (CLI, for migrations) and outpost-server (server) from this PR's source.
Write .outpost.yaml inline (test API key, JWT secret, the four TEST_TOPICS).
Run migrations via outpost migrate apply --yes.
Start outpost-server in singular mode (api + log + delivery in one process) in background.
Poll GET /healthz for up to 60s.
Install Speakeasy CLI + ./spec-sdk-tests/scripts/regenerate-sdk.sh TS — regen TS SDK from this PR's spec, then build. Without this we'd be testing whatever SDK src/ is checked in, which can lag the spec arbitrarily — the point of the workflow is to validate spec ↔ SDK ↔ server agree.
npm install (no lock file in spec-sdk-tests/.gitignore), write .env pointing at localhost, npm test.
On failure: dump outpost log. Always: kill background process.

Result

Dogfooded on this PR's own changes — 153 passing / 0 failing / 15 pending in 31s of test time, 3m18s total.

Decisions captured along the way

Singular-mode server (no separate api/delivery processes) — enough for contract tests, halves moving parts.
redis/redis-stack-server rather than vanilla redis — tenants.list returns 501 without RediSearch.
Regen from spec, not just build checked-in src/ — without regen, drift hides instead of surfacing. The whole reason this workflow exists.
Reuse regenerate-sdk.sh instead of inlining speakeasy run -t outpost-ts && npm run build — single source of truth between local and CI.
npm install for spec-sdk-tests — package-lock.json is gitignored there (it's a test suite, not a published package).

Test plan

Workflow YAML lints clean (python3 yaml.safe_load).
Dogfood run on this branch goes green — 153 / 0 / 15.
Follow-up tracked for the second scenario — ci: validate newly-regen'd SDKs against the latest released Outpost #926.
When fix(spec): DestinationUpdate discriminator, drop pagination overlay, event.data oneOf #924 (umbrella) merges to main, this workflow comes with it — merged at bf559988, base auto-retargeted to main.

🤖 Generated with Claude Code

Trigger: the DestinationUpdate union lost its discriminator in 8be278c (May 2025), when `type` wasn't on PATCH. Server has since added `type` acceptance and RFC 7396 merge-patch (bd7701b), making the missing discriminator an active bug — Hookdeck's permissive `config: {}` variant greedily matched any partial-config payload, the SDK sent it through with no field remap (camelCase reached the server), and merge-patch silently no-op'd. All non-Webhook partial config updates affected. Spec: `type` becomes a required, enum-locked discriminator on every DestinationUpdate* variant; 17 new `*ConfigUpdate`/`*CredentialsUpdate` companions model the now-explicit partial PATCH shape; Hookdeck's `config: {}` removed. Impact: * Breaking for typed SDK consumers: `update()` must include `type` after regen. Server is more lenient (still accepts PATCH without `type`), so raw HTTP callers keep working but are spec-noncompliant. * SDK regen left to the bot — separate PR. CI compile-check on this PR will fail until that lands; tests were validated locally against a 1.4.1 regen. spec-sdk-tests (knock-on): `type` added to 46 update sites; paginator shape fixed (supersedes fix/spec-sdk-tests-paginator-shape); SDK retries enabled; cleanup parallelized; cleanup-error logging hardened. Test status (managed Outpost, local regen'd SDK): 133→152 passing, 14→0 failing. spec-sdk-tests have no CI today; follow-up PR will add one. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(spec-sdk): fix paginator response shape (response.result.models) Speakeasy CLI 1.741.7 → 1.753.0 (commit 3f311e0, 2026-03-13) changed the generated TS SDK return type for list operations from the flat paginated body to a PageIterator wrapper: ListEventsResponse now wraps the body in `result`, so callers access `response.result.models` instead of `response.models`. The shape change was a side-effect of the Speakeasy version bump and wasn't surfaced in the SDK regen PR changelog (which only flagged the unrelated `request` query-param restyle). Tests in spec-sdk-tests/tests/{events,tenants}.test.ts still used the pre-bump accessor and failed to compile against the current SDK. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(spec-sdk): assert event.data is exposed when include=event.data Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(spec): order EventFull before EventSummary in Attempt.event oneOf Speakeasy's TS generator emits zod unions in declaration order. The non-strict EventSummary matched first and silently stripped event.data from the parsed attempt — losing the payload added by include=event.data. Declaring EventFull first lets it match when data is present, with EventSummary as the fallback. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…#919) The shared pagination-config-overlay added x-speakeasy-pagination to list endpoints, which Speakeasy then converted into PageIterator wrappers (TS), res.Next() helpers (Go), and res.next() chaining (Python). In TS the wrapper changed the response shape to response.result.models, which is a noisy regression for the common single-page case. In Go and Python the helper saves one line at the cost of hiding which request params get carried across the cursor call. Drop the overlay from all three sources for symmetric, explicit DX: callers paginate manually using res.pagination.next on the flat response. SDK regeneration is intentionally left out of this commit so the workflow change can be reviewed in isolation. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…verlay drop Trigger: #919 (merged into umbrella) drops the x-speakeasy-pagination overlay, so the TS SDK no longer wraps list responses in PageIterator. events.list()/tenants.list() return EventPaginatedResult/TenantPaginatedResult directly — accessor is `response.models`, not `response.result.models`. Tests in #920 had been adapted to the wrapper shape; this realigns them back to the flat shape post-regen. Local run after fresh TS SDK regen: 153 passing / 0 failing / 15 pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… a local Outpost Closes #921. Trigger: PRs touching the spec, TS SDK, Speakeasy config, handlers, destination providers, models, or server entry. Plus workflow_dispatch and a weekly cron as drift safety net. Job: spin up Postgres/Redis/RabbitMQ as service containers, build the outpost binary from source, run migrations, start api + delivery in background, wait for /healthz, build the TS SDK from this PR's spec state, then run `npm test` in spec-sdk-tests pointed at localhost. Notes: * Uses RabbitMQ for the message queue (default in .outpost.yaml.dev). * spec-sdk-tests/.env is written inline so we never need to commit a CI-specific .env to the repo. * Server logs are dumped on failure for debuggability. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The quoted-string form 'run: "$OUTPOST_BIN" migrate apply --yes' confuses the YAML parser (treats the leading quote as opening a scalar). Block-scalar form matches the other run: steps in the file.

…wrapper The 'outpost' binary is just a CLI wrapper that delegates 'serve' to a separate 'outpost-server' binary. Running it without a subcommand prints help and exits — which is why /healthz never came up. Switch to building both binaries: 'outpost' for migrations, then 'outpost-server' for the actual server. Drop SERVICE=api / SERVICE=delivery in favour of singular mode (empty SERVICE → api + log + delivery in one process), which is enough for the contract tests and halves the moving parts.

alexluong · 2026-05-29T08:40:57Z

@leggetter should we consider running against a production Outpost instead?

Also I think this should only run for SpeakEasy's PRs. The reason is that if we release a change in API that's breaking, this test is expected to fail. I guess we can be aware of that and merge the PRs even if CI doesn't pass, but it feels a bit off.

spec-sdk-tests/.gitignore excludes package-lock.json (intentional — the test suite isn't published, and treating SDK contract tests as production-locked dependencies adds noise to PRs without value). npm ci needs a lock file; switch to npm install.

The whole point of this workflow is to validate that spec ↔ SDK ↔ server agree. If we test against the checked-in SDK src/ (which can lag the spec arbitrarily, since regen is bot-driven), we're testing the wrong thing — drift hides instead of surfacing. Add a Speakeasy regen step before the existing build. Now any PR that changes docs/apis/openapi.yaml gets its tests run against an SDK regen'd from that PR's spec, exercising the actual contract this CI exists to validate. Uses the SPEAKEASY_API_KEY secret already configured for the publish workflows.

tenants.list requires RediSearch; vanilla redis:7-alpine returns 501 'list tenant feature is not enabled'. Swap to the official redis/redis-stack-server image which bundles RediSearch. Same port (6379), same redis-cli ping for health check.

Makes it clear at the top of the file what this workflow validates (PR's server vs PR's spec), what it does not (managed drift, SDK version compat, post-deploy smoke), and why we picked this shape over alternatives. Spares future maintainers from re-litigating the design decision.

The cron was copied from the docs-eval pattern out of habit. In this workflow it adds essentially no signal — between PRs, main's state hasn't changed, so a scheduled run re-tests what we last tested. The "deploy drift" question a cron would help with is out of scope for this workflow anyway (it's about PR's spec vs PR's server, not managed vs main).

Copilot

Pull request overview

Adds a CI contract-test workflow that builds a local Outpost server, regenerates the TypeScript SDK from the PR spec, and runs spec-sdk-tests against it. The PR also updates the OpenAPI spec and SDK test suite to support discriminated partial destination updates and validates event.data parsing.

Changes:

Added .github/workflows/spec-sdk-tests.yml for PR-triggered local contract testing.
Updated DestinationUpdate schemas with required type discriminators and partial PATCH config/credential schemas.
Adjusted spec-sdk-tests for the new PATCH shape, SDK retries, cleanup handling, and include=event.data.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`.github/workflows/spec-sdk-tests.yml`	Adds the local Outpost + regenerated SDK contract-test workflow.
`docs/apis/openapi.yaml`	Adds partial update schemas, discriminator mapping, and reorders attempt event variants.
`.speakeasy/workflow.yaml`	Removes shared pagination overlay from SDK source generation.
`sdks/schemas/pagination-config-overlay.yaml`	Deletes the Speakeasy pagination overlay.
`spec-sdk-tests/utils/sdk-client.ts`	Adds retry configuration to the SDK test client.
`spec-sdk-tests/tests/events.test.ts`	Adds coverage for `listAttempts` with `include=event.data`.
`spec-sdk-tests/tests/destinations/webhook.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/webhook-merge-patch.test.ts`	Adds update discriminators and parallelizes cleanup.
`spec-sdk-tests/tests/destinations/rabbitmq.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/hookdeck.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/gcp-pubsub.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/azure-servicebus.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/aws-sqs.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/aws-s3.test.ts`	Adds update discriminators and safer cleanup logging.
`spec-sdk-tests/tests/destinations/aws-kinesis.test.ts`	Adds update discriminators and safer cleanup logging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

leggetter · 2026-05-29T13:54:44Z

+      - "cmd/outpost/**"
+      - ".github/workflows/spec-sdk-tests.yml"


Fixed in 6e83be30 — added cmd/outpost-server/** to the path filter. Good catch.

leggetter · 2026-05-29T13:54:46Z

          oneOf:
-            - $ref: "#/components/schemas/EventSummary"
            - $ref: "#/components/schemas/EventFull"
+            - $ref: "#/components/schemas/EventSummary"


Fair point. This was introduced in #917 (now in main via #924), and works in practice because Speakeasy's TS gen uses z.union first-match dispatch — but strict OAS 3.x validators would flag the ambiguity. Out of scope for #925; tracking for follow-up.

leggetter · 2026-05-29T13:54:48Z

+      - name: Install spec-sdk-tests dependencies
+        working-directory: spec-sdk-tests
+        # spec-sdk-tests/.gitignore excludes package-lock.json, so `npm ci`
+        # doesn't work — use `npm install` instead.
+        run: npm install


Confirmed wrong path (three levels up resolves outside the repo) but harmless today: tests import via direct relative path (../../sdks/outpost-typescript/dist/commonjs) and never resolve @hookdeck/outpost-sdk, so npm install succeeded on the dogfood run (450 packages added, no error). Tracking the cleanup in #928.

leggetter · 2026-05-29T13:54:50Z

+      - name: Regenerate + build TypeScript SDK from this PR's spec
+        env:
+          SPEAKEASY_API_KEY: ${{ secrets.SPEAKEASY_API_KEY }}
+        # Same script developers run locally — keeps CI and local in sync.
+        run: ./spec-sdk-tests/scripts/regenerate-sdk.sh TS


Verified against the green dogfood run — Speakeasy CLI runs npm install --ignore-scripts --prefer-dedupe and npm rebuild bun esbuild inside speakeasy run before regenerate-sdk.sh invokes npm run build. From the log: » npm install --ignore-scripts --prefer-dedupe... then » npm rebuild bun esbuild.... So the deps are installed; no explicit step needed.

Workflow builds and runs ./cmd/outpost-server but the path filter only watched cmd/outpost/** (the CLI wrapper). Changes to the actual server entrypoint could land without triggering the contract suite. Spotted by Copilot review on #925.

leggetter and others added 7 commits May 28, 2026 12:04

ci(spec-sdk-tests): fix YAML — use block scalar for migrations run step

468756f

The quoted-string form 'run: "$OUTPOST_BIN" migrate apply --yes' confuses the YAML parser (treats the leading quote as opening a scalar). Block-scalar form matches the other run: steps in the file.

Base automatically changed from fix/spec-sdk-fixes to main May 29, 2026 08:37

leggetter added 2 commits May 29, 2026 10:06

leggetter force-pushed the ci/spec-sdk-tests branch from 9518d29 to ef7998f Compare May 29, 2026 09:15

leggetter added 2 commits May 29, 2026 10:20

leggetter mentioned this pull request May 29, 2026

ci: validate newly-regen'd SDKs against the latest released Outpost #926

Closed

leggetter mentioned this pull request May 29, 2026

ci(spec-sdk-tests-vs-release): test PR's SDK against latest released Outpost #927

Merged

3 tasks

leggetter marked this pull request as ready for review May 29, 2026 12:35

Copilot AI review requested due to automatic review settings May 29, 2026 12:35

Copilot started reviewing on behalf of leggetter May 29, 2026 12:35 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

leggetter mentioned this pull request May 29, 2026

fix(spec-sdk-tests): @hookdeck/outpost-sdk file: path and tsconfig paths are wrong (vestigial) #928

Open

leggetter merged commit 83e04e5 into main May 29, 2026
10 of 11 checks passed

leggetter deleted the ci/spec-sdk-tests branch May 29, 2026 14:32

leggetter mentioned this pull request May 29, 2026

ci(spec-sdk-tests-vs-release): pin spec-sdk-tests/ alongside SDK on sdk_version override #929

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(spec-sdk-tests): add workflow that runs the contract suite against a local Outpost#925

ci(spec-sdk-tests): add workflow that runs the contract suite against a local Outpost#925
leggetter merged 13 commits into
mainfrom
ci/spec-sdk-tests

leggetter commented May 28, 2026 •

edited

Loading

Uh oh!

alexluong commented May 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

leggetter May 29, 2026

Uh oh!

leggetter May 29, 2026

Uh oh!

leggetter May 29, 2026

Uh oh!

leggetter May 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

leggetter commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this workflow validates

What it does not validate

Triggers

Job shape

Result

Decisions captured along the way

Test plan

Uh oh!

alexluong commented May 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

leggetter May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leggetter May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leggetter May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leggetter May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leggetter commented May 28, 2026 •

edited

Loading