ci(spec-sdk-tests-vs-release): test PR's SDK against latest released Outpost#927
Conversation
…Outpost Closes #926. Trigger: PRs touching sdks/outpost-typescript/** (where the Speakeasy bot regen PRs land). Resolves the latest non-prerelease Outpost tag dynamically via the GitHub releases API (the repo uses namespaced tags like sdks/outpost-typescript/v1.3.0 for SDK releases, so the bare vX.Y.Z pattern correctly picks out the Outpost release). Question this answers: "Will the newly-regen'd SDK in this PR work against the version of Outpost that customers are already running?" Distinct from the existing spec-sdk-tests.yml workflow which asks "does this PR's spec match this PR's server" — both are needed, neither subsumes the other. Job shape: pull hookdeck/outpost:<tag> as a docker image, run it alongside the same service containers as the sibling workflow (Postgres, redis-stack-server for RediSearch, RabbitMQ), build the SDK from the PR with no regen step (the regen IS the PR), run the contract suite. Not dogfooded on this PR — the trigger filter only matches SDK paths, which this PR doesn't touch. First real run will be on the next Speakeasy bot regen PR after this lands.
…dispatch overrides
Lets you trigger the workflow from the Actions UI with optional inputs
for ad-hoc compat testing:
sdk_version pins the SDK to a specific release tag (or uses the
dispatch branch's contents if empty).
outpost_version pins the server to a specific Outpost release (or
resolves the latest non-prerelease release if empty).
Both accept "1.3.0" or "v1.3.0" — leading "v" is normalized.
Inputs only affect workflow_dispatch runs; pull_request triggers
ignore them, so the gate behaviour for bot regen PRs is unchanged.
Single workflow rather than a sibling file — the job body is ~95%
identical between PR gate and compat testing; the only material
differences are two variables (which SDK, which Outpost).
There was a problem hiding this comment.
Pull request overview
Adds a new CI workflow to run the spec-sdk-tests contract suite using the SDK as committed in the PR (or optionally a pinned SDK tag for manual runs) against the latest released Outpost Docker image, closing the “SDK regen PR vs released server” validation gap described in #926.
Changes:
- Introduces
.github/workflows/spec-sdk-tests-vs-release.ymlto spin up service containers, pull the latest Outpost release image, run migrations, start Outpost, build the PR’s TypeScript SDK, and executespec-sdk-tests. - Adds optional
workflow_dispatchinputs (sdk_version,outpost_version) to pin SDK/server versions for ad-hoc compatibility testing.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| OVERRIDE: ${{ inputs.outpost_version }} |
There was a problem hiding this comment.
Fair concern. Per GH Actions docs, inputs.* is only populated on workflow_dispatch (and workflow_call). Practically, accessing it on PR events returns null/empty rather than erroring — so the existing code would have worked — but the explicit guard is unambiguous and costs nothing. Fixed in 1f0b884f using the short-circuit ternary: OVERRIDE: \${{ github.event_name == 'workflow_dispatch' && inputs.outpost_version || '' }}.
| - name: Override SDK with sdk_version (workflow_dispatch only) | ||
| if: inputs.sdk_version != '' | ||
| # Replace the working tree's sdks/outpost-typescript with the contents |
There was a problem hiding this comment.
Fixed in 1f0b884f — if: now prepends github.event_name == 'workflow_dispatch', so the inputs.sdk_version check is short-circuited away on PR runs.
| env: | ||
| SDK_VERSION_INPUT: ${{ inputs.sdk_version }} | ||
| run: | |
There was a problem hiding this comment.
The step's if: is now gated on github.event_name == 'workflow_dispatch' (1f0b884f), so this step only runs on dispatch. Step-level env: is evaluated when the step actually runs, so inputs.sdk_version here is never read on PR events. Kept the modern inputs.* rather than reverting to github.event.inputs.* — GH explicitly recommends inputs.* for new workflows.
…w_dispatch event check Defensive pattern flagged by Copilot review on #927: inputs.* context is officially only populated on workflow_dispatch (and workflow_call). Practically this works on PR events too — inputs.x evaluates to null which compares as empty — but the explicit guard is unambiguous and costs almost nothing. Two changes: * OVERRIDE env in the tag resolver uses the short-circuit ternary (github.event_name == 'workflow_dispatch' && inputs.x || ''). * SDK override step's if: prepends event_name == 'workflow_dispatch' so the inputs.sdk_version check is only evaluated on dispatch runs.
PRs that touch only this workflow file would fire it against main's state — currently NEW tests + OLD SDK (regen still pending) + OLD released Outpost — and fail at TS compile with 'type does not exist in type DestinationUpdate'. That's predicted transitional-state noise, not a real bug, but it leaves a permanently-red dogfood result that future reviewers have to recognize as expected. Drop the workflow file from its own trigger paths. The actual scenario this workflow exists for — Speakeasy bot regen PRs — always touches sdks/outpost-typescript/**, so the gate still catches them. Local iteration on the workflow file itself uses 'gh workflow run --ref'. Spotted while inspecting failing PR runs on #927.
Closes #926. Companion to #925 (PR-vs-itself); together they cover the two scenarios from the design discussion:
go build ./cmd/outpost-serverfrom PRsdks/outpost-typescript/**hookdeck/outpost:<latest-release>docker imageWhy this exists
Release order: Outpost is tagged →
sdk-generate-on-releasefires → bot opens an SDK PR. At step 3, nothing exercises the newly-regen'd SDK against a running server before publish. If the regen produces a SDK that silently misroutes requests (the discriminator-union bug we hit in #920 is the canonical example), it ships and we hear from a customer.This workflow closes that gap. On every bot regen PR, the contract suite runs with that PR's SDK against the released server it'll be paired with.
Key implementation choices
gh release viewreturnssdks/outpost-typescript/v1.3.0instead ofv1.0.3— I caught that during verification, see the explicitgh api ... test("^v[0-9]+\\.[0-9]+\\.[0-9]+$")filter in the workflow.speakeasy runwould defeat the workflow's purpose.go build. We want exactly what customers run, not an approximation.--network hoston the outpost container so it reaches the service containers onlocalhost:5432/:6379/:5672.Manual runs from the Actions UI
workflow_dispatchaccepts two optional inputs for ad-hoc compat testing:sdk_versionsdks/outpost-typescript/v<x>). Replacessdks/outpost-typescript/working-tree contents from that tag.outpost_versionBoth accept
1.3.0orv1.3.0— leadingvis normalized.Inputs only affect manual runs. Pull-request triggers ignore them entirely, so the PR-gate behaviour for bot regen PRs is unchanged. Same workflow rather than a sibling file because the job body is ~95% identical — overrides are two variables, not a different shape.
Note on the historical failed runs
Earlier pushes to this branch (
587f2c7c,c4b22455,1f0b884f) triggered the workflow when the file path itself was still in the trigger filter. Those runs failed at TS compile (error TS2353: 'type' does not exist in type 'DestinationUpdate') — expected transitional-state noise:maincurrently has the new tests (post-#924) + old committed SDK (regen still pending) + old released Outpost server. The workflow's intended scenario — bot regen PR with new SDK + old released server — aligns and would pass.From
f8b8740eonward, the workflow file is excluded from its own trigger paths, so further edits won't accumulate red runs on this PR. The visible red checks are historical and won't update.Not dogfooded on this PR
The trigger filter is
sdks/outpost-typescript/**+spec-sdk-tests/**, which this PR doesn't touch. First real run will be on the next Speakeasy bot regen PR.Test plan
python3 yaml.safe_load).gh apitag resolution returnsv1.0.3(currently the latest release) — verified locally; will reconfirm in the live run.🤖 Generated with Claude Code