Skip to content

test: support a hermetic local API reference - #3542

Merged
apcha-oai merged 2 commits into
mainfrom
apcha/hermetic-api-reference
Jul 27, 2026
Merged

test: support a hermetic local API reference#3542
apcha-oai merged 2 commits into
mainfrom
apcha/hermetic-api-reference

Conversation

@apcha-oai

Copy link
Copy Markdown
Contributor

Allow the mock server to validate and use a local API reference when generation metadata is available, while preserving the existing network-backed fallback.

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thermo-nuclear code-quality review: no high-confidence maintainability findings. The 9-line change is cohesive within the existing spec-resolution flow; scripts/mock remains 61 lines and no file approaches the 1,000-line threshold. Approved.

@HAYDEN-OAI HAYDEN-OAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact head against the hermetic generation-input contract, explicit-path precedence, legacy network fallback, artifact/hash failure modes, and packaging. One actionable correctness issue prevents the real sealed local API reference from being used and can instead force a network request or abort mock startup; see the inline comment.

Comment thread scripts/mock
if [[ -n "$1" && "$1" != '--'* ]]; then
URL="$1"
shift
elif [[ -f .castiron.stats.yml && -f api_reference/openapi.transformed.yml ]]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Consume the actual sealed local API-reference contract

The producer seals the local reference at .castiron/api-definition.yaml in .castiron/generation-inputs.json with a SHA-256 digest; .castiron.stats.yml contains the nested generation-lineage marker, not openapi_transformed_spec_hash. With the actual artifact set, this condition is false, so scripts/mock and scripts/test select the network-backed .stats.yml URL and fail in a hermetic environment. If api_reference/openapi.transformed.yml also exists, awk finds no expected hash and mock startup fails before invoking Steady. Resolve the API-reference path from the canonical generation-input descriptor, verify its SHA-256 and a repo-local regular file, and use the network fallback only when the descriptor/reference pair is absent.

@apcha-oai
apcha-oai marked this pull request as ready for review July 27, 2026 17:07
@apcha-oai
apcha-oai requested a review from a team as a code owner July 27, 2026 17:07
@apcha-oai
apcha-oai merged commit 90483ad into main Jul 27, 2026
20 checks passed
@apcha-oai
apcha-oai deleted the apcha/hermetic-api-reference branch July 27, 2026 17:07
@openai-sdks

openai-sdks Bot commented Jul 27, 2026

Copy link
Copy Markdown

OkTest Summary

237/237 SDK tests passed in 10.189s for Python SDK PR #3542.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 209ms
tests/chat-completions-create.test.ts ✅ Passed 232ms
tests/chat-completions-stream.test.ts ✅ Passed 149ms
tests/files-content-binary.test.ts ✅ Passed 187ms
tests/files-create-multipart.test.ts ✅ Passed 172ms
tests/files-list-pagination.test.ts ✅ Passed 171ms
tests/initialize-config.test.ts ✅ Passed 165ms
tests/instance-isolation.test.ts ✅ Passed 154ms
tests/models-list.test.ts ✅ Passed 200ms
tests/responses-background-lifecycle.test.ts ✅ Passed 174ms
tests/responses-body-method-errors.test.ts ✅ Passed 268ms
tests/responses-cancel-timeout.test.ts ✅ Passed 192ms
tests/responses-cancel.test.ts ✅ Passed 176ms
tests/responses-compact-retries.test.ts ✅ Passed 306ms
tests/responses-compact.test.ts ✅ Passed 188ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 111ms
tests/responses-create-advanced.test.ts ✅ Passed 181ms
tests/responses-create-disconnect.test.ts ✅ Passed 1.423s
tests/responses-create-errors.test.ts ✅ Passed 178ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 264ms
tests/responses-create-retries.test.ts ✅ Passed 274ms
tests/responses-create-stream-failures.test.ts ✅ Passed 171ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 263ms
tests/responses-create-stream-wire.test.ts ✅ Passed 3.083s
tests/responses-create-stream.test.ts ✅ Passed 208ms
tests/responses-create-terminal-states.test.ts ✅ Passed 221ms
tests/responses-create-timeout.test.ts ✅ Passed 196ms
tests/responses-create.test.ts ✅ Passed 196ms
tests/responses-delete.test.ts ✅ Passed 188ms
tests/responses-input-items-errors.test.ts ✅ Passed 355ms
tests/responses-input-items-list.test.ts ✅ Passed 338ms
tests/responses-input-items-options.test.ts ✅ Passed 285ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 229ms
tests/responses-input-tokens-count.test.ts ✅ Passed 176ms
tests/responses-malformed-inputs.test.ts ✅ Passed 2.44s
tests/responses-not-found-errors.test.ts ✅ Passed 246ms
tests/responses-parse.test.ts ✅ Passed 287ms
tests/responses-retrieve-retries.test.ts ✅ Passed 172ms
tests/responses-retrieve.test.ts ✅ Passed 196ms
tests/responses-stored-method-errors.test.ts ✅ Passed 613ms
tests/retry-behavior.test.ts ✅ Passed 3.259s
tests/sdk-error-shape.test.ts ✅ Passed 322ms

View OkTest run #30287866236

SDK merge (90483adb3034) · head (6a626fc01701) · base (d4c151d92ba7) · OkTest (91635c6a2723)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants