feat: enable Speakeasy contract testing + exhaustive SDK test suite#379
feat: enable Speakeasy contract testing + exhaustive SDK test suite#379Nelson-PROIA wants to merge 2 commits intomainfrom
Conversation
- Enable `generateTests: true` and `testing: enabled: true` in all gen.yaml/workflow.yaml
- Bootstrap arazzo test definitions and generated pytest files for main, Azure, and GCP SDKs
- Add `.genignore` entries to protect hand-written tests from being overwritten
- Add `pytest-xdist` dev dependency for parallel test execution
- Add GitHub Actions workflow for contract tests on PRs
- Disable `persistentEdits` on main SDK (workaround for mockserver incompatibility)
- Fix `prompt_audio_seconds: {}` → `null` in agents arazzo tests (OpenAPI spec bug workaround)
Results: main SDK 230/238 pass (96.6%), Azure 2/3, GCP 0/2 (blocked by path rewriting hook)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nelson-PROIA
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| # _build_url produces the HTTPS URL; the WSS conversion happens in connect(). | ||
| # Verify the URL is HTTPS-based (connect() replaces https -> wss) | ||
| assert url.startswith("https://api.mistral.ai") |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Copilot Autofix
AI 3 days ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
| query_params={}, | ||
| ) | ||
|
|
||
| assert url.startswith("https://custom.server.com") |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Copilot Autofix
AI 3 days ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
| content = request.content.decode("utf-8", errors="replace") | ||
| assert "mistral-large-latest" in content | ||
| # The URL may be URL-encoded in form data | ||
| assert "example.com" in content |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Copilot Autofix
AI 3 days ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
|
❌ SDK Test Report: all — tests failed |
1 participant
Summary
scripts/run_all_tests.shfor running complete test suiteTest Coverage
Changes
tests/unit/- 31 new unit test files covering all SDK endpointspackages/azure/tests/unit/- Azure-specific unit testspackages/gcp/tests/unit/- GCP-specific unit teststests/test_integration.py- Main SDK integration testsscripts/run_all_tests.sh- Script to run all test suites.github/workflows/test_custom_code.yaml- Updated CI to run all testspyproject.toml- Added pytest filterwarnings.genignorefiles - Updated to protect custom test codeTest plan