ci: add conformance test workflow#36
ci: add conformance test workflow#36sbeashwar wants to merge 3 commits intoUniversal-Commerce-Protocol:mainfrom
Conversation
sbeashwar
requested review from
MitkoDeyanovMitev,
cusell-google,
ptiper and
richmolj
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
sbeashwar
force-pushed
the
ci-conformance-tests
branch
3 times, most recently
from
18fbdc3 to
328f2b5
Compare
Adds a CI workflow that: - Checks out conformance, samples, and SDK repos - Starts the Flower Shop reference server with test data - Runs the full conformance test suite against it - Triggers on PRs, pushes to main, and nightly schedule Results are informational (continue-on-error) so the workflow does not block merges. This surfaces test failures early without gating PRs on a suite that depends on cross-repo compatibility. Addresses Universal-Commerce-Protocol#38
sbeashwar
force-pushed
the
ci-conformance-tests
branch
from
328f2b5 to
940bcaa
Compare
|
@cusell-google gentle ping - this is a small, focused change ready for review when you have time. |
cusell-google
left a comment
cusell-google
left a comment
There was a problem hiding this comment.
Thanks again @sbeashwar for the nice, small incremental PRs!
| done | ||
|
|
||
| - name: Run conformance tests | ||
| continue-on-error: true |
There was a problem hiding this comment.
+1 on not blocking PRs, as there may be parallel changes to conformance and samples that would need to go in at the same time for this to succeed. But if I understand correctly, this will show as a green success even on failure, making it non-obvious that conformance is failing. Can you make this false for non-pr runs (e.g. the nightly scheduled ones?)
continue-on-error: ${{ github.event_name == 'pull_request' }}
There was a problem hiding this comment.
Applied your suggestion. PRs stay non-blocking, scheduled/push runs now fail loudly on test failures.
| - name: Check out SDK repo | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: Universal-Commerce-Protocol/sdk |
There was a problem hiding this comment.
this should be python-sdk (for now). https://github.com/Universal-Commerce-Protocol/python-sdk Fix the directories below too (line 65). Support for js and others may come later.
Please verify it succeeds before merging. You should be able to run this on a branch of your own fork.
There was a problem hiding this comment.
Done — updated to python-sdk on both lines 54 and 65. Will verify on my fork before this merges.
|
Verified end-to-end on my fork: https://github.com/sbeashwar/conformance/actions/runs/24915126403 All 13 test files pass (59/59). The red status on the run is from a post-job |
|
On a second thought, updated to disable post-job uv cache prune (was timing out on a lock and turning the run red even though tests passed). Latest run is fully green: https://github.com/sbeashwar/conformance/actions/runs/24915690807 |
2 participants
Summary
*_test.pyfiles and reports per-file results using GitHub log groupscontinue-on-error) - does not block mergesThis would have caught the breakage from #28 before merge (see #37).
Addresses #38
Test plan