test(polyfill): vendor the WICG polyfill test corpus as a second conformance vector#18
Merged
Conversation
…d conformance vector
Adds a parallel cross-implementation conformance net beyond the upstream
WPT corpus. The WICG urlpattern-polyfill bundles its own snapshot of
urlpatterntestdata.json (336 entries, ~85 KB); running every entry of
that snapshot against yarlpattern gives us redundant coverage of the
~328 shared cases and flags any divergence from the polyfill's
expectations on the 8 entries it carries that aren't in upstream WPT.
Implementation:
* scripts/fetch_polyfill_corpus.sh — new fetcher mirroring the
fetch_wpt_corpus.sh security model byte-for-byte:
- pinned SHA (f147a0f4..., 2025-05-07) — matches the dev-side pin
in scripts/fetch_references.sh
- HTTPS-only sparse-checkout of just test/
- filter=blob:none clone, post-fetch git rev-parse HEAD verify
- per-file size cap (10 MiB), JSON well-formedness + shape check,
--verify mode for restored caches
* tests/conftest.py — adds load_polyfill_cases() + polyfill_data_path
(overrideable via URLPATTERN_POLYFILL_DATA), parametrizes a new
polyfill_case fixture across all 336 entries. The 8 entries
where the polyfill expects a constructor error but the current
WHATWG spec does not (e.g. {hostname: 'bad#hostname'} — the
spec accepts these with Chromium-style truncation) are wrapped in
pytest.mark.skip with an explicit tracked divergence
reason. Skipping the divergences is deliberate, not failure-hiding;
the count surfaces in pytest output.
* tests/test_polyfill.py — thin shim that delegates to the existing
test_wpt_case driver. The data shape is identical, so the driver
reuses without modification.
* pyproject.toml — registers polyfill and wpt markers under
[tool.pytest.ini_options]. Suppresses the longstanding
PytestUnknownMarkWarning the WPT suite was already triggering.
* .github/workflows/ci.yml — extends the wpt-corpus job to also
fetch and cache the polyfill data. The artifact (renamed
wpt-corpus but containing both reference/wpt and
reference/polyfill) excludes both .git directories from
upload to keep the Windows digest check happy. Matrix-job
downloads switch to path: . (workspace root) so each
reference/<...> subtree lands where the harness expects it.
Test counts:
Before: 580 passed, 19 skipped
After: 908 passed, 27 skipped
↑ +328 polyfill cases passing
+8 polyfill-vs-WHATWG-spec divergences skipped
…e root upload-artifact with multiple paths under a common ancestor strips that ancestor from the archive root. reference/wpt and reference/polyfill share reference/ as the common ancestor, so the artifact internally stores files as wpt/... and polyfill/... (not reference/wpt/...). The matrix download was extracting with path: . which dropped files at workspace root → wpt/urlpattern/... instead of reference/wpt/urlpattern/.... Tests fail-fast on missing fixtures (per the v0.1 design), so all 10 matrix shards aborted at collection. Fix: extract under reference/ so the archive's wpt/... / polyfill/... lands at reference/wpt/... / reference/polyfill/... where conftest.py looks for them. test-prospective already worked because it runs from a fresh checkout (not the sdist + rm src/ flow).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a parallel cross-implementation conformance net beyond the upstream WPT corpus. The WICG urlpattern-polyfill bundles its own snapshot of
urlpatterntestdata.json(336 entries, ~85 KB); running every entry against yarlpattern gives redundant coverage of the ~328 shared cases and flags divergences from polyfill expectations on the 8 entries it carries that aren't in upstream WPT.Test counts
Changes
scripts/fetch_polyfill_corpus.shf147a0f4..., HTTPS-only sparse-checkout oftest/, post-fetch SHA verify, per-file size cap, JSON shape check,--verifymodetests/conftest.pyload_polyfill_cases(),polyfill_data_path(env-overrideable viaURLPATTERN_POLYFILL_DATA), parametrizes a newpolyfill_casefixture. The 8 polyfill-vs-WHATWG-spec divergences getpytest.mark.skip(reason="tracked divergence")tests/test_polyfill.pytest_wpt_casedriver. Same data shape, no duplicated logicpyproject.tomlwptandpolyfillmarkers, suppressing the longstandingPytestUnknownMarkWarning.github/workflows/ci.ymlwpt-corpusjob to also fetch + cache polyfill. Artifact contains bothreference/wpt/andreference/polyfill/. Matrix downloads switch to workspace-root extractTest plan
just lintclean (11 / 11)pytest tests/test_polyfill.py→ 328 passed, 8 skipped (the tracked divergences)scripts/fetch_polyfill_corpus.shsmoke-tested locally (fresh clone +--verify)actionlint+zizmorclean on the workflow update