feat(fidelity): publish the cross-check as a living fidelity report (+ corpus 56→75, 3 mock fixes)#59
Merged
Merged
Conversation
Turns the behavioral cross-check from an internal CI gate into a published, self-updating proof of mock-vs-real-React-Native parity. - crosscheck.mjs writes an ephemeral combined report (RN version, per-probe parity) to crosscheck/.out/report.json. - New scripts/fidelity-report.mjs renders two committed artifacts from it: a shields.io endpoint badge (live probe count) and website/guide/fidelity.md (the full corpus table + a known-differences ledger). The published count is derived from the corpus, never hand-typed, so it cannot drift as probes grow; the CI version range is read from the matrix workflow for the same reason. - fidelity:check (--check) re-runs the cross-check and fails if the committed page/badge no longer match the live corpus; wired into the CI gate so the published proof can never silently go stale. - Adds a README fidelity badge, a Reference > Fidelity Report nav entry, and links the existing cross-check prose to the new page. The known-differences ledger states divergences honestly (device-metric defaults) rather than hiding them — trust is built on precision, not claims.
Adds 12 probes, all verified matching between the mock engine and real React Native: processColor (named/hex/rgba), I18nManager.isRTL, PixelRatio rounding helpers, typeof Platform.Version, partial Platform.select, StyleSheet.flatten over falsy entries, queryByTestId miss, getByText throw-on-miss, toContainElement, not-on-screen-after-unmount, accessibilityLabel read, and Text numberOfLines. Each new probe both widens verified fidelity coverage and grows the published count. The fidelity report and badge regenerate from the corpus automatically.
A bug-hunting expansion of the cross-check corpus surfaced three behaviors where
the mock engine diverged from real React Native:
- Pressable `style={({pressed}) => ...}` and `children={({pressed}) => ...}`
were passed through untouched. Real RN resolves them against its press state,
so the mock now tracks `pressed` (toggled on press-in/out) and resolves both,
matching real RN's resting render and updating under press.
- processColor() returned opaque black for an unparseable color; real RN returns
undefined (normalizeColor fails). A 'conformance' test asserting the old black
fallback was itself non-conforming and is corrected to match real RN.
Adds 7 cross-check probes covering these plus Pressable disabled a11y state,
fireEvent.press, press-in/out, and TextInput maxLength (corpus 68 → 75, all
matching). Switch valueChange, Text-onPress role, and Appearance color scheme
were probed but left ungated as version-/environment-variant.
Regenerates the fidelity badge and page (75/75).
- CRITICAL: fidelity:check now propagates a non-zero cross-check exit even when every probe matched. crosscheck.mjs fails when a suite exits non-zero despite matching probes (the !native.ok guard); the wrapper previously only warned and exited on allMatch, so that class of failure could pass CI with a green badge. Now fidelity:check is a strict superset of crosscheck. Verified by simulating a suite that exits non-zero with all probes matching (gate now fails). - MAJOR: the known-differences ledger contradicted the live corpus — it called Dimensions/PixelRatio a divergence while those probes are gated as matching, and claimed entries were excluded from the corpus. Reframed the section as 'Not gated by the cross-check' and rewrote the ledger to describe what is deliberately ungated and why (device-, version-, or environment-variant), which is accurate and no longer self-contradictory. - MINOR: prepublishOnly now runs fidelity:check too, so publish and per-commit CI use the same strict entry point.
Documents the two user-facing mock behavior fixes (Pressable function style/children, processColor undefined for unparseable input) and the new fidelity report so they appear in the changelog and trigger a patch release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the behavioral cross-check from an internal CI gate into a published, self-updating proof of mock-vs-real-React-Native fidelity — and, in the process, expands the corpus and fixes three real divergences it surfaced.
The cross-check has always been the trust mechanism for the
mockengine (the same probes run under the mock and under real RN; any divergence fails CI). This makes that proof visible and impossible to doubt, then puts it to work.What's included
1. A living fidelity report (
feat).scripts/fidelity-report.mjsrenders two committed artifacts from the cross-check corpus: a shields.io endpoint badge (live probe count) and a publishedwebsite/guide/fidelity.mdpage (full corpus table + an honest known-differences ledger).fidelity:check(--check) re-runs the cross-check and fails if the committed page/badge no longer match the live corpus; it replaces the cross-check step in the CI gate, so the published proof can never silently go stale.2. Corpus expansion 56 → 75 probes.
Platform.Versiontype, partialPlatform.select, falsy-flatten, query misses,getByTextthrow,toContainElement, unmount, a11y label,numberOfLines) — all matching.valueChange, Text-onPressaccessibilityRole, andAppearance.getColorScheme()were probed but left ungated as version-/environment-variant.3. Three real fidelity fixes the hunt surfaced (
fix).style/children.style={({pressed}) => ...}andchildren={({pressed}) => ...}were passed through untouched. The mock now trackspressed(toggled on press-in/out) and resolves both, matching real RN's resting render and updating under press.processColor()unparseable input returned opaque black; real RN returnsundefined(its normalizer fails). Fixed to match.processColor"conformance" test was asserting the old black fallback — itself non-conforming. Corrected to real RN'sundefined, verified by the cross-check.Validation
fidelity:check(the new CI gate step) passes and guards the published artifacts against drift.Why this matters
Fidelity is the moat the mock engine has that a hand-mocked setup cannot claim — and trust is built on precision, not assertions. This converts "trust us, it's verified" into "here is every behavior we verify, the live count, and exactly where we differ," makes that claim structurally incapable of going stale, and demonstrates its value by catching three divergences (including a test that was codifying a bug).