test: E2E confidence suite (iOS/Android scripted + agent evals)#47
Merged
Conversation
Pre-release confidence tests that verify sim-use itself — the CLI and the bundled agent skill — against live devices, with the Playground apps as controlled fixtures. Scripted E2E: - Fix four iOS E2E suites (Key, KeyCombo, KeySequence, StreamVideo) that had failed since the 0.5.x move of the five iOS-only verbs under the `ios` namespace — they still invoked the top-level forms. - Fix KeyComboTests' Cmd+A assertion: an empty UITextField exposes its placeholder as the accessibility value, so the cleared-field check could never pass. - test-runner.sh now runs every suite past a failure and prints a full pass/fail map; the hardcoded list gains the missing KeyboardStateTests and the corrected StreamVideoDebugTest name (the old plural matched nothing). - iOS Playground gains paste-test, orientation-test, and permissions-test screens with matching suites (PasteTests, OrientationTests, PermissionAlertTests): paste via Cmd+V and --via-menu, AX-selector taps after rotation, and system-alert dismissal via describe-ui + tap. - New Android Playground fixture (bridge/playground) and seven Android device E2E suites gated by SIM_USE_E2E_ANDROID + ANDROID_SERIAL; `make e2e-android` builds, installs, inits the bridge, and runs them with a summary. Agent evals (e2e/agent-evals/): - Natural-language cases executed by a headless `claude -p` agent using the bundled skill against the Playground apps, judged by deterministic device-side post-condition checks. Catches skill-prose drift and verb-steering regressions (paste vs US-ASCII type, Android paste fallback) that the scripted suites cannot. Docs: design note under docs/ai/; the release skill's pre-flight now requires green make e2e / make e2e-android. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
One entry point that checks the environment, prints a cost estimate, and prompts before running — since each case spins a real `claude -p` agent that makes live API calls. - scripts/eval.sh: verify claude + sim-use present, auto-detect reachable platform(s), count the selected cases (new `run.py --count`), print an order-of-magnitude cost warning, then confirm ([y/N] on a tty, -y or a non-interactive shell proceeds) before delegating to the runner. - Makefile: `make eval` (env overrides PLATFORM/TAGS/DEVICE, or ARGS="…"). - Release pre-flight and the agent-evals README point at it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
PermissionAlertTests matched the alert buttons by ASCII label ("Don't
Allow"), but on an English simulator the button is "Don’t Allow" with a
U+2019 curly apostrophe, so the deny tap silently missed. Because the tap
was conditional on a label match, the prompt was left on screen — and a
SpringBoard permission alert survives an app terminate/relaunch, so it
covered the playground and swallowed every gesture in the suites that ran
after it (SwipeTests failing with Count: 0 was this cascade, not a swipe
bug). A JP simulator matched the JP label and never hit it, which is why it
passed in-house.
- SpringBoard alert buttons expose no accessibility id or resource-id (both
null in describe-ui --json), only a localized label. Select them by
positional list alias instead — order is stable across locales:
#1@1 Allow Once / #2@1 Allow While Using App / #3@1 Don't Allow. The tap is
now unconditional, so the alert is always dismissed regardless of locale.
- launchPlaygroundApp: if a system alert is frontmost after launch (a leak
from an earlier suite), dismiss it (tap #1@1) and relaunch once, so suites
stay independent of run order.
- PermissionAlertTests clears any stray alert before each test as well.
Verified on a JP simulator: PermissionAlertTests 2/2, and SwipeTests 5/5
even with a permission alert deliberately left up beforehand.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: onevcat <onevcat@gmail.com>
PasteTests wrapped the "--replace via menu overwrites" assertion in a plain `withKnownIssue`, asserting the iOS-26 Select-All gap always reproduces. It does not: on some simulators the menu path replaces correctly, so the block passed unexpectedly and Swift Testing failed the suite with "Known issue was not recorded" (seen on an English sim). The Select-All edit-menu step is genuinely unreliable, not uniformly broken. Assert only that the second paste is delivered (replaced or appended), and check strict replacement under `withKnownIssue(isIntermittent: true)` so the suite stays green whichever way the menu path behaves. Strict replacement is still reliably covered on the Cmd+V path. Fix candidate noted: IOSSimPasteCommand.pasteViaEditMenu select-all sequencing. Verified green on iOS 26.4 JP (menu path appends → known issue recorded and tolerated); the intermittent wrapper also absorbs the replace-works case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
…timing note - `make e2e` now runs the iOS then Android suites in sequence, continuing past a platform failure and failing if either did. The iOS-only target is renamed to `make e2e-ios`; `make e2e-android` is unchanged. - Document that a full green iOS E2E run is ~15 minutes (per-suite HID/animation waits dominate — expected, not a hang) so future runs/agents budget for it. Added to the Makefile target comments, `make help`, AGENTS.md (new "End-to-end tests" section — README stays user-facing and has none), the release pre-flight, CONTRIBUTING, and the design note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
The type-append test polled for the combined "abcde" echo with the default 8s waitForOutline timeout. Android accessibility text is eventually consistent — TYPE_VIEW_TEXT_CHANGED events are coalesced and the a11y node cache can lag the field by several seconds on a loaded/cold emulator — so on a slower host the echo still read "abc" (char_count 3) when the window expired, even though the field already held "abcde". - Poll the final "abcde" settle with a 20s timeout, and raise the shared waitForOutline default from 8s to 12s as slow-machine insurance (the predicate short-circuits, so fast runs pay nothing). - Keep the two types in quick succession: `type` appends at the caret only while the caret is at the end; waiting for the first type to fully settle lets the IME commit and the selection reset, after which the second type replaces instead of appends (verified — the intermediate "wait for abc" variant produced "de"). Comment documents this so it is not reintroduced. Verified: typeAppendsAtCaret green across repeated runs on emulator-5554. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
`resolve_device` read `candidates[0]["udid"]`, but `sim-use devices --json` emits `deviceId` (the legacy `udid` key was removed in 0.10.0), so auto-resolving a device crashed with KeyError. The bug was latent because every prior run passed --device explicitly. Now it reads `deviceId`, exits cleanly with the candidate ids listed when the platform has 0 or >1 reachable devices, and `make eval` auto-resolves a single emulator/simulator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
A running case only printed "[reset] launching…" then went silent for 1-3 min while `claude -p` worked (its stream went to the transcript file, not the console) and its cold start delays the first tokens — so a live run looked hung. Emit a compact indented line per agent step (tool_use name + a short command/selector preview, plus assistant text snippets and a start marker), while the full stream still lands in the transcript. No behavior change to the agent itself. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
The two test-fixture apps now sit symmetrically under Playgrounds/: - SimUsePlaygroundApp/ -> Playgrounds/iOS/ (xcodegen project unchanged). - bridge/playground/ (a module of the device-bridge Gradle build) -> Playgrounds/Android/ as its own standalone Gradle project. The bridge project is the shipped device-bridge product whose APK is bundled into the CLI; a test-only fixture had no business living inside it. It now has its own wrapper/settings and is fully decoupled — `:playground` is gone from bridge/settings.gradle.kts. Bundle ids are unchanged (com.cameroncooke.SimUsePlayground / com.linecorp.simuse.playground), so the E2E suites need no edits. Updated: scripts/test-runner.sh (iOS project path), build-playground-android.sh + test-runner-android.sh (new project dir, `:app:assembleDebug`, app-debug.apk), .gitignore, and the docs/CHANGELOG/CONTRIBUTING references. Verified from the new layout: both playgrounds build; the Android runner (build APK -> install -> init -> AndroidTapTests) and the iOS runner (xcodegen -> build -> install -> TapTests) pass end-to-end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A pre-release confidence layer that verifies sim-use itself — the CLI and the bundled agent skill — against live devices. The Playground apps are controlled fixtures that echo input back as assertable accessibility state; they are not the subject under test.
Why
The iOS E2E suites existed but had rotted invisibly: four suites (Key, KeyCombo, KeySequence, StreamVideo) failed continuously after the 0.5.x move of the five iOS-only verbs under the
iosnamespace — the runner aborted at the first failure and its hardcoded list had drifted, so nobody saw it. There was also no Android device E2E at all, and no way to catch skill-prose regressions that only surface with an agent in the loop.Changes
Scripted E2E
sim-use ios <verb>.UITextFieldreports its placeholder as the AX value, so the cleared-field check never passed).scripts/test-runner.shruns every suite past a failure and prints a full pass/fail map; list corrected (+KeyboardStateTests,StreamVideoDebugTestname fix — the old plural matched zero tests).paste-test,orientation-test,permissions-testscreens +PasteTests/OrientationTests/PermissionAlertTests.bridge/playground) + seven Android E2E suites gated bySIM_USE_E2E_ANDROID+ANDROID_SERIAL;make e2e-android.Agent evals (
e2e/agent-evals/)claude -pagent using the bundled skill against the Playground apps, judged by deterministic device-side checks.Docs: design note under
docs/ai/;/releasepre-flight now requires greenmake e2e/make e2e-android.Verification
swift build --build-testsandmake testgreen (one pre-existing load-sensitive latency test flakes under device contention; passes in isolation).Open item
Android
describe-ui --include-offscreenappears to be a no-op for recycled off-screen list cells (the outline renderer dropsvisibleToUser==falsenodes before the geometric filter the flag relaxes), and the top-level--helptext contradicts the code.AndroidDescribeUITestspins the honest behavior; the fix/doc decision is left to a follow-up.🤖 Generated with Claude Code