From b99100f8427de2d43f09bc34f64cbc71c3595386 Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 21:16:44 -0700 Subject: [PATCH 01/29] Add Android Emulator E2E Tester skill Adds the android-emulator-e2e-tester skill: takes an in-development Android Auth feature and proves it works on a real emulator end to end - provisions the device, deploys the app, drives the scenario (auto-handling inputs the AI can), reads logcat to decide pass/fail, and loops fix-and-retest until it passes or escalates. Contents: SKILL.md (workflow), references/ (app-and-module-map, log-signals, troubleshooting, ui-interaction), and scripts/ (emulator, appcontrol, deviceui, authlogs). Committed on its own branch so the skill can be iterated without touching existing code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 184 ++++++++++ .../references/app-and-module-map.md | 125 +++++++ .../references/log-signals.md | 115 ++++++ .../references/troubleshooting.md | 91 +++++ .../references/ui-interaction.md | 94 +++++ .../scripts/appcontrol.ps1 | 154 ++++++++ .../scripts/authlogs.ps1 | 185 ++++++++++ .../scripts/deviceui.ps1 | 188 ++++++++++ .../scripts/emulator.ps1 | 330 ++++++++++++++++++ 9 files changed, 1466 insertions(+) create mode 100644 .github/skills/android-emulator-e2e-tester/SKILL.md create mode 100644 .github/skills/android-emulator-e2e-tester/references/app-and-module-map.md create mode 100644 .github/skills/android-emulator-e2e-tester/references/log-signals.md create mode 100644 .github/skills/android-emulator-e2e-tester/references/troubleshooting.md create mode 100644 .github/skills/android-emulator-e2e-tester/references/ui-interaction.md create mode 100644 .github/skills/android-emulator-e2e-tester/scripts/appcontrol.ps1 create mode 100644 .github/skills/android-emulator-e2e-tester/scripts/authlogs.ps1 create mode 100644 .github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 create mode 100644 .github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md new file mode 100644 index 00000000..95d51ee0 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -0,0 +1,184 @@ +--- +name: android-emulator-e2e-tester +description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator. Finds or creates a suitable AVD and reuses a running one or boots it, builds/installs the right test app, drives the UI, and verifies via logcat. Use when asked to 'test this feature on the emulator', 'run the E2E test', 'verify the feature end to end', 'does this work on a device', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from the session's design spec, implementation diff, or user-provided test steps. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint) and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail and drives a fix-and-retest loop until it passes." +--- + +# Android Emulator E2E Tester + +Take an in-development Android Auth feature and prove it works on a real emulator: provision the +device, deploy the app, run the scenario (auto-handling every input the AI reasonably can), read the +logs to decide pass/fail, and loop fix-and-retest until it passes — or stop and ask when genuinely blocked. + +## When this runs + +- **Manually** — the developer asks to test/verify a feature on the emulator. +- **Automatically** — right after a feature finishes development and is ready for E2E (e.g. the + feature-orchestrator's testing step, or after a coding-agent PR lands locally). When auto-invoked, + still run Phase 1 to confirm *what* to test before touching the device. + +## Run folder + +Put all artifacts (log snapshots, screenshots, per-iteration notes) **outside the repo** so nothing is +committed: +``` +$env:USERPROFILE\android-e2e-runs\-\ +``` +Pass this as `-Out` to `authlogs.ps1`/`deviceui.ps1`. Reuse the resolved emulator `-Serial` on every +script call so commands target the right device. + +## Scripts + +All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` mentally via the +`.SYNOPSIS` in each file. They self-resolve the SDK; no hardcoded paths. + +| Script | Purpose | Key commands | +|---|---|---| +| `emulator.ps1` | Emulator/AVD lifecycle | `ensure`, `status`, `list`, `list-images`, `create`, `start` | +| `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | +| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `screenshot`, `current-app` | +| `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | + +## Workflow + +Execute in order. Announce a one-line status at each phase. + +### Phase 1 — Determine what to test + +Synthesize a concrete E2E scenario from the session context, in priority order: + +1. **User-provided test steps** in the session (explicit steps or a scenario the user described). +2. **A test plan** (from the `test-planner` skill or a linked ADO test case). +3. **The design spec** (`design-docs/`) — its acceptance criteria and flows. +4. **The implementation diff** — `git -C diff` across changed repos to see what actually + changed and which app/module it affects. + +Produce: the **feature summary**, the **target app/module** (see +[references/app-and-module-map.md](references/app-and-module-map.md)), and a **step-by-step scenario** +with an explicit, observable **success criterion** (e.g. "AcquireTokenSilent returns a token; logs show +`executed successfully` with a correlation_id; no crash"). + +**Ask the user if** the scenario is ambiguous, multiple flows could be meant, or you cannot tell what +"working" looks like. Do not guess a scenario when the intent is unclear. + +### Phase 2 — Provision the emulator + +Derive requirements from the feature (min API, Google Play services for broker/push flows) — see +"Emulator requirements per feature" in the app-and-module map — then one command finds-or-creates, +reuses-or-starts, and waits for boot: + +```powershell +./scripts/emulator.ps1 ensure -RequireGoogleApis -ApiLevel 30 -Wait +# or target a specific AVD: ./scripts/emulator.ps1 ensure -Avd Pixel_7 -Wait +``` +Capture the printed `SERIAL=` and use it as `-Serial` everywhere. If provisioning fails, see +[references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, boot hang). + +### Phase 3 — Deploy the app-under-test + +1. Decide the test surface and whether a **broker** must also be installed (brokered flows need a + calling app + a broker) — see the app-and-module map's pairing rules. +2. Prefer an APK Android Studio already built (`appcontrol.ps1 list-apks`); otherwise build + (`appcontrol.ps1 build -Module <:module>`). Gradle builds need the repo's Maven creds — if they fail + with 401, that's an environment blocker for the user. +3. Install, and for a clean run reset state: + ```powershell + ./scripts/appcontrol.ps1 install -Module :msalTestApp -Serial + ./scripts/appcontrol.ps1 clear -Package -Serial # clean slate + ./scripts/appcontrol.ps1 launch -Package -Serial + ``` + Verify the exact package/activity at runtime (the map shows how) — don't trust hardcoded IDs. + +### Phase 4 — Execute the scenario (auto-handle inputs) + +Clear the log buffer first, then drive the flow: + +```powershell +./scripts/authlogs.ps1 clear -Serial +``` + +Loop per step using [references/ui-interaction.md](references/ui-interaction.md): `wait-text` for the +expected element → act (`tap-text` / `input-text` / `key` / `finger`) → re-`dump` to confirm. Save a +`screenshot` at each major step into the run folder. + +**Auto-handle** everything the AI reasonably can: type lab test credentials, tap Next/Sign in/Accept/ +Allow/Yes, pick an account, grant permissions, simulate a fingerprint (`finger`), enter a TOTP if the +seed is known, set/enter a device PIN. **Do not** print or commit credentials. + +**Stop and ask the user** only for genuine blockers (see the consolidated list below). + +### Phase 5 — Verify from logs + +```powershell +./scripts/authlogs.ps1 scan -Package -Serial -Out \iter1\logcat.txt +``` +`scan` prints a heuristic verdict plus evidence (crash stacks, failure lines, success lines, AADSTS +codes, correlation IDs). **Reason over the evidence** — don't blindly trust PASS/FAIL. Confirm the +scenario's specific success criterion is met (a positive success signal, not just absence of errors), +per [references/log-signals.md](references/log-signals.md). For automation-test mode, also read the +`build/reports/androidTests/` results. + +Classify the outcome: +- **PASS** — success criterion met, no crash → go to Phase 7. +- **FAIL (real defect)** — crash in changed code, `AADSTS50011/700016`, regressed silent auth, broken + IPC → go to Phase 6. +- **Environment/harness problem** (missing image/creds, signature mismatch, stale snapshot, offline + adb) → fix the setup per troubleshooting, then re-run from Phase 2/3. Do **not** send these to the fix loop. +- **INCONCLUSIVE** — wrong package filter, action never executed, logging off → re-run the scenario + cleanly before deciding. + +### Phase 6 — Iterate: fix and retest + +For a real defect: + +1. **Root-cause** from the evidence: the failing signal, correlation_id, stack, and the code in the + diff it points to. State the hypothesis with the log line that supports it. +2. **Get it fixed:** + - If the code is in a checked-out sub-repo the agent can edit, **fix it directly** (Kotlin-first per + repo conventions), then rebuild. + - If the work is managed via a coding-agent PR, hand the diagnosis to that agent (e.g. `@copilot` + on the PR, or the `pbi-dispatcher` skill) with the exact evidence and root cause. +3. **Reinstall → clear → re-run the scenario → re-scan** (Phases 3–5). +4. Repeat, **max 3–5 iterations**. If still failing, **stop and escalate** to the user with the full + evidence trail (what was tried, current hypothesis, why it's stuck). Do not loop forever. + +Track iterations in the run folder (one subfolder per attempt) so regressions/progress are diffable. + +### Phase 7 — Report + +Summarize: scenario tested, target app/emulator, final verdict, iterations taken, key evidence +(success signal + correlation_id, or the blocker), and links to the run-folder artifacts (logs, +screenshots). If blocked, state exactly what you need from the user to proceed. + +## When to ask the user + +Ask (don't guess) when: +- The **scenario/intent is unclear**, or several flows could be meant, or success can't be defined. +- A **blocker** needs a human: real push-notification MFA on another device, SMS/phone OTP, a hardware + key/NFC/QR/camera, CAPTCHA, a credential the AI doesn't have, or a tenant/CA policy it can't provision. +- The **implementation is incomplete** for the path under test (stubs, `TODO`, missing wiring, feature + flag off with no way to enable) — report the gap instead of forcing a fake pass. +- **Environment setup** is missing and only the user can fix it (Maven creds/PAT, no sub-repo checkout, + no system image, no lab account). + +When you can complete a manual step *with* the user live (e.g. they approve a push), pause, let them do +it, then resume the automated flow. + +## References + +Load these as needed (don't preload all): + +| File | Read it when | +|---|---| +| [references/app-and-module-map.md](references/app-and-module-map.md) | Choosing/deploying the test app, package discovery, broker pairing, credentials, emulator requirements | +| [references/log-signals.md](references/log-signals.md) | Interpreting logcat, success/failure patterns, AADSTS codes, per-flow pass criteria, eSTS correlation | +| [references/ui-interaction.md](references/ui-interaction.md) | Driving auth screens, AI-vs-human inputs, FLAG_SECURE gotcha, selector strategy | +| [references/troubleshooting.md](references/troubleshooting.md) | Emulator/build/install/uiautomator/broker failures; env-vs-defect triage | + +## Guardrails + +- **Never** print, log, or commit real or lab credentials, tokens, or secrets. Type them onto the device only. +- **Artifacts stay outside the repo** (the run folder). Never commit logs/screenshots. +- **Environment problems are not code defects** — fix setup and re-run; only hand real defects to the fix loop. +- **Cap the loop** (3–5 iterations) and escalate with evidence rather than looping indefinitely. +- **Require a positive success signal** matching the scenario before declaring PASS. +- Follow repo conventions when fixing code (Kotlin for new code, the `Logger` class, minimal changes). diff --git a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md b/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md new file mode 100644 index 00000000..a12cf4d9 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md @@ -0,0 +1,125 @@ +# App & Module Map — What to Build, Install, and Test + +Table of contents: +- [Pick the test surface from the feature](#pick-the-test-surface-from-the-feature) +- [Module → path → typical package](#module--path--typical-package) +- [Discover the exact package & launch activity at runtime](#discover-the-exact-package--launch-activity-at-runtime) +- [Two E2E execution modes](#two-e2e-execution-modes) +- [Broker pairing rules](#broker-pairing-rules) +- [Test credentials (Lab API) and when they block](#test-credentials-lab-api-and-when-they-block) +- [Emulator requirements per feature](#emulator-requirements-per-feature) + +The repo root (`android-complete`) aggregates sub-repos cloned via `git droidSetup`: +`msal/`, `common/`, `broker/`, `adal/`, and optionally `authenticator/`. Gradle modules are +declared in `settings.gradle`. If a sub-repo folder is missing, the checkout hasn't been set up — +that is a blocker to raise with the user. + +## Pick the test surface from the feature + +Map the changed repo(s) (from `git diff` / the design) to the app you drive: + +| Feature lives in | App to install & drive | Also install (dependency) | +|---|---|---| +| **msal** (SDK) | `:msalTestApp` (interactive) or `:msalautomationapp` (automated) | A broker for brokered flows (see pairing) | +| **broker / broker4j** | `:brokerHost` test host + a broker (`:AADAuthenticator`) | `:msalTestApp` as the calling client | +| **common / common4j** (IPC, cache, telemetry) | Both sides: `:msalTestApp` **and** a broker | — | +| **adal** (legacy) | `:adalTestApp` | A broker for brokered ADAL flows | +| **authenticator** (opt-in) | `:MSAuthenticator` | — | + +Isolated broker testing without real Authenticator/Company Portal can use the **mock brokers**: +`:mockauthapp` (mock Authenticator), `:mockcp` (mock Company Portal), `:mockltw` (mock Link-to-Windows). + +## Module → path → typical package + +Package IDs below are **typical hints** — always verify at runtime (next section), because they +vary by build variant (e.g. `localDebug` often adds a `.local` suffix). + +| Gradle module | Project path | Typical applicationId | +|---|---|---| +| `:msalTestApp` | `msal/testapps/testapp` | `com.msft.identity.client.sample.local` | +| `:msalautomationapp` | `msal/msalautomationapp` | `com.microsoft.identity.client.msalautomationapp` | +| `:brokerHost` | `broker/userapp` | `com.microsoft.identity.testuserapp` | +| `:brokerautomationapp` | `broker/brokerautomationapp` | `com.microsoft.identity.client.brokerautomationapp` | +| `:AADAuthenticator` | `broker/AADAuthenticator` | `com.azure.authenticator` | +| `:adalTestApp` | `adal/userappwithbroker` | `com.microsoft.aad.adal.testapp` | +| `:mockcp` | `broker/mockbrokers/mockcp` | mock Company Portal | +| `:mockauthapp` | `broker/mockbrokers/mockauthapp` | mock Authenticator | +| `:MSAuthenticator` | `authenticator/PhoneFactor/app` | `com.azure.authenticator` | + +Real production brokers used in some E2E: Microsoft Authenticator `com.azure.authenticator`, +Intune Company Portal `com.microsoft.windowsintune.companyportal`, Link to Windows `com.microsoft.appmanager`. + +## Discover the exact package & launch activity at runtime + +Do not trust the hints above blindly. Resolve the real values: + +```powershell +# applicationId from the module's build.gradle (accounts for variant suffixes): +Select-String -Path msal/testapps/testapp/build.gradle -Pattern 'applicationId' + +# After install, confirm the installed package and find the LAUNCHER activity: +./scripts/appcontrol.ps1 is-installed -Package com.msft.identity.client.sample.local +adb shell cmd package resolve-activity --brief -c android.intent.category.LAUNCHER + +# Or read a built APK directly: +& "$env:ANDROID_HOME\build-tools\\aapt.exe" dump badging | Select-String 'package:|launchable-activity' +``` + +Then launch with the resolved values: +```powershell +./scripts/appcontrol.ps1 launch -Package -Activity +``` + +## Two E2E execution modes + +**Mode A — Automated instrumented tests (preferred when a matching test exists).** +The automation apps drive full sign-in flows and fetch lab accounts automatically via the Lab API. +Run the connected test task for the variant: +```powershell +./gradlew :msalautomationapp:connectedLocalDebugAndroidTest --tests "**" +./gradlew :brokerautomationapp:connectedLocalDebugAndroidTest --tests "**" +``` +Results land in `build/reports/androidTests/connected/` and `build/outputs/androidTest-results/`. +Read those reports plus `./scripts/authlogs.ps1 scan` to judge pass/fail. + +**Mode B — Interactive UI-driven (for new features with no automated test yet).** +Install the test app, launch it, and drive the UI with `deviceui.ps1` (tap the acquire-token +button, sign in, handle prompts), then verify with `authlogs.ps1 scan`. Use this when the feature +is brand-new or the scenario is exploratory. See [ui-interaction.md](ui-interaction.md). + +Choose Mode A if an automation test already covers the flow; otherwise Mode B. + +## Broker pairing rules + +Brokered auth requires **a calling app + a broker app**, both installed: + +- The broker must be a build the client trusts (signature/redirect-URI must match). Local `localDebug` + test apps are built to trust the `localDebug` broker signatures — keep both on the same variant. +- Signature mismatch on install shows `INSTALL_FAILED_UPDATE_INCOMPATIBLE` → uninstall the old one first + (`./scripts/appcontrol.ps1 uninstall -Package `). +- Only one active broker is used at a time. If both Authenticator and Company Portal are present, the + broker-selection logic picks one; for deterministic tests install a single broker (or a mock). +- After installing/replacing a broker, **clear the client app state** (`appcontrol.ps1 clear`) so it + re-discovers the broker cleanly. + +## Test credentials (Lab API) and when they block + +- Automated tests (Mode A) obtain lab accounts through `common/labapi` / the Lab API automatically — + no manual credential handling. +- Interactive tests (Mode B) need a test account. Preferred sources, in order: + 1. A lab account the user provides in the session. + 2. A credential the automation harness/Lab API exposes. +- **Blocker to raise with the user** if none is available, or if the scenario needs a real MSA/consumer + account, a specific tenant/CA policy, or a federated account that the AI cannot provision. +- **Never** hardcode, print, or commit real credentials. Type them into the device only; never echo them + into logs or the transcript. + +## Emulator requirements per feature + +Pass these to `emulator.ps1 ensure`: + +- **Broker / push / GMS-dependent flows** → `-RequireGoogleApis` (needs Google Play services for FCM, + account manager). Prefer a `google_apis` or `google_apis_playstore` image. +- **Min OS behavior** (e.g. scoped storage, notification permission on API 33+) → `-ApiLevel `. +- **Play Store app installs** (rare in tests) → `-RequirePlayStore`. +- Most auth E2E works on a standard `google_apis` image, API 30+. diff --git a/.github/skills/android-emulator-e2e-tester/references/log-signals.md b/.github/skills/android-emulator-e2e-tester/references/log-signals.md new file mode 100644 index 00000000..20a9d862 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/references/log-signals.md @@ -0,0 +1,115 @@ +# Log Signals — Judging E2E Pass/Fail from logcat + +Table of contents: +- [How auth logging works](#how-auth-logging-works) +- [Capture discipline](#capture-discipline) +- [Success signals](#success-signals) +- [Failure signals](#failure-signals) +- [Operation flow markers](#operation-flow-markers) +- [Common AADSTS / error codes](#common-aadsts--error-codes) +- [Judging success per flow](#judging-success-per-flow) +- [Correlating with eSTS](#correlating-with-ests) + +`authlogs.ps1 scan` automates most of this. Use this doc to interpret the evidence it prints and +to reason beyond the heuristic verdict. + +## How auth logging works + +MSAL, Common, Broker, and ADAL all log through the shared **`Logger`** class (not `android.util.Log` +directly). Lines carry a component tag and, for most auth operations, a **correlation_id** (a GUID) +that ties a client request to the broker and to the eSTS token service. Verbose/PII logging is off by +default; tokens and secrets are redacted. Do not expect to see token values — presence of a +`correlation_id` plus a success marker is the signal, not the token itself. + +Typical tags to filter on: `MSAL`, `Broker`, `Common`, `ADAL`, `OneAuth`, plus operation classes like +`BrokerMsalController`, `CommandDispatcher`, `SilentTokenCommand`, `InteractiveTokenCommand`. + +## Capture discipline + +1. `authlogs.ps1 clear` **immediately before** running the scenario — otherwise stale lines pollute the verdict. +2. Run the scenario. +3. `authlogs.ps1 scan -Package ` — always pass `-Package` so pass/fail signals are + scoped to auth-relevant + app lines (reduces false positives from system noise). +4. Save every run's snapshot into the run folder so failures can be diffed across iterations. + +## Success signals + +| Pattern | Meaning | +|---|---| +| `executed successfully` | A broker/command operation completed | +| `AcquireToken...success` / `Token ... acquired` | Token obtained (silent or interactive) | +| `TokenResult ... SUCCESS` | Result object reports success | +| `Retrieved ... token from cache` | Silent cache hit | +| `Saved ... token` | Token cached after acquisition | +| `PRT is already registered` / `Loading Workplace Join entry` | Healthy device/PRT state | + +## Failure signals + +| Pattern | Meaning / likely cause | +|---|---| +| `FATAL EXCEPTION`, `E AndroidRuntime` | App crash — always a FAIL; capture the stack | +| `AADSTS\d+` | eSTS-side error (see codes below) | +| `error_code=` / `errorCode:` | Broker/MSAL error surfaced to caller | +| `No PRT present` | Missing Primary Refresh Token — silent auth will fail | +| `INTERACTION_REQUIRED` | Silent failed; interactive needed (may be expected) | +| `invalid_grant` | Token/refresh token rejected | +| `BrokerCommunicationException` | Client↔broker IPC broke (bind/permission/signature) | +| `NullPointerException`, `NoSuchMethodError`, `ClassNotFoundException` | Code/wiring bug in the change under test | +| `CertPathValidatorException` | TLS/cert issue (often network/proxy) | + +## Operation flow markers + +Trace which operations ran (helps localize where a flow broke): + +| Log marker | Operation | +|---|---| +| `GetDeviceModeMsalBrokerOperation` | Check if Shared Device Mode is enabled | +| `AcquireTokenSilentMsalBrokerOperation` | Silent token acquisition | +| `AcquireTokenInteractiveMsalBrokerOperation` | Interactive auth | +| `GetCurrentAccountMsalBrokerOperation` | Fetch signed-in account | +| `SignOutFromSharedDeviceMsalBrokerOperation` | SDM sign-out | +| `RemoveAccount` / `removeAccount` | Account removal | + +A healthy interactive sign-in usually shows: command dispatched → interactive operation → eSTS round +trip (correlation_id) → token saved → `executed successfully`. + +## Common AADSTS / error codes + +| Code | Meaning | Usually means | +|---|---|---| +| `AADSTS50011` | Redirect URI mismatch | App registration / signature / redirect config wrong | +| `AADSTS65001` | Consent required | Grant consent in the flow (AI can tap Accept) | +| `AADSTS50076` / `50079` | MFA required | Interaction/MFA step needed | +| `AADSTS50126` | Invalid username/password | Wrong test credential | +| `AADSTS700016` | App not found in tenant | Wrong client id / tenant | +| `AADSTS50058` | Silent sign-in failed, no session | Expected before an interactive sign-in | + +`AADSTS50011`/`700016` typically indicate a **real defect** in the change under test → root-cause and +hand to the fix loop. `AADSTS65001`/`50076`/`50058` are often **flow steps**, not defects. + +## Judging success per flow + +- **Silent token (AcquireTokenSilent):** PASS = token retrieved from cache/refresh with a success + marker and no `INTERACTION_REQUIRED`/`No PRT`. If it returns `INTERACTION_REQUIRED` when a valid + account exists, that is a regression. +- **Interactive sign-in (AcquireToken):** PASS = the flow reaches the account/consent, completes, and + logs token saved + `executed successfully`. A lingering login page or an `AADSTS` error is a FAIL. +- **Sign-out / account removal:** PASS = removal operation runs and a subsequent `GetCurrentAccount` + shows no account. +- **Crash at any point → FAIL**, regardless of other signals. + +Do not declare PASS on the absence of errors alone — require a positive success signal that matches the +scenario. Absent both → verdict is INCONCLUSIVE; investigate (wrong package filter, logging off, or the +action never executed). + +## Correlating with eSTS + +When on-device logs are ambiguous, take the `correlation_id` from the snapshot and correlate with eSTS +telemetry using the **kusto-analyst** or **incident-investigator** skill. Basic query shape: + +```kql +AllPerRequestTable +| where env_time >= ago(1d) +| where CorrelationId == "" +| project env_time, CorrelationId, Call, Result, ErrorCode +``` diff --git a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md new file mode 100644 index 00000000..bf9deab6 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md @@ -0,0 +1,91 @@ +# Troubleshooting — Environment, Build, Install, and Test Failures + +Table of contents: +- [SDK / tooling not found](#sdk--tooling-not-found) +- [Emulator won't start or boot](#emulator-wont-start-or-boot) +- [adb device issues](#adb-device-issues) +- [Build failures](#build-failures) +- [Install failures](#install-failures) +- [uiautomator dump failures](#uiautomator-dump-failures) +- [Broker pairing failures](#broker-pairing-failures) +- [Distinguishing an env/test-harness problem from a real defect](#distinguishing-an-envtest-harness-problem-from-a-real-defect) + +## SDK / tooling not found + +- `emulator.ps1 resolve-sdk` prints the resolved SDK and tool paths. If it throws, `ANDROID_HOME` is + unset or points nowhere valid. Note: on some machines `ANDROID_HOME` is stored **unexpanded** + (literally `%LOCALAPPDATA%\Android\Sdk`) — the scripts expand it, but if you invoke adb/emulator + yourself, expand it too. +- `avdmanager`/`sdkmanager` may live under `cmdline-tools\latest\bin\` **or** a standalone + `cmdline-tools\latest\`. They are **slow** (Java startup) — prefer `emulator -list-avds` and reading + the `system-images/` directory over `avdmanager list`. +- No system images installed → `emulator.ps1 list-images` is empty. Install one: + `sdkmanager "system-images;android-34;google_apis;x86_64"` (or via Android Studio SDK Manager). + +## Emulator won't start or boot + +| Symptom | Fix | +|---|---| +| Hangs before adb registers | Increase `-TimeoutSec`; the first cold boot can take minutes. | +| Boot never completes | Cold boot clean: `emulator.ps1 start -Avd -ColdBoot -Wait`. | +| HAXM/WHPX/hypervisor error | Enable Windows Hypervisor Platform, or start emulator once from Android Studio to repair. | +| Stuck snapshot / corrupted state | `-ColdBoot` (`-no-snapshot-load`) to skip the stale snapshot. | +| Transient "System UI isn't responding" ANR after boot | `deviceui.ps1 tap-text -Text "Wait"`; not a test failure. | +| Need it faster / CI | `-NoWindow` (headless) — uiautomator still works. | + +## adb device issues + +- `adb offline` or missing: `adb kill-server; adb start-server`, then `emulator.ps1 status`. +- Multiple emulators running: always pass `-Serial emulator-XXXX` to every script so commands target + the right device. +- Keyguard/lock screen covering the app: `emulator.ps1` dismisses a non-secure keyguard after boot; if a + PIN is set, enter it via `deviceui.ps1`. + +## Build failures + +- **401 Unauthorized / could not resolve dependency** → missing Maven creds. The repo needs + `vstsMavenAccessToken` (and possibly `adoMsazureAuthAppAccessToken`) in `~/.gradle/gradle.properties` + (see README). This is an environment blocker — tell the user; the AI cannot mint tokens. +- **Wrong variant** → use `localDebug` for test apps (`assembleLocalDebug` / `installLocalDebug`). + `MSAuthenticator` uses `devDebug` (PROD) or `integrationDebug` (INT). +- **JDK mismatch** → the Gradle build needs the JDK the project targets (often 17). If the shell's + default `java` is older, build from the same environment Android Studio uses, or set `JAVA_HOME`. +- Prefer reusing an APK Android Studio already built (`appcontrol.ps1 list-apks`) instead of a fresh + Gradle build when one exists — it's faster and avoids credential setup. + +## Install failures + +| adb error | Cause | Fix | +|---|---|---| +| `INSTALL_FAILED_UPDATE_INCOMPATIBLE` | Signature mismatch with an already-installed build | Uninstall first: `appcontrol.ps1 uninstall -Package ` | +| `INSTALL_FAILED_VERSION_DOWNGRADE` | Installing older versionCode | Uninstall the newer build first | +| `INSTALL_FAILED_INSUFFICIENT_STORAGE` | Emulator disk full | Wipe data / create a larger AVD | +| `INSTALL_FAILED_NO_MATCHING_ABIS` | APK ABI ≠ emulator ABI | Use an `x86_64` system image, or build a universal APK | +| `INSTALL_FAILED_USER_RESTRICTED` | Play-protect / user prompt | Use a non-Play-Store `google_apis` image | + +## uiautomator dump failures + +- "could not get idle state" / empty tree: the screen is animating or a secure window is up. Retry + (the script retries 3×), wait for a stable element with `wait-text`, or see the FLAG_SECURE notes in + [ui-interaction.md](ui-interaction.md). +- WebView-heavy login pages: some nodes are absent; you can still often type into the focused field. + +## Broker pairing failures + +- `BrokerCommunicationException` / bind failures: the broker isn't installed, isn't the trusted build, + or the client wasn't cleared after swapping brokers. Reinstall the matching-variant broker, then + `appcontrol.ps1 clear` the client. +- Two brokers installed causing nondeterminism: keep exactly one broker (or a mock) for a clean run. +- Push/GMS-dependent broker flows failing on a bare image: recreate the emulator with `-RequireGoogleApis`. + +## Distinguishing an env/test-harness problem from a real defect + +Before blaming the code under test, rule out the harness: + +- Missing SDK image / creds / offline adb / signature mismatch / stale snapshot → **environment**, fix + the setup and re-run; do **not** hand these to the code-fix loop. +- Crash in the changed code (`E AndroidRuntime` with the app package + a class from the diff), + `AADSTS50011`/`700016`, a regressed `INTERACTION_REQUIRED`, or a broken IPC path introduced by the + change → **real defect**, hand to the fix loop with the evidence. +- If unsure, reproduce once more from a clean state (`clear` app, `clear` logcat, cold-boot if needed). + A failure that reproduces deterministically from clean state is almost certainly a real defect. diff --git a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md new file mode 100644 index 00000000..fd1acb2a --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md @@ -0,0 +1,94 @@ +# UI Interaction — Auto-Handling Inputs and Knowing When to Ask + +Table of contents: +- [Interaction loop](#interaction-loop) +- [Selector strategy](#selector-strategy) +- [Common auth screens and how to drive them](#common-auth-screens-and-how-to-drive-them) +- [Inputs the AI handles automatically](#inputs-the-ai-handles-automatically) +- [Inputs that require the user (blockers)](#inputs-that-require-the-user-blockers) +- [The FLAG_SECURE gotcha](#the-flag_secure-gotcha) +- [Robustness tips](#robustness-tips) + +Drive the UI with `scripts/deviceui.ps1`. The goal: perform every input a human tester *could* +reasonably do, automatically — and stop to ask only when an input genuinely cannot be produced by the AI. + +## Interaction loop + +For each step: +1. `deviceui.ps1 dump` (or `wait-text`) to read what is on screen **now**. +2. Decide the next action from the visible elements. +3. Act: `tap-text` / `tap-desc` / `input-text` / `key` / `finger`. +4. Re-`dump` to confirm the screen advanced. The UI tree changes after every action — never reuse a + stale dump. +5. On an unexpected screen, `screenshot` to a run file and reason about it before continuing. + +## Selector strategy + +Prefer, in order: visible **text** → **content-desc** → **resource-id**. `tap-text`/`find-text` match +text first, then content-desc, case-insensitive substring (use `-Exact` for equality, `-Index N` to +pick among duplicates). When text is empty (icon-only buttons), use `tap-desc`. As a last resort, read +`Bounds` from `dump` and `tap-xy`. + +## Common auth screens and how to drive them + +| Screen | What to do (AI) | +|---|---| +| **App first-run / runtime permissions** | Tap `Allow` / `While using the app` / `Continue`; or pre-grant with `appcontrol.ps1 grant`. | +| **Test app main screen** | Tap the acquire-token control (e.g. `Acquire Token`, `Sign in`, `AcquireTokenSilent`). Set scopes/authority fields first if the scenario needs them. | +| **Broker account picker** | If the target account is listed, `tap-text` it (SSO). If `Add account` / `Use another account`, tap it to start interactive sign-in. | +| **eSTS email page** | `input-text` the username, then `tap-text "Next"`. | +| **eSTS password page** | `input-text` the password, then `tap-text "Sign in"`. Never log the value. | +| **Consent / permissions requested** | `tap-text "Accept"` (safe for test apps/accounts). | +| **"Stay signed in?" / KMSI** | `tap-text "Yes"` (or `No` if the scenario needs a fresh session). | +| **"Approve sign in request" (push MFA)** | Usually a **blocker** — see below, unless a TOTP/seed is available. | +| **Fingerprint / biometric prompt** | On emulator, `deviceui.ps1 finger -Text 1` simulates an enrolled fingerprint. Enroll first if needed (Settings → Security), or fall back to PIN. | +| **Device PIN/pattern (keyguard)** | Set a known PIN via adb during setup, then enter it; or `emulator.ps1` dismisses a no-secure keyguard automatically. | + +## Inputs the AI handles automatically + +Do these without asking: + +- Typing a **lab/test** username and password into fields. +- Tapping `Next`, `Sign in`, `Accept`, `Allow`, `Continue`, `Yes`, `Got it`, `Done`. +- Selecting an account from a picker; choosing `Add account`. +- Granting runtime permissions (`appcontrol.ps1 grant` or tapping the dialog). +- Dismissing benign system dialogs/ANRs (`Wait`), closing bottom sheets (`key BACK`). +- **Simulating a fingerprint** on the emulator (`finger`). +- Entering a **TOTP** code when the authenticator seed/secret is known to the session (compute it). +- Setting a device PIN during environment setup and re-entering it later. +- Toggling in-app test switches (feature flags exposed in the test app UI). + +## Inputs that require the user (blockers) + +Stop and ask the user (or report as blocked) when the step needs something the AI cannot produce: + +- **Real push-notification MFA approval** on a separate physical device/Authenticator (no TOTP seed). +- **SMS / phone-call OTP** sent to a real phone number. +- **Hardware** security key (FIDO2), NFC, or a real camera/QR scan that the emulator can't provide. +- **CAPTCHA** / "prove you're human" challenges. +- A **credential the AI doesn't have** (no lab account; account needs a password only the user knows). +- A tenant/policy the AI can't provision (specific Conditional Access, federation, sovereign cloud). +- Any step the design says is **manual sign-off only**. + +When blocked, state exactly which step and why, and what you need from the user (e.g. "approve the push +on your phone, then tell me to continue"). If the user can complete the manual step live, continue the +automated flow afterward. + +## The FLAG_SECURE gotcha + +eSTS login pages and some broker screens may set `FLAG_SECURE`, which can make `screenshot` return black +and can hide fields from `uiautomator dump`. Mitigations: +- You can often still `input-text` into a focused field even when it doesn't appear in the dump (the + field has focus after the page loads or after tapping where it should be). +- Use `deviceui.ps1 current-app` to confirm you're on the expected login activity. +- If the tree is genuinely unreadable, fall back to `tap-xy` using known layout positions, or prefer + **Mode A** automation tests (they use instrumented hooks that bypass this). +- If nothing works, treat it as a blocker and tell the user. + +## Robustness tips + +- Always `wait-text` for the expected element before acting (default 20s) instead of fixed sleeps. +- After `input-text`, verify with `find-text` that the value landed (for non-secure fields). +- Web/ESTS pages load slowly — allow longer `-TimeoutSec` (30–60s) on sign-in pages. +- If an element isn't found, re-`dump` once more (the page may still be rendering) before deciding it's a blocker. +- Keep a screenshot per major step in the run folder for the final report. diff --git a/.github/skills/android-emulator-e2e-tester/scripts/appcontrol.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/appcontrol.ps1 new file mode 100644 index 00000000..d19b74a4 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/scripts/appcontrol.ps1 @@ -0,0 +1,154 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +<# +.SYNOPSIS + Build, install, launch, and reset the app-under-test on a device/emulator for E2E runs. + +.PARAMETER Command + build | install | launch | stop | clear | uninstall | list-apks | grant | is-installed + +.DESCRIPTION + Thin wrappers over gradlew (build/install) and adb (launch/state/permissions). + `clear` resets app state for a clean E2E run without a factory reset. + +.EXAMPLE + ./appcontrol.ps1 build -Module :msalTestApp -Variant localDebug + ./appcontrol.ps1 install -Module :msalTestApp -Variant localDebug # gradle install + ./appcontrol.ps1 install -Apk C:\path\app-localDebug.apk # direct apk (adb -r -g) + ./appcontrol.ps1 clear -Package com.msft.identity.client.sample.local + ./appcontrol.ps1 launch -Package com.msft.identity.client.sample.local + ./appcontrol.ps1 grant -Package -Permission android.permission.POST_NOTIFICATIONS + +.NOTES + Gradle builds require the repo's Maven credentials (see README) and can take minutes. + Prefer reusing an APK already built by Android Studio when one exists (list-apks). +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('build', 'install', 'launch', 'stop', 'clear', 'uninstall', 'list-apks', 'grant', 'is-installed')] + [string]$Command = 'list-apks', + + [string]$Serial, + [string]$Module, # gradle module path, e.g. :msalTestApp + [string]$Package, # applicationId, e.g. com.microsoft.brokerhost + [string]$Apk, # explicit apk path for direct install + [string]$Activity, # optional fully-qualified activity for launch + [string]$Variant = 'localDebug', + [string]$Permission, + [string]$RepoRoot +) + +$ErrorActionPreference = 'Stop' + +function Get-Adb { + foreach ($v in @($env:ANDROID_HOME, $env:ANDROID_SDK_ROOT)) { + if ($v) { + $p = Join-Path ([Environment]::ExpandEnvironmentVariables($v)) 'platform-tools\adb.exe' + if (Test-Path $p) { return $p } + } + } + foreach ($root in @((Join-Path $env:LOCALAPPDATA 'Android\Sdk'), (Join-Path $HOME 'AppData\Local\Android\Sdk'), (Join-Path $HOME 'Library/Android/sdk'), (Join-Path $HOME 'Android/Sdk'))) { + foreach ($exe in @('platform-tools\adb.exe', 'platform-tools/adb')) { + $p = Join-Path $root $exe + if ($root -and (Test-Path $p)) { return $p } + } + } + $cmd = Get-Command adb -ErrorAction SilentlyContinue + if ($cmd) { return $cmd.Source } + throw "adb not found. Set ANDROID_HOME to your SDK root." +} + +function Find-RepoRoot { + if ($RepoRoot) { return $RepoRoot } + $d = Get-Location + while ($d) { + foreach ($w in @('gradlew.bat', 'gradlew')) { + if (Test-Path (Join-Path $d $w)) { return $d.Path } + } + $parent = Split-Path $d -Parent + if (-not $parent -or $parent -eq $d) { break } + $d = Get-Item $parent + } + throw "Could not find the android-complete repo root (no gradlew found walking up). Pass -RepoRoot." +} + +function Cap { param([string]$s) if ($s) { $s.Substring(0, 1).ToUpper() + $s.Substring(1) } else { $s } } + +$adb = Get-Adb +function Adb { + if ($Serial) { & $adb -s $Serial @args } else { & $adb @args } +} + +switch ($Command) { + 'build' { + if (-not $Module) { throw "Provide -Module (e.g. :msalTestApp)." } + $root = Find-RepoRoot + $gw = if (Test-Path (Join-Path $root 'gradlew.bat')) { Join-Path $root 'gradlew.bat' } else { Join-Path $root 'gradlew' } + $task = "assemble$(Cap $Variant)" + Write-Host "Building $Module`:$task (cwd=$root)" + Push-Location $root + try { & $gw "$Module`:$task" } + finally { Pop-Location } + } + 'install' { + if ($Apk) { + if (-not (Test-Path $Apk)) { throw "APK not found: $Apk" } + Write-Host "Installing (adb -r -g): $Apk" + Adb install -r -g $Apk | Write-Host + } + elseif ($Module) { + $root = Find-RepoRoot + $gw = if (Test-Path (Join-Path $root 'gradlew.bat')) { Join-Path $root 'gradlew.bat' } else { Join-Path $root 'gradlew' } + $task = "install$(Cap $Variant)" + Write-Host "Installing via gradle: $Module`:$task" + Push-Location $root + try { & $gw "$Module`:$task" } + finally { Pop-Location } + } + else { throw "Provide -Apk or -Module <:module>." } + } + 'list-apks' { + $root = Find-RepoRoot + $apks = Get-ChildItem $root -Recurse -Filter '*.apk' -ErrorAction SilentlyContinue | + Where-Object { $_.FullName -match 'build[\\/]outputs[\\/]apk' } + if ($Variant) { $apks = $apks | Where-Object { $_.Name -match $Variant -or $_.FullName -match $Variant } } + if (-not $apks) { Write-Host "No built APKs found under build/outputs/apk. Build first (Android Studio or ./appcontrol.ps1 build)."; exit 0 } + $apks | Select-Object FullName, @{n = 'MB'; e = { [math]::Round($_.Length / 1MB, 1) } }, LastWriteTime | + Sort-Object LastWriteTime -Descending | Format-Table -AutoSize | Out-String | Write-Host + } + 'launch' { + if (-not $Package) { throw "Provide -Package." } + if ($Activity) { + Adb shell am start -n "$Package/$Activity" | Write-Host + } + else { + Adb shell monkey -p $Package -c android.intent.category.LAUNCHER 1 2>$null | Out-Null + Write-Host "Launched default activity of $Package" + } + } + 'stop' { + if (-not $Package) { throw "Provide -Package." } + Adb shell am force-stop $Package | Out-Null + Write-Host "Force-stopped $Package" + } + 'clear' { + if (-not $Package) { throw "Provide -Package." } + Adb shell pm clear $Package | Write-Host + Write-Host "Cleared app state for $Package" + } + 'uninstall' { + if (-not $Package) { throw "Provide -Package." } + Adb uninstall $Package | Write-Host + } + 'grant' { + if (-not $Package -or -not $Permission) { throw "Provide -Package and -Permission." } + Adb shell pm grant $Package $Permission | Write-Host + Write-Host "Granted $Permission to $Package" + } + 'is-installed' { + if (-not $Package) { throw "Provide -Package." } + $found = (Adb shell pm list packages $Package 2>$null | Out-String) + if ($found -match [regex]::Escape("package:$Package")) { Write-Host "INSTALLED: $Package"; exit 0 } + Write-Host "NOT INSTALLED: $Package"; exit 2 + } +} diff --git a/.github/skills/android-emulator-e2e-tester/scripts/authlogs.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/authlogs.ps1 new file mode 100644 index 00000000..205c4c28 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/scripts/authlogs.ps1 @@ -0,0 +1,185 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +<# +.SYNOPSIS + Capture and analyze logcat to decide whether an E2E scenario passed, and to surface + root-cause evidence (crashes, exceptions, error codes, eSTS correlation IDs) when it fails. + +.PARAMETER Command + clear | snapshot | scan | grep | watch + +.DESCRIPTION + clear -> flush the logcat buffer right before running the scenario. + snapshot -> dump the current buffer to a file (raw + auth-filtered view). + scan -> classify PASS / FAIL / INCONCLUSIVE from success & failure signals and + extract evidence (crash stacks, AADSTS codes, correlation IDs, error codes). + grep -> search the buffer/file for a custom -Pattern. + watch -> stream live for -TimeoutSec, teeing to a file (long scenarios). + +.EXAMPLE + ./authlogs.ps1 clear + # ... run the scenario ... + ./authlogs.ps1 scan -Package com.microsoft.brokerhost -Out C:\runs\r1\logcat.txt + +.NOTES + The verdict is a heuristic. Always read the extracted evidence and reason about it; + do not blindly trust PASS/FAIL. See references/log-signals.md. +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('clear', 'snapshot', 'scan', 'grep', 'watch')] + [string]$Command = 'scan', + + [string]$Serial, + [string]$Package, + [string]$Pattern, + [string]$Out, + [int]$TimeoutSec = 60, + [int]$Context = 2 +) + +$ErrorActionPreference = 'Stop' + +function Get-Adb { + foreach ($v in @($env:ANDROID_HOME, $env:ANDROID_SDK_ROOT)) { + if ($v) { + $p = Join-Path ([Environment]::ExpandEnvironmentVariables($v)) 'platform-tools\adb.exe' + if (Test-Path $p) { return $p } + } + } + foreach ($root in @((Join-Path $env:LOCALAPPDATA 'Android\Sdk'), (Join-Path $HOME 'AppData\Local\Android\Sdk'), (Join-Path $HOME 'Library/Android/sdk'), (Join-Path $HOME 'Android/Sdk'))) { + foreach ($exe in @('platform-tools\adb.exe', 'platform-tools/adb')) { + $p = Join-Path $root $exe + if ($root -and (Test-Path $p)) { return $p } + } + } + $cmd = Get-Command adb -ErrorAction SilentlyContinue + if ($cmd) { return $cmd.Source } + throw "adb not found. Set ANDROID_HOME to your SDK root." +} + +$adb = Get-Adb +function Adb { + if ($Serial) { & $adb -s $Serial @args } else { & $adb @args } +} + +# Keywords that identify auth-relevant log lines across MSAL / Common / Broker. +$AuthKeywords = 'MSAL|Broker|Common|ADAL|OneAuth|AcquireToken|PRT|eSTS|ESTS|correlation|AADSTS|TokenResult|SilentToken|InteractiveToken|WorkplaceJoin|SSO|Authenticator|CompanyPortal|identity' + +# Failure signals (regex, case-sensitive where it matters for tags like "E "). +$FailSignals = @( + 'FATAL EXCEPTION', 'E AndroidRuntime', 'AndroidRuntime:.*Exception', + 'ANR in ', 'Force finishing', 'process .* died', + 'AADSTS\d+', 'error_code[=:\s]', 'errorCode[=:\s]', 'invalid_grant', + 'INTERACTION_REQUIRED', 'UNKNOWN_ERROR', 'No PRT present', + 'CertPathValidatorException', 'NullPointerException', 'NoSuchMethodError', + 'ClassNotFoundException', 'SecurityException', 'BrokerCommunicationException', + 'failed to acquire', 'Authentication failed', 'operation failed', ' FAILED' +) +# Success signals. +$PassSignals = @( + 'executed successfully', 'AcquireToken.*[Ss]uccess', 'Token.*acquired', + 'TokenResult.*SUCCESS', 'SILENT.*SUCCESS', 'Silent request.*success', + 'succeeded', 'SUCCEEDED', 'Saved.*token', 'Retrieved.*token from cache' +) + +function Read-Buffer { + if ($Out -and (Test-Path $Out)) { return Get-Content $Out } + return (Adb logcat -d -v threadtime 2>$null) +} + +function Ensure-Dir { param([string]$Path) $d = Split-Path $Path -Parent; if ($d -and -not (Test-Path $d)) { New-Item -ItemType Directory -Force -Path $d | Out-Null } } + +function Save-Snapshot { + if (-not $Out) { + $dir = Join-Path ([System.IO.Path]::GetTempPath()) 'android-e2e' + New-Item -ItemType Directory -Force -Path $dir | Out-Null + $Out = Join-Path $dir ("logcat_{0}.txt" -f (Get-Date -Format 'yyyyMMdd_HHmmss')) + } + Ensure-Dir $Out + $lines = Adb logcat -d -v threadtime 2>$null + $lines | Set-Content -Path $Out -Encoding utf8 + return @{ Path = $Out; Lines = $lines } +} + +switch ($Command) { + 'clear' { + Adb logcat -c 2>$null | Out-Null + Write-Host "logcat buffer cleared." + } + 'snapshot' { + $s = Save-Snapshot + $auth = $s.Lines | Where-Object { $_ -match $AuthKeywords } + $authPath = ($s.Path -replace '\.txt$', '') + '.auth.txt' + $auth | Set-Content -Path $authPath -Encoding utf8 + Write-Host "Raw log: $($s.Path) ($($s.Lines.Count) lines)" + Write-Host "Auth log: $authPath ($($auth.Count) lines)" + } + 'grep' { + if (-not $Pattern) { throw "Provide -Pattern." } + $lines = Read-Buffer + $lines | Select-String -Pattern $Pattern -Context $Context, $Context | ForEach-Object { $_.ToString() } | Write-Host + } + 'watch' { + if (-not $Out) { $Out = Join-Path ([System.IO.Path]::GetTempPath()) ("e2e_watch_{0}.txt" -f (Get-Date -Format 'yyyyMMdd_HHmmss')) } + Write-Host "Streaming logcat for ${TimeoutSec}s -> $Out (Ctrl-C to stop early)" + $job = Start-Job -ScriptBlock { param($a, $s) if ($s) { & $a -s $s logcat -v threadtime } else { & $a logcat -v threadtime } } -ArgumentList $adb, $Serial + Start-Sleep -Seconds $TimeoutSec + Receive-Job $job | Set-Content -Path $Out -Encoding utf8 + Stop-Job $job -ErrorAction SilentlyContinue; Remove-Job $job -Force -ErrorAction SilentlyContinue + Write-Host "Saved: $Out" + } + 'scan' { + $s = Save-Snapshot + $lines = $s.Lines + if ($Package) { + $pid0 = (Adb shell pidof $Package 2>$null | Out-String).Trim() + $filtered = $lines | Where-Object { $_ -match $AuthKeywords -or ($Package -and $_ -match [regex]::Escape($Package)) -or ($pid0 -and $_ -match "\s$pid0\s") } + } + else { + $filtered = $lines | Where-Object { $_ -match $AuthKeywords } + } + + $failRe = ($FailSignals -join '|') + $passRe = ($PassSignals -join '|') + # Scope pass/fail signals to auth-relevant / app-filtered lines to avoid unrelated + # system-log noise (e.g. "Failed Usage reports"). Crashes stay global because + # AndroidRuntime crash stacks are not auth-tagged. + $failHits = @($filtered | Select-String -Pattern $failRe -CaseSensitive:$false) + $passHits = @($filtered | Select-String -Pattern $passRe -CaseSensitive:$false) + $crash = @($lines | Select-String -Pattern 'FATAL EXCEPTION|E AndroidRuntime' -CaseSensitive:$false) + + $joined = ($lines -join "`n") + $aadsts = @([regex]::Matches($joined, 'AADSTS\d+') | ForEach-Object { $_.Value }) | Sort-Object -Unique + $corr = @([regex]::Matches($joined, 'correlation[_ ]?id["\s:=]+([0-9a-fA-F-]{36})') | ForEach-Object { $_.Groups[1].Value }) | Sort-Object -Unique + + $verdict = 'INCONCLUSIVE' + if ($crash.Count -gt 0) { $verdict = 'FAIL (crash)' } + elseif ($failHits.Count -gt 0 -and $passHits.Count -eq 0) { $verdict = 'FAIL' } + elseif ($passHits.Count -gt 0 -and $crash.Count -eq 0 -and $failHits.Count -eq 0) { $verdict = 'PASS' } + elseif ($passHits.Count -gt 0 -and $failHits.Count -gt 0) { $verdict = 'MIXED (review evidence)' } + + Write-Host "==================== E2E LOG SCAN ====================" + Write-Host "Verdict: $verdict" + Write-Host "Snapshot: $($s.Path)" + Write-Host "Pass signals: $($passHits.Count) Fail signals: $($failHits.Count) Crashes: $($crash.Count)" + if ($aadsts.Count -gt 0) { Write-Host "AADSTS codes: $($aadsts -join ', ')" } + if ($corr.Count -gt 0) { Write-Host "Correlation IDs: $($corr -join ', ')" } + Write-Host "" + if ($crash.Count -gt 0) { + Write-Host "--- CRASH (first 25 lines) ---" + $idx = [array]::IndexOf($lines, $crash[0].Line) + if ($idx -ge 0) { ($lines[$idx..([Math]::Min($idx + 24, $lines.Count - 1))]) | Write-Host } + } + if ($failHits.Count -gt 0) { + Write-Host "--- FAILURE EVIDENCE (top 15) ---" + $failHits | Select-Object -First 15 | ForEach-Object { $_.Line } | Write-Host + } + if ($passHits.Count -gt 0) { + Write-Host "--- SUCCESS EVIDENCE (top 8) ---" + $passHits | Select-Object -First 8 | ForEach-Object { $_.Line } | Write-Host + } + Write-Host "======================================================" + if ($verdict -like 'FAIL*') { exit 1 } + } +} diff --git a/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 new file mode 100644 index 00000000..e4e83a6a --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 @@ -0,0 +1,188 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +<# +.SYNOPSIS + Read and drive an Android device/emulator UI via adb + uiautomator so the agent can + perform AI-handleable inputs automatically (tap buttons, type credentials, grant + permission dialogs, simulate a fingerprint, read on-screen text, capture screenshots). + +.PARAMETER Command + dump | find-text | tap-text | tap-desc | input-text | wait-text | screenshot | + key | finger | current-app | tap-xy + +.EXAMPLE + ./deviceui.ps1 wait-text -Text "Sign in" -TimeoutSec 30 + ./deviceui.ps1 tap-text -Text "Sign in" + ./deviceui.ps1 input-text -Text "user@contoso.com" + ./deviceui.ps1 key -Text ENTER + ./deviceui.ps1 finger -Text 1 # simulate enrolled fingerprint id 1 + ./deviceui.ps1 screenshot -Out C:\runs\step1.png + ./deviceui.ps1 current-app # resolved/focused package + activity + +.NOTES + Text matching is case-insensitive substring by default. Use -Exact for equality. + Always re-`dump` after an action; the UI tree changes between steps. +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('dump', 'find-text', 'tap-text', 'tap-desc', 'input-text', 'wait-text', + 'screenshot', 'key', 'finger', 'current-app', 'tap-xy')] + [string]$Command = 'dump', + + [string]$Serial, + [string]$Text, + [int]$X, + [int]$Y, + [int]$Index = 0, + [switch]$Exact, + [int]$TimeoutSec = 20, + [string]$Out +) + +$ErrorActionPreference = 'Stop' + +function Get-Adb { + foreach ($v in @($env:ANDROID_HOME, $env:ANDROID_SDK_ROOT)) { + if ($v) { + $p = Join-Path ([Environment]::ExpandEnvironmentVariables($v)) 'platform-tools\adb.exe' + if (Test-Path $p) { return $p } + $p2 = Join-Path ([Environment]::ExpandEnvironmentVariables($v)) 'platform-tools/adb' + if (Test-Path $p2) { return $p2 } + } + } + foreach ($root in @((Join-Path $env:LOCALAPPDATA 'Android\Sdk'), (Join-Path $HOME 'AppData\Local\Android\Sdk'), (Join-Path $HOME 'Library/Android/sdk'), (Join-Path $HOME 'Android/Sdk'))) { + foreach ($exe in @('platform-tools\adb.exe', 'platform-tools/adb')) { + $p = Join-Path $root $exe + if ($root -and (Test-Path $p)) { return $p } + } + } + $cmd = Get-Command adb -ErrorAction SilentlyContinue + if ($cmd) { return $cmd.Source } + throw "adb not found. Set ANDROID_HOME to your SDK root." +} + +$adb = Get-Adb +function Adb { + if ($Serial) { & $adb -s $Serial @args } else { & $adb @args } +} + +function Get-UiXml { + for ($i = 0; $i -lt 3; $i++) { + Adb shell uiautomator dump /sdcard/window_dump.xml 2>$null | Out-Null + $xml = (Adb exec-out cat /sdcard/window_dump.xml 2>$null | Out-String) + if ($xml -match '|;&*~"''`$\\])', '\$1' + $s = $s -replace ' ', '%s' + return $s +} + +switch ($Command) { + 'dump' { + Get-Nodes (Get-UiXml) | Where-Object { $_.Text -or $_.Desc } | + Select-Object Text, Desc, Res, Clickable, Bounds | Format-Table -AutoSize | Out-String | Write-Host + } + 'find-text' { + $m = Find-ByField 'Text' $Text + if (-not $m) { $m = Find-ByField 'Desc' $Text } + if (-not $m) { Write-Host "NOT FOUND: '$Text'"; exit 2 } + $m | Select-Object Text, Desc, Res, Bounds, Cx, Cy | Format-Table -AutoSize | Out-String | Write-Host + } + 'tap-text' { + $m = @(Find-ByField 'Text' $Text) + if (-not $m) { $m = @(Find-ByField 'Desc' $Text) } + if (-not $m) { Write-Host "NOT FOUND: '$Text'"; exit 2 } + $t = $m[[Math]::Min($Index, $m.Count - 1)] + if ($null -eq $t.Cx) { Write-Host "Element '$Text' has no tappable bounds."; exit 3 } + Adb shell input tap $t.Cx $t.Cy | Out-Null + Write-Host "Tapped '$Text' at ($($t.Cx),$($t.Cy))" + } + 'tap-desc' { + $m = @(Find-ByField 'Desc' $Text) + if (-not $m) { Write-Host "NOT FOUND (content-desc): '$Text'"; exit 2 } + $t = $m[[Math]::Min($Index, $m.Count - 1)] + Adb shell input tap $t.Cx $t.Cy | Out-Null + Write-Host "Tapped desc '$Text' at ($($t.Cx),$($t.Cy))" + } + 'tap-xy' { + Adb shell input tap $X $Y | Out-Null + Write-Host "Tapped ($X,$Y)" + } + 'input-text' { + $enc = Encode-Input $Text + Adb shell input text $enc | Out-Null + Write-Host "Typed: $Text" + } + 'wait-text' { + $deadline = (Get-Date).AddSeconds($TimeoutSec) + while ((Get-Date) -lt $deadline) { + $m = Find-ByField 'Text' $Text + if (-not $m) { $m = Find-ByField 'Desc' $Text } + if ($m) { Write-Host "FOUND: '$Text'"; exit 0 } + Start-Sleep -Seconds 2 + } + Write-Host "TIMEOUT waiting for '$Text' (${TimeoutSec}s)"; exit 4 + } + 'screenshot' { + if (-not $Out) { $Out = Join-Path (Get-Location) ("screen_{0}.png" -f (Get-Date -Format 'yyyyMMdd_HHmmss')) } + $d = Split-Path $Out -Parent; if ($d -and -not (Test-Path $d)) { New-Item -ItemType Directory -Force -Path $d | Out-Null } + Adb shell screencap -p /sdcard/_sc.png | Out-Null + Adb pull /sdcard/_sc.png $Out | Out-Null + Adb shell rm -f /sdcard/_sc.png | Out-Null + Write-Host "Saved screenshot: $Out" + } + 'key' { + $map = @{ BACK = 4; HOME = 3; ENTER = 66; TAB = 61; MENU = 82; APP_SWITCH = 187; DEL = 67; ESCAPE = 111; SEARCH = 84 } + $code = if ($map.ContainsKey($Text.ToUpper())) { $map[$Text.ToUpper()] } else { $Text } + Adb shell input keyevent $code | Out-Null + Write-Host "Key: $Text ($code)" + } + 'finger' { + $id = if ($Text) { $Text } else { '1' } + $ser = if ($Serial) { $Serial } else { '' } + if ($ser) { & $adb -s $ser emu finger touch $id | Out-Null } else { & $adb -e emu finger touch $id | Out-Null } + Write-Host "Simulated fingerprint id=$id" + } + 'current-app' { + $win = (Adb shell dumpsys window 2>$null | Out-String) + $focus = ($win -split "`n" | Where-Object { $_ -match 'mCurrentFocus|mFocusedApp' }) -join "`n" + Write-Host $focus + } +} diff --git a/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 new file mode 100644 index 00000000..fef1da9e --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 @@ -0,0 +1,330 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +<# +.SYNOPSIS + Android emulator lifecycle helper for E2E testing: resolve the SDK, list/inspect + AVDs and running devices, create an AVD, start one, wait for boot, or do all of + that in one shot with `ensure`. + +.DESCRIPTION + Self-resolves the Android SDK location (handles unexpanded %LOCALAPPDATA% in + ANDROID_HOME) and never hardcodes machine paths. Prefers fast operations + (`emulator -list-avds`, reading the system-images/ dir) over the slow + avdmanager/sdkmanager where possible. + +.PARAMETER Command + resolve-sdk | list | list-images | status | create | start | ensure + +.EXAMPLE + # One-shot: guarantee a booted emulator meeting the feature's needs, print its serial. + ./emulator.ps1 ensure -ApiLevel 34 -RequireGoogleApis -Wait + +.EXAMPLE + ./emulator.ps1 ensure -Avd Pixel_7 -Wait # use/create+start a named AVD + ./emulator.ps1 status # what is running right now + ./emulator.ps1 list-images # installed system images (for create) +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('resolve-sdk', 'list', 'list-images', 'status', 'create', 'start', 'ensure')] + [string]$Command = 'status', + + [string]$Avd, # explicit AVD name to use/create + [int]$ApiLevel = 0, # minimum API level requirement (0 = any) + [string]$Image, # explicit system-image package id for create + [string]$Device = 'pixel_7', # device profile for create + [switch]$RequireGoogleApis, # require google_apis or google_apis_playstore tag (GMS/push) + [switch]$RequirePlayStore, # require google_apis_playstore tag + [switch]$Wait, # wait for full boot + [switch]$ColdBoot, # start with a clean state (-no-snapshot-load) + [switch]$NoWindow, # headless (-no-window) + [int]$TimeoutSec = 300 +) + +$ErrorActionPreference = 'Stop' + +# --------------------------------------------------------------------------- +# SDK + tool resolution +# --------------------------------------------------------------------------- +function Resolve-Sdk { + $candidates = @() + foreach ($v in @($env:ANDROID_HOME, $env:ANDROID_SDK_ROOT)) { + if ($v) { $candidates += [Environment]::ExpandEnvironmentVariables($v) } + } + if ($env:LOCALAPPDATA) { $candidates += (Join-Path $env:LOCALAPPDATA 'Android\Sdk') } + $candidates += (Join-Path $HOME 'AppData\Local\Android\Sdk') # Windows + $candidates += (Join-Path $HOME 'Library/Android/sdk') # macOS + $candidates += (Join-Path $HOME 'Android/Sdk') # Linux + foreach ($c in $candidates) { + if ($c -and (Test-Path (Join-Path $c 'platform-tools'))) { return (Resolve-Path $c).Path } + } + throw "Could not locate the Android SDK. Set ANDROID_HOME to your SDK root (the folder containing platform-tools)." +} + +function Get-Tool { + param([string]$Sdk, [string]$RelNoExt, [string[]]$Extra) + $exe = if ($IsWindows -or $env:OS -eq 'Windows_NT') { '.exe' } else { '' } + $bat = if ($IsWindows -or $env:OS -eq 'Windows_NT') { '.bat' } else { '' } + $probes = @( + (Join-Path $Sdk "$RelNoExt$exe"), + (Join-Path $Sdk "$RelNoExt$bat"), + (Join-Path $Sdk $RelNoExt) + ) + if ($Extra) { $probes += $Extra } + foreach ($p in $probes) { if ($p -and (Test-Path $p)) { return $p } } + return $null +} + +function Get-Tools { + $sdk = Resolve-Sdk + $cmdlineExtra = @( + (Join-Path $sdk 'cmdline-tools\latest\bin\avdmanager.bat'), + (Join-Path $sdk 'cmdline-tools\latest\bin\sdkmanager.bat'), + 'C:\Android\cmdline-tools\latest\avdmanager.bat', + 'C:\Android\cmdline-tools\latest\sdkmanager.bat' + ) + [pscustomobject]@{ + Sdk = $sdk + Adb = Get-Tool $sdk 'platform-tools\adb' + Emulator = Get-Tool $sdk 'emulator\emulator' + AvdManager = Get-Tool $sdk 'cmdline-tools\latest\bin\avdmanager' ($cmdlineExtra | Where-Object { $_ -like '*avdmanager*' }) + SdkManager = Get-Tool $sdk 'cmdline-tools\latest\bin\sdkmanager' ($cmdlineExtra | Where-Object { $_ -like '*sdkmanager*' }) + } +} + +# --------------------------------------------------------------------------- +# AVD + device inspection +# --------------------------------------------------------------------------- +function Get-AvdHome { + if ($env:ANDROID_AVD_HOME) { return $env:ANDROID_AVD_HOME } + return (Join-Path $HOME '.android\avd') +} + +function Get-Avds { + param($Tools) + $names = @() + if ($Tools.Emulator) { $names = & $Tools.Emulator -list-avds 2>$null | Where-Object { $_ -and $_.Trim() } } + $avdHome = Get-AvdHome + $names | ForEach-Object { + $name = $_.Trim() + $api = 0; $tag = ''; $dev = '' + $cfg = Join-Path $avdHome "$name.avd\config.ini" + if (Test-Path $cfg) { + $lines = Get-Content $cfg + $sysdir = ($lines | Where-Object { $_ -like 'image.sysdir.1=*' }) -replace '.*=', '' + if ($sysdir -match 'android-(\d+)') { $api = [int]$Matches[1] } + if ($sysdir -match 'google_apis_playstore') { $tag = 'google_apis_playstore' } + elseif ($sysdir -match 'google_apis') { $tag = 'google_apis' } + elseif ($sysdir -match 'system-images/[^/]+/([^/]+)/') { $tag = $Matches[1] } + $tagLine = ($lines | Where-Object { $_ -like 'tag.id=*' }) -replace '.*=', '' + if (-not $tag -and $tagLine) { $tag = $tagLine } + $dev = (($lines | Where-Object { $_ -like 'hw.device.name=*' }) -replace '.*=', '') + } + [pscustomobject]@{ Name = $name; Api = $api; Tag = $tag; Device = $dev } + } +} + +function Get-RunningEmulators { + param($Tools) + & $Tools.Adb start-server 2>$null | Out-Null + $out = & $Tools.Adb devices 2>$null + $result = @() + foreach ($line in $out) { + if ($line -match '^(emulator-\d+)\s+device') { + $serial = $Matches[1] + $name = '' + $an = & $Tools.Adb -s $serial emu avd name 2>$null + if ($an) { $name = ($an | Where-Object { $_ -and $_.Trim() -ne 'OK' } | Select-Object -First 1).Trim() } + $booted = (& $Tools.Adb -s $serial shell getprop sys.boot_completed 2>$null | Out-String).Trim() + $result += [pscustomobject]@{ Serial = $serial; Avd = $name; Booted = ($booted -eq '1') } + } + } + return $result +} + +function Get-InstalledImages { + param($Tools) + $root = Join-Path $Tools.Sdk 'system-images' + if (-not (Test-Path $root)) { return @() } + $images = @() + Get-ChildItem $root -Directory | ForEach-Object { + $apiDir = $_.Name + Get-ChildItem $_.FullName -Directory | ForEach-Object { + $tag = $_.Name + Get-ChildItem $_.FullName -Directory | ForEach-Object { + $abi = $_.Name + $api = 0; if ($apiDir -match 'android-(\d+)') { $api = [int]$Matches[1] } + $images += [pscustomobject]@{ + Package = "system-images;$apiDir;$tag;$abi"; Api = $api; Tag = $tag; Abi = $abi + } + } + } + } + return $images +} + +# --------------------------------------------------------------------------- +# Actions +# --------------------------------------------------------------------------- +function Wait-Boot { + param($Tools, [string]$Serial, [int]$Timeout) + $deadline = (Get-Date).AddSeconds($Timeout) + Write-Host "Waiting for $Serial to finish booting (timeout ${Timeout}s)..." + & $Tools.Adb -s $Serial wait-for-device 2>$null + while ((Get-Date) -lt $deadline) { + $b = (& $Tools.Adb -s $Serial shell getprop sys.boot_completed 2>$null | Out-String).Trim() + if ($b -eq '1') { + Start-Sleep -Seconds 2 + & $Tools.Adb -s $Serial shell input keyevent 82 2>$null | Out-Null # dismiss keyguard + & $Tools.Adb -s $Serial shell wm dismiss-keyguard 2>$null | Out-Null + Write-Host "Boot complete: $Serial" + return $true + } + Start-Sleep -Seconds 3 + } + throw "Emulator $Serial did not finish booting within ${Timeout}s." +} + +function Select-Image { + param($Tools, [int]$MinApi, [switch]$NeedGoogle, [switch]$NeedPlay) + $imgs = Get-InstalledImages $Tools + if ($MinApi -gt 0) { $imgs = $imgs | Where-Object { $_.Api -ge $MinApi } } + if ($NeedPlay) { $imgs = $imgs | Where-Object { $_.Tag -eq 'google_apis_playstore' } } + elseif ($NeedGoogle) { $imgs = $imgs | Where-Object { $_.Tag -like 'google_apis*' } } + # Prefer x86_64/arm64 host-matching abi, then lowest API that still satisfies MinApi. + $imgs = $imgs | Sort-Object Api + return ($imgs | Select-Object -First 1) +} + +function New-Avd { + param($Tools, [string]$Name, [string]$Package, [string]$DeviceProfile) + if (-not $Tools.AvdManager) { throw "avdmanager not found; cannot create AVD '$Name'." } + if (-not $Package) { throw "No system image available to create AVD '$Name'. Install one via Android Studio SDK Manager, or run: sdkmanager 'system-images;android-34;google_apis;x86_64'" } + Write-Host "Creating AVD '$Name' from '$Package' (device: $DeviceProfile)..." + 'no' | & $Tools.AvdManager create avd -n $Name -k $Package -d $DeviceProfile --force 2>&1 | Write-Host + Write-Host "Created AVD '$Name'." +} + +function Start-Emu { + param($Tools, [string]$Name, [switch]$Cold, [switch]$Headless, [int]$Timeout, [switch]$DoWait) + $before = (Get-RunningEmulators $Tools).Serial + $emuArgs = @('-avd', $Name, '-no-boot-anim', '-netdelay', 'none', '-netspeed', 'full') + if ($Cold) { $emuArgs += '-no-snapshot-load' } + if ($Headless) { $emuArgs += '-no-window'; $emuArgs += '-gpu'; $emuArgs += 'swiftshader_indirect' } + Write-Host "Starting emulator: $($Tools.Emulator) $($emuArgs -join ' ')" + Start-Process -FilePath $Tools.Emulator -ArgumentList $emuArgs -WindowStyle Minimized | Out-Null + # Wait for a NEW emulator serial to appear that reports the target AVD name. + $deadline = (Get-Date).AddSeconds($Timeout) + $serial = $null + while ((Get-Date) -lt $deadline) { + Start-Sleep -Seconds 3 + $running = Get-RunningEmulators $Tools + $candidate = $running | Where-Object { $_.Avd -eq $Name -or $before -notcontains $_.Serial } + $match = $candidate | Where-Object { $_.Avd -eq $Name } | Select-Object -First 1 + if (-not $match) { $match = $candidate | Where-Object { $before -notcontains $_.Serial } | Select-Object -First 1 } + if ($match) { $serial = $match.Serial; break } + } + if (-not $serial) { throw "Emulator for AVD '$Name' did not register with adb within ${Timeout}s." } + Write-Host "Emulator online: $serial (avd=$Name)" + if ($DoWait) { Wait-Boot $Tools $serial $Timeout | Out-Null } + return $serial +} + +# --------------------------------------------------------------------------- +# Command dispatch +# --------------------------------------------------------------------------- +$tools = Get-Tools + +switch ($Command) { + 'resolve-sdk' { + Write-Host "SDK: $($tools.Sdk)" + Write-Host "adb: $($tools.Adb)" + Write-Host "emulator: $($tools.Emulator)" + Write-Host "avdmanager: $($tools.AvdManager)" + Write-Host "sdkmanager: $($tools.SdkManager)" + } + 'list' { + Write-Host "== AVDs ==" + Get-Avds $tools | Format-Table -AutoSize | Out-String | Write-Host + Write-Host "== Running ==" + Get-RunningEmulators $tools | Format-Table -AutoSize | Out-String | Write-Host + } + 'list-images' { + Get-InstalledImages $tools | Sort-Object Api | Format-Table -AutoSize | Out-String | Write-Host + } + 'status' { + $running = Get-RunningEmulators $tools + if ($Avd) { $running = $running | Where-Object { $_.Avd -eq $Avd } } + if (-not $running) { Write-Host 'No matching emulator running.'; exit 0 } + $running | Format-Table -AutoSize | Out-String | Write-Host + } + 'create' { + if (-not $Avd) { throw "Provide -Avd to create." } + $pkg = $Image + if (-not $pkg) { + $sel = Select-Image $tools -MinApi $ApiLevel -NeedGoogle:$RequireGoogleApis -NeedPlay:$RequirePlayStore + if ($sel) { $pkg = $sel.Package } + } + New-Avd $tools $Avd $pkg $Device + } + 'start' { + if (-not $Avd) { throw "Provide -Avd to start." } + $running = Get-RunningEmulators $tools | Where-Object { $_.Avd -eq $Avd } | Select-Object -First 1 + if ($running) { + Write-Host "Already running: $($running.Serial)" + if ($Wait) { Wait-Boot $tools $running.Serial $TimeoutSec | Out-Null } + Write-Host "SERIAL=$($running.Serial)"; exit 0 + } + $serial = Start-Emu $tools $Avd -Cold:$ColdBoot -Headless:$NoWindow -Timeout $TimeoutSec -DoWait:$Wait + Write-Host "SERIAL=$serial" + } + 'ensure' { + # 1) Pick or create an AVD that satisfies the feature's requirements. + $target = $null + $avds = Get-Avds $tools + if ($Avd) { + $target = $avds | Where-Object { $_.Name -eq $Avd } | Select-Object -First 1 + if (-not $target) { + $pkg = $Image + if (-not $pkg) { + $sel = Select-Image $tools -MinApi $ApiLevel -NeedGoogle:$RequireGoogleApis -NeedPlay:$RequirePlayStore + if ($sel) { $pkg = $sel.Package } + } + New-Avd $tools $Avd $pkg $Device + $target = [pscustomobject]@{ Name = $Avd } + } + } + else { + $match = $avds + if ($ApiLevel -gt 0) { $match = $match | Where-Object { $_.Api -ge $ApiLevel -or $_.Api -eq 0 } } + if ($RequirePlayStore) { $match = $match | Where-Object { $_.Tag -eq 'google_apis_playstore' } } + elseif ($RequireGoogleApis) { $match = $match | Where-Object { $_.Tag -like 'google_apis*' } } + # Prefer an already-running AVD among the matches. + $runningNames = (Get-RunningEmulators $tools).Avd + $target = $match | Where-Object { $runningNames -contains $_.Name } | Select-Object -First 1 + if (-not $target) { $target = $match | Sort-Object Api | Select-Object -First 1 } + if (-not $target) { + # No suitable AVD exists: create one. + $name = "android_auth_e2e" + if ($ApiLevel -gt 0) { $name += "_api$ApiLevel" } + $sel = Select-Image $tools -MinApi $ApiLevel -NeedGoogle:$RequireGoogleApis -NeedPlay:$RequirePlayStore + if (-not $sel) { throw "No installed system image satisfies the requirements (minApi=$ApiLevel, googleApis=$RequireGoogleApis, playStore=$RequirePlayStore). Install one first." } + New-Avd $tools $name $sel.Package $Device + $target = [pscustomobject]@{ Name = $name } + } + } + + # 2) Reuse if running, else start. Then wait for boot. + $running = Get-RunningEmulators $tools | Where-Object { $_.Avd -eq $target.Name } | Select-Object -First 1 + if ($running) { + Write-Host "Reusing running emulator: $($running.Serial) (avd=$($target.Name))" + $serial = $running.Serial + Wait-Boot $tools $serial $TimeoutSec | Out-Null + } + else { + $serial = Start-Emu $tools $target.Name -Cold:$ColdBoot -Headless:$NoWindow -Timeout $TimeoutSec -DoWait + } + Write-Host "AVD=$($target.Name)" + Write-Host "SERIAL=$serial" + } +} From fc7b5566774ba319f67be77aa019646fa411e82d Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:05:37 -0700 Subject: [PATCH 02/29] e2e skill: capture lessons learned from real runs Fold hard-won failures from actual E2E runs into the references so the next run doesn't rediscover them: the flights-manager-never-initialized trap (flag reads its source default), local-library publish-to-mavenLocal flow, NDK/CMake toolchain + ABI matching, wrong-keystore AADSTS50011, non-CA-approved app config AADSTS530021, WebView login-field focus/verification, Play Store install-button targeting, BACK-exits-app vs ESCAPE, special-char password typing, time-boxed sink-wait TTLs, and the broker telemetry keys (broker_app_used/request_eligible_for_broker) that are the clearest brokered pass/fail signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/log-signals.md | 20 ++++ .../references/troubleshooting.md | 105 ++++++++++++++++++ .../references/ui-interaction.md | 48 ++++++++ 3 files changed, 173 insertions(+) diff --git a/.github/skills/android-emulator-e2e-tester/references/log-signals.md b/.github/skills/android-emulator-e2e-tester/references/log-signals.md index 20a9d862..d6a59ee2 100644 --- a/.github/skills/android-emulator-e2e-tester/references/log-signals.md +++ b/.github/skills/android-emulator-e2e-tester/references/log-signals.md @@ -83,9 +83,29 @@ trip (correlation_id) → token saved → `executed successfully`. | `AADSTS50126` | Invalid username/password | Wrong test credential | | `AADSTS700016` | App not found in tenant | Wrong client id / tenant | | `AADSTS50058` | Silent sign-in failed, no session | Expected before an interactive sign-in | +| `AADSTS530021` | App not approved-client-app (CA) | Wrong/unapproved app config — see troubleshooting | `AADSTS50011`/`700016` typically indicate a **real defect** in the change under test → root-cause and hand to the fix loop. `AADSTS65001`/`50076`/`50058` are often **flow steps**, not defects. +`AADSTS530021` is an **environment/config** block (the signed-in app config isn't CA-approved) that stops +the flow *before* the step under test — switch to an approved app config; do not treat it as a defect. + +## Broker telemetry keys worth grepping + +Brokered flows emit per-request telemetry that is often the clearest pass/fail signal — grep the snapshot +for these keys (values are logged as `Key: , Value: <...>`): + +| Key | Meaning | +|---|---| +| `broker_app_used` | `true` = the request was actually serviced through the broker; `false` = it wasn't (e.g. broker ineligible / not found) | +| `request_eligible_for_broker` | `true` = the request passed broker-eligibility and was routed to the broker | +| `is_successful` | Overall request success as the SDK reports it | +| `api_status_code` | e.g. `RequiredBrokerMissing`, `UserCanceled`, a success status | +| `auth_flow` | `Broker` vs non-broker path taken | + +For a brokered scenario, `broker_app_used=true` + `request_eligible_for_broker=true` is a strong PASS +signal that the broker path executed; `broker_app_used=false` with a `RequiredBrokerMissing` status +points at broker discovery/eligibility rather than the token exchange. ## Judging success per flow diff --git a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md index bf9deab6..4050cf84 100644 --- a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md +++ b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md @@ -5,11 +5,28 @@ Table of contents: - [Emulator won't start or boot](#emulator-wont-start-or-boot) - [adb device issues](#adb-device-issues) - [Build failures](#build-failures) +- [Testing a local library change (publish to mavenLocal)](#testing-a-local-library-change-publish-to-mavenlocal) +- [Native (NDK/CMake) build toolchain](#native-ndkcmake-build-toolchain) - [Install failures](#install-failures) +- [Signing & redirect-URI mismatch (AADSTS50011)](#signing--redirect-uri-mismatch-aadsts50011) +- [Conditional Access blocks the flow (AADSTS530021, approved-client-app)](#conditional-access-blocks-the-flow-aadsts530021-approved-client-app) +- [Feature flags / flights not taking effect](#feature-flags--flights-not-taking-effect) - [uiautomator dump failures](#uiautomator-dump-failures) - [Broker pairing failures](#broker-pairing-failures) - [Distinguishing an env/test-harness problem from a real defect](#distinguishing-an-envtest-harness-problem-from-a-real-defect) +> **Lessons from real runs are folded into the sections below.** The five that cost the most time +> historically: (1) a feature flag that only reads its **source default** because the app never +> initializes the flights manager — see [Feature flags / flights](#feature-flags--flights-not-taking-effect); +> (2) the test app **signed with the wrong keystore** → `AADSTS50011` — see +> [Signing](#signing--redirect-uri-mismatch-aadsts50011); (3) a **non-CA-approved app config** → +> `AADSTS530021` before the flow under test is even reached — see +> [Conditional Access](#conditional-access-blocks-the-flow-aadsts530021-approved-client-app); +> (4) a change in a **library** (common/common4j) that must be **published to mavenLocal** and consumed +> by the test app — see [Testing a local library change](#testing-a-local-library-change-publish-to-mavenlocal); +> (5) a missing **NDK/CMake** toolchain for apps with native code — see +> [Native build toolchain](#native-ndkcmake-build-toolchain). + ## SDK / tooling not found - `emulator.ps1 resolve-sdk` prints the resolved SDK and tool paths. If it throws, `ANDROID_HOME` is @@ -53,6 +70,47 @@ Table of contents: - Prefer reusing an APK Android Studio already built (`appcontrol.ps1 list-apks`) instead of a fresh Gradle build when one exists — it's faster and avoids credential setup. +## Testing a local library change (publish to mavenLocal) + +When the change under test is in a **library** (`common` / `common4j`, or another SDK the test app +consumes as a Maven dependency), building the test app is **not** enough — the app resolves the library +from a Maven repo, not from your working tree. You must publish your local library, then build the app +against that version: + +1. **Publish the library to mavenLocal** from the working tree that has your change (a git *worktree* is + still the right source — publish from the folder that actually holds the edited code, not a sibling + checkout on a different branch). Use a unique version so the app can't silently pull a cached one: + ```powershell + $ver = "0.0.0-local--$(Get-Date -Format yyyyMMddHHmmss)" + ./gradlew :common4j:publishToMavenLocal "-PprojVersion=$ver" --console=plain + ./gradlew :common:publishDistReleasePublicationToMavenLocal "-PprojVersion=$ver" "-PdistCommon4jVersion=$ver" --console=plain + ``` +2. **Build/install the test app pointing at that version** — pass the app's version-override property + (e.g. `-PandroidCommonVersion=$ver`) and ensure `mavenLocal()` is on its repository list (some repos + inject this via a Gradle init script rather than editing the app). Confirm the artifact exists first: + `~/.m2/repository/com/microsoft/identity/common//`. +3. If the app repo doesn't already read `mavenLocal()`, add it via an **init script** (`--init-script`) + rather than committing a repo change. + +If a repo ships helper scripts for exactly this (e.g. a `scripts/local-common/` folder with +`publish-common-to-maven-local.ps1` + `build--with-local-common.ps1`), prefer them — they encode +the correct version-override property and init-script wiring. + +## Native (NDK/CMake) build toolchain + +Apps with native code (some test apps and OneAuth-based apps) fail the Gradle build if the NDK/CMake +toolchain is missing: + +- **Symptom:** `CMake ... was not found`, `NDK not configured`, or a `externalNativeBuild` task failure. +- **Fix:** install the exact CMake version the app's `build.gradle` pins (e.g. `cmake;3.31.5`) and a + matching NDK via `sdkmanager`, or Android Studio SDK Manager → SDK Tools. The `cmdline-tools` package + must be present for `sdkmanager` to run at all. +- **ABI:** build the ABI that matches the device. Emulators are usually `x86_64`; physical devices are + usually `arm64-v8a`. Pass the app's ABI-selection property (e.g. `-PabiSelection=x86_64`) or build a + universal APK so install doesn't fail with `INSTALL_FAILED_NO_MATCHING_ABIS`. +- If native build isn't needed for the segment you're testing, some apps accept `-PskipNativeBuild=true` + for a faster compile/resolve check — but a real functional run needs the native libs. + ## Install failures | adb error | Cause | Fix | @@ -63,6 +121,53 @@ Table of contents: | `INSTALL_FAILED_NO_MATCHING_ABIS` | APK ABI ≠ emulator ABI | Use an `x86_64` system image, or build a universal APK | | `INSTALL_FAILED_USER_RESTRICTED` | Play-protect / user prompt | Use a non-Play-Store `google_apis` image | +## Signing & redirect-URI mismatch (AADSTS50011) + +`AADSTS50011` ("redirect URI ... does not match") after a sign-in almost always means the test app is +**signed with a key the app registration doesn't recognize**. The eSTS-registered redirect URI encodes +the app's package name **and its signing-cert hash** (`msauth:///`), so a debug +build signed with the wrong keystore is rejected even though the package name is right. + +- **Confirm the installed signature:** `adb shell dumpsys package ` and look at the signing cert; + compare its Base64 SHA-1 to what the app registration expects. +- **Fix:** build/sign the app with the **registered keystore** for that app (some 1P test apps require a + team keystore fetched from secure storage rather than the default `~/.android/debug.keystore`). If you + can't obtain the right keystore, this is a **user blocker** — say so. +- This is an **environment/config** problem, not a defect in the feature under test — do not send it to + the code-fix loop. + +## Conditional Access blocks the flow (AADSTS530021, approved-client-app) + +`AADSTS530021` ("Application does not meet the approved-client-app requirement") means a Conditional +Access policy blocked the app **before** the flow you're trying to test could run. This happens when the +`clientId` / app configuration you signed in with is **not on the tenant's approved-client-app list**. + +- **Why it bites E2E:** you may be testing a MAM / CA-driven flow, but if the test app's configured + client id isn't CA-approved, eSTS returns `530021` and you never reach the step under test (e.g. the + install-broker prompt). +- **Fix:** run with an **approved** app configuration. Many test apps let you pick the app config / + client id at runtime (a spinner or a build flag). A well-known first-party client id with an OOB + redirect (e.g. an Office client id) is CA-approved and reaches CA-gated flows, whereas a custom test + client id is not. Pick the approved config for the scenario, or ask the user which config to use. +- Like `50011`, this is **environment/config**, not a code defect. + +## Feature flags / flights not taking effect + +A flag can silently read a value you didn't set. The common trap: **the app never initializes the +flights manager**, so every lookup falls back to the flag's **source default**. + +- **Symptom:** you enabled a flight (via test-app UI, config, or an override) but logs show the + flag-gated code path never runs, and no error explains why. +- **Diagnose:** find how the flight is read. If the code goes through a flights provider that must be + installed at startup (e.g. `CommonFlightsManager` / `setFlightsProvider(...)`) and the app never calls + that, the provider is the **default** one that just returns each flag's coded default. +- **Fix for a local run:** flip the flag's **source default** to the value you need (a temporary code + change in the library, e.g. `MyFlight("Key", true)`), publish + rebuild (see + [Testing a local library change](#testing-a-local-library-change-publish-to-mavenlocal)), and **revert + the change after the run**. Leave a `TODO: revert` marker and never commit or push the flip. +- See [mocking-flights-and-segments.md](mocking-flights-and-segments.md) for the full temp-change + + revert discipline. + ## uiautomator dump failures - "could not get idle state" / empty tree: the screen is animating or a secure window is up. Retry diff --git a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md index fd1acb2a..b4186276 100644 --- a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md +++ b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md @@ -85,6 +85,54 @@ and can hide fields from `uiautomator dump`. Mitigations: **Mode A** automation tests (they use instrumented hooks that bypass this). - If nothing works, treat it as a blocker and tell the user. +## Driving WebView login fields reliably (hard-won) + +The eSTS email/password pages are a **WebView**, not native widgets. Typing into them fails silently in +a few ways — these fixes came from real runs: + +- **Tap the field, then confirm focus before typing.** A blind `input-text` goes nowhere if no field is + focused. After tapping, verify focus by checking that the soft keyboard is up + (`adb shell dumpsys input_method | Select-String mInputShown,mServedView` — a WebView `mServedView` + with `mInputShown=true` means the field is focused), then re-`dump` to read the field node. +- **The visible field is lower than the heading.** On the password page, the big "Enter password" text is + *not* the input — the actual `EditText` sits below it. Tapping the heading does nothing. Get the real + node's bounds from the accessibility tree (`uiautomator dump` still exposes the WebView's `EditText` + node with a `resource-id` like `i0118` even when `FLAG_SECURE` blanks the screenshot) and tap its + center. +- **Verify the text landed before submitting.** For a password field, re-dump and check the node's + `text` length or `password="true"` state — do not tap **Sign in** until you've confirmed characters + were entered, or you'll loop on a blank field. +- **Type passwords with special characters literally.** `~ ! # $ & * ( )` are shell metacharacters. + `deviceui.ps1 input-text` escapes them, but if you call adb directly, single-quote the whole string + **device-side**: `adb shell "input text 'P@ss~!word'"`. Never echo the password into the transcript. +- **Some CP/broker WebViews reject programmatic submit entirely.** With `FLAG_SECURE`, typing may work + but tapping **Sign in** / pressing Enter doesn't register on an emulator. If a secure broker screen + won't submit, note it as a **harness limitation** (not a defect) and, if the segment under test is + already proven, stop there — see [mocking-flights-and-segments.md](mocking-flights-and-segments.md). + +## Play Store installs (referrer / broker-install flows) + +Installing an app from the Play Store during a flow (e.g. installing Company Portal for a MAM/CA flow): + +- **Target the real Install button, not the rating chip.** `tap-text "Install"` can hit the "Everyone" + content-rating label or an unrelated element. Read the button's bounds from `dump` and `tap-xy` its + center, and dismiss the interstitial **"Got it"** age-rating dialog first if it appears. +- **Installs are slow and interstitials vary.** Poll for completion by package presence + (`appcontrol.ps1 is-installed -Package `) in a loop rather than assuming a fixed delay. +- **Mind time-boxed waits.** If the feature parks a request with a TTL (e.g. a sink-wait that expires + after N minutes) and a slow emulator install blows past it, the original request may lapse. That's a + harness-timing artifact, not a defect: re-trigger the flow with the app already installed to exercise + the same resume path, and note the timing in the report. + +## Keyboard & navigation gotchas + +- **`BACK` can exit the app.** On a main screen with nothing to pop, `key BACK` (keyevent 4) closes the + activity/app. To only dismiss the soft keyboard, use `key ESCAPE` (keyevent 111), which hides the IME + without navigating back. +- **Re-dump after every action.** The tree changes; a stale dump makes you tap the wrong place. +- **Compute bounds-center for taps when `tap-text` is unreliable** (icon buttons, WebView chrome, + duplicate labels) — read `Bounds` from `dump` and `tap-xy` the midpoint. + ## Robustness tips - Always `wait-text` for the expected element before acting (default 20s) instead of fixed sleeps. From c9e70bbe5e18620b01f1fc0c8bedba9fa15d828e Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:10:17 -0700 Subject: [PATCH 03/29] e2e skill: add real devices to the device pool The skill was emulator-only; now the device pool also includes any real device connected over adb (USB or Wi-Fi). emulator.ps1 gains: physical-device enumeration (getprop API level, GMS/Play presence, boot state), a unified 'pool' command (-Json for tooling), physical devices shown in list/status, and ensure support for -PreferPhysical (favor a connected real device), -NoPhysical (emulators only), and -Serial (pin an exact device). Real devices are never created/booted - only used if already connected and booted, and only when they meet the feature's API/Google-APIs/Play requirements. Default ensure order: running emulator -> matching real device -> boot/create AVD. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 27 ++-- .../references/app-and-module-map.md | 4 +- .../scripts/emulator.ps1 | 134 ++++++++++++++++-- 3 files changed, 147 insertions(+), 18 deletions(-) diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index 95d51ee0..f342a397 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -1,6 +1,6 @@ --- name: android-emulator-e2e-tester -description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator. Finds or creates a suitable AVD and reuses a running one or boots it, builds/installs the right test app, drives the UI, and verifies via logcat. Use when asked to 'test this feature on the emulator', 'run the E2E test', 'verify the feature end to end', 'does this work on a device', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from the session's design spec, implementation diff, or user-provided test steps. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint) and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail and drives a fix-and-retest loop until it passes." +description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator or a connected real device. Builds a device pool from emulators and any adb-connected hardware, finds or creates a suitable AVD (or reuses a running emulator / real device), leases the device so concurrent tests don't collide, builds/installs the right test app, drives the UI, and verifies via logcat. Delegates the actual run to a sub-agent (same model as the parent) and waits for its verdict. Use when asked to 'test this feature on the emulator', 'run the E2E test', 'verify the feature end to end', 'does this work on a device', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from the session's design spec, implementation diff, or user-provided test steps. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint), mocks unavailable dependencies and sets feature flags via temporary code changes when needed, and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail and drives a fix-and-retest loop until it passes." --- # Android Emulator E2E Tester @@ -33,7 +33,7 @@ All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` | Script | Purpose | Key commands | |---|---|---| -| `emulator.ps1` | Emulator/AVD lifecycle | `ensure`, `status`, `list`, `list-images`, `create`, `start` | +| `emulator.ps1` | Device pool: emulators **and** real devices | `ensure`, `status`, `list`, `pool`, `list-images`, `create`, `start` | | `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | | `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `screenshot`, `current-app` | | `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | @@ -60,18 +60,29 @@ with an explicit, observable **success criterion** (e.g. "AcquireTokenSilent ret **Ask the user if** the scenario is ambiguous, multiple flows could be meant, or you cannot tell what "working" looks like. Do not guess a scenario when the intent is unclear. -### Phase 2 — Provision the emulator +### Phase 2 — Provision the device (emulator or real device) -Derive requirements from the feature (min API, Google Play services for broker/push flows) — see -"Emulator requirements per feature" in the app-and-module map — then one command finds-or-creates, -reuses-or-starts, and waits for boot: +The **device pool** is every usable target: emulators **and** any real device connected over adb. +`emulator.ps1 pool` lists them. Derive requirements from the feature (min API, Google Play services for +broker/push flows) — see "Emulator / real-device requirements per feature" in the app-and-module map — +then one command finds-or-creates, reuses-or-starts, and waits for boot: ```powershell ./scripts/emulator.ps1 ensure -RequireGoogleApis -ApiLevel 30 -Wait +# prefer a connected real device if one meets the requirements: +./scripts/emulator.ps1 ensure -RequireGoogleApis -PreferPhysical -Wait +# pin an exact device (emulator serial or real-device serial): +./scripts/emulator.ps1 ensure -Serial 39121FDJH003ZK # or target a specific AVD: ./scripts/emulator.ps1 ensure -Avd Pixel_7 -Wait +# stick to emulators only: add -NoPhysical ``` -Capture the printed `SERIAL=` and use it as `-Serial` everywhere. If provisioning fails, see -[references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, boot hang). +A connected, booted real device that satisfies the requirements is a valid target (real devices are +never created/booted — only used if already connected). By default `ensure` reuses a running emulator +first, then a matching real device, then boots/creates an AVD; `-PreferPhysical` flips it to try real +devices first. Capture the printed `SERIAL=` and use it as `-Serial` everywhere. **When multiple tests +may run concurrently, lease the device first** (see Phase 2a) so two runs don't collide. If provisioning +fails, see [references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, +boot hang). ### Phase 3 — Deploy the app-under-test diff --git a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md b/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md index a12cf4d9..c240939a 100644 --- a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md +++ b/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md @@ -114,9 +114,9 @@ Brokered auth requires **a calling app + a broker app**, both installed: - **Never** hardcode, print, or commit real credentials. Type them into the device only; never echo them into logs or the transcript. -## Emulator requirements per feature +## Emulator / real-device requirements per feature -Pass these to `emulator.ps1 ensure`: +Pass these to `emulator.ps1 ensure` (they gate both AVD selection and real-device eligibility): - **Broker / push / GMS-dependent flows** → `-RequireGoogleApis` (needs Google Play services for FCM, account manager). Prefer a `google_apis` or `google_apis_playstore` image. diff --git a/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 index fef1da9e..e436867f 100644 --- a/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 +++ b/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 @@ -1,9 +1,9 @@ # Copyright (c) Microsoft Corporation. All rights reserved. <# .SYNOPSIS - Android emulator lifecycle helper for E2E testing: resolve the SDK, list/inspect - AVDs and running devices, create an AVD, start one, wait for boot, or do all of - that in one shot with `ensure`. + Android device-pool helper for E2E testing: resolve the SDK, enumerate the device pool + (AVDs, running emulators, AND connected real devices), create an AVD, start one, wait for + boot, or do all of that in one shot with `ensure`. .DESCRIPTION Self-resolves the Android SDK location (handles unexpanded %LOCALAPPDATA% in @@ -11,8 +11,13 @@ (`emulator -list-avds`, reading the system-images/ dir) over the slow avdmanager/sdkmanager where possible. + The "device pool" is emulators PLUS any real devices connected over adb. `ensure` can hand + back a connected real device that meets the feature's requirements (use `-PreferPhysical` to + favor one, or `-NoPhysical` to stick to emulators). Real devices are never created/booted — + they're only used if already connected and booted. + .PARAMETER Command - resolve-sdk | list | list-images | status | create | start | ensure + resolve-sdk | list | list-images | status | pool | create | start | ensure .EXAMPLE # One-shot: guarantee a booted emulator meeting the feature's needs, print its serial. @@ -20,24 +25,30 @@ .EXAMPLE ./emulator.ps1 ensure -Avd Pixel_7 -Wait # use/create+start a named AVD + ./emulator.ps1 ensure -PreferPhysical -Wait # prefer a connected real device if one fits + ./emulator.ps1 pool # the whole device pool (emulators + real devices) ./emulator.ps1 status # what is running right now ./emulator.ps1 list-images # installed system images (for create) #> [CmdletBinding()] param( [Parameter(Position = 0)] - [ValidateSet('resolve-sdk', 'list', 'list-images', 'status', 'create', 'start', 'ensure')] + [ValidateSet('resolve-sdk', 'list', 'list-images', 'status', 'pool', 'create', 'start', 'ensure')] [string]$Command = 'status', [string]$Avd, # explicit AVD name to use/create + [string]$Serial, # pin an exact device (emulator serial or real-device serial) [int]$ApiLevel = 0, # minimum API level requirement (0 = any) [string]$Image, # explicit system-image package id for create [string]$Device = 'pixel_7', # device profile for create [switch]$RequireGoogleApis, # require google_apis or google_apis_playstore tag (GMS/push) [switch]$RequirePlayStore, # require google_apis_playstore tag + [switch]$PreferPhysical, # prefer a connected real device over booting an emulator + [switch]$NoPhysical, # exclude connected real devices from the pool [switch]$Wait, # wait for full boot [switch]$ColdBoot, # start with a clean state (-no-snapshot-load) [switch]$NoWindow, # headless (-no-window) + [switch]$Json, # machine-readable output (for `pool`) [int]$TimeoutSec = 300 ) @@ -142,6 +153,60 @@ function Get-RunningEmulators { return $result } +# Real hardware connected over adb (USB / Wi-Fi / cloud) — anything in 'device' state whose +# serial is not an emulator-XXXX console serial. These join the pool alongside emulators. +function Get-PhysicalDevices { + param($Tools) + & $Tools.Adb start-server 2>$null | Out-Null + $out = & $Tools.Adb devices 2>$null + $result = @() + foreach ($line in $out) { + # SERIALdevice — exclude emulators, the header, and offline/unauthorized states. + if ($line -match '^(\S+)\s+device(\s|$)' -and $line -notmatch '^emulator-' -and $Matches[1] -ne 'List') { + $serial = $Matches[1] + $model = (& $Tools.Adb -s $serial shell getprop ro.product.model 2>$null | Out-String).Trim() + $sdk = (& $Tools.Adb -s $serial shell getprop ro.build.version.sdk 2>$null | Out-String).Trim() + $api = 0; if ($sdk -match '^\d+$') { $api = [int]$sdk } + $gms = (& $Tools.Adb -s $serial shell pm list packages com.google.android.gms 2>$null | Out-String) + $play = (& $Tools.Adb -s $serial shell pm list packages com.android.vending 2>$null | Out-String) + $booted = (& $Tools.Adb -s $serial shell getprop sys.boot_completed 2>$null | Out-String).Trim() + $result += [pscustomobject]@{ + Serial = $serial + Model = $model + Api = $api + GoogleApis = ($gms -match 'com.google.android.gms') + PlayStore = ($play -match 'com.android.vending') + Booted = ($booted -eq '1') + } + } + } + return $result +} + +# Does a real device satisfy the feature's requirements? (Emulator reqs are checked via AVD tags.) +function Test-PhysicalMeetsReqs { + param($Dev, [int]$MinApi, [switch]$NeedGoogle, [switch]$NeedPlay) + if ($MinApi -gt 0 -and $Dev.Api -gt 0 -and $Dev.Api -lt $MinApi) { return $false } + if ($NeedPlay -and -not $Dev.PlayStore) { return $false } + if ($NeedGoogle -and -not $Dev.GoogleApis) { return $false } + return $true +} + +# Unified device pool: running emulators + connected real devices, as one list with a Type column. +function Get-DevicePool { + param($Tools, [switch]$NoPhysical) + $pool = @() + foreach ($e in (Get-RunningEmulators $Tools)) { + $pool += [pscustomobject]@{ Type = 'emulator'; Serial = $e.Serial; Name = $e.Avd; Api = $null; GoogleApis = $null; PlayStore = $null; Booted = $e.Booted } + } + if (-not $NoPhysical) { + foreach ($p in (Get-PhysicalDevices $Tools)) { + $pool += [pscustomobject]@{ Type = 'physical'; Serial = $p.Serial; Name = $p.Model; Api = $p.Api; GoogleApis = $p.GoogleApis; PlayStore = $p.PlayStore; Booted = $p.Booted } + } + } + return $pool +} + function Get-InstalledImages { param($Tools) $root = Join-Path $Tools.Sdk 'system-images' @@ -246,8 +311,23 @@ switch ($Command) { 'list' { Write-Host "== AVDs ==" Get-Avds $tools | Format-Table -AutoSize | Out-String | Write-Host - Write-Host "== Running ==" + Write-Host "== Running emulators ==" Get-RunningEmulators $tools | Format-Table -AutoSize | Out-String | Write-Host + if (-not $NoPhysical) { + Write-Host "== Connected real devices ==" + $phys = Get-PhysicalDevices $tools + if ($phys) { $phys | Format-Table -AutoSize | Out-String | Write-Host } else { Write-Host "(none)`n" } + } + } + 'pool' { + $pool = Get-DevicePool $tools -NoPhysical:$NoPhysical + if ($Json) { + $pool | ConvertTo-Json -Depth 4 + } + else { + if (-not $pool) { Write-Host 'Device pool is empty (no running emulator or connected real device).'; exit 0 } + $pool | Format-Table -AutoSize Type, Serial, Name, Api, GoogleApis, PlayStore, Booted | Out-String | Write-Host + } } 'list-images' { Get-InstalledImages $tools | Sort-Object Api | Format-Table -AutoSize | Out-String | Write-Host @@ -255,8 +335,10 @@ switch ($Command) { 'status' { $running = Get-RunningEmulators $tools if ($Avd) { $running = $running | Where-Object { $_.Avd -eq $Avd } } - if (-not $running) { Write-Host 'No matching emulator running.'; exit 0 } - $running | Format-Table -AutoSize | Out-String | Write-Host + $phys = if ($NoPhysical) { @() } else { Get-PhysicalDevices $tools } + if (-not $running -and -not $phys) { Write-Host 'No emulator running and no real device connected.'; exit 0 } + if ($running) { Write-Host "== Running emulators =="; $running | Format-Table -AutoSize | Out-String | Write-Host } + if ($phys) { Write-Host "== Connected real devices =="; $phys | Format-Table -AutoSize | Out-String | Write-Host } } 'create' { if (-not $Avd) { throw "Provide -Avd to create." } @@ -279,6 +361,31 @@ switch ($Command) { Write-Host "SERIAL=$serial" } 'ensure' { + # 0) If a specific device serial was pinned, verify it's connected and use it as-is. + if ($Serial) { + $pool = Get-DevicePool $tools -NoPhysical:$NoPhysical + $pinned = $pool | Where-Object { $_.Serial -eq $Serial } | Select-Object -First 1 + if (-not $pinned) { throw "Pinned device '$Serial' is not connected (not in the device pool)." } + if ($pinned.Type -eq 'emulator') { Wait-Boot $tools $Serial $TimeoutSec | Out-Null } + elseif (-not $pinned.Booted) { throw "Pinned real device '$Serial' is connected but not fully booted." } + Write-Host "Using pinned device: $Serial (type=$($pinned.Type))" + Write-Host "SERIAL=$Serial" + exit 0 + } + + # 0b) Prefer a connected real device when asked (and one meets the requirements). + if ($PreferPhysical -and -not $NoPhysical) { + $physMatch = Get-PhysicalDevices $tools | + Where-Object { $_.Booted -and (Test-PhysicalMeetsReqs $_ -MinApi $ApiLevel -NeedGoogle:$RequireGoogleApis -NeedPlay:$RequirePlayStore) } | + Select-Object -First 1 + if ($physMatch) { + Write-Host "Using connected real device: $($physMatch.Serial) (model=$($physMatch.Model), api=$($physMatch.Api))" + Write-Host "SERIAL=$($physMatch.Serial)" + exit 0 + } + Write-Host "No connected real device meets the requirements; falling back to an emulator." + } + # 1) Pick or create an AVD that satisfies the feature's requirements. $target = $null $avds = Get-Avds $tools @@ -302,6 +409,17 @@ switch ($Command) { # Prefer an already-running AVD among the matches. $runningNames = (Get-RunningEmulators $tools).Avd $target = $match | Where-Object { $runningNames -contains $_.Name } | Select-Object -First 1 + # Next best: a connected real device that meets the requirements (reuse beats a cold boot). + if (-not $target -and -not $NoPhysical) { + $physMatch = Get-PhysicalDevices $tools | + Where-Object { $_.Booted -and (Test-PhysicalMeetsReqs $_ -MinApi $ApiLevel -NeedGoogle:$RequireGoogleApis -NeedPlay:$RequirePlayStore) } | + Select-Object -First 1 + if ($physMatch) { + Write-Host "Using connected real device: $($physMatch.Serial) (model=$($physMatch.Model), api=$($physMatch.Api))" + Write-Host "SERIAL=$($physMatch.Serial)" + exit 0 + } + } if (-not $target) { $target = $match | Sort-Object Api | Select-Object -First 1 } if (-not $target) { # No suitable AVD exists: create one. From 18771ec75e4f6a782ed7bd77205b99676281977f Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:15:34 -0700 Subject: [PATCH 04/29] e2e skill: lease devices to prevent concurrent-test conflicts Adds devicelease.ps1 so parallel E2E runs don't collide on one device. A lease is a per-device JSON file (outside the repo, under ~/android-e2e-runs/.leases) recording serial, owner (agent/session id), feature, host, pid, and a heartbeat. acquire reaps abandoned leases, atomically claims a free device (create-new file wins the race), respects -MaxPoolSize, and boots/creates an emulator when the pool has no free device and the cap allows. Liveness is heartbeat-based: a lease older than -StaleMinutes (or a dead PID on the same host) is reclaimed on the next acquire/reap, so a device whose owning agent/session died is reused automatically. heartbeat/release/list/reap round out the lifecycle. Wires a Phase 2a (lease) + Phase 7 (release) into SKILL.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 30 ++ .../scripts/devicelease.ps1 | 307 ++++++++++++++++++ 2 files changed, 337 insertions(+) create mode 100644 .github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index f342a397..0223c19d 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -34,6 +34,7 @@ All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` | Script | Purpose | Key commands | |---|---|---| | `emulator.ps1` | Device pool: emulators **and** real devices | `ensure`, `status`, `list`, `pool`, `list-images`, `create`, `start` | +| `devicelease.ps1` | Lease a device so concurrent tests don't collide | `acquire`, `heartbeat`, `release`, `list`, `reap` | | `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | | `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `screenshot`, `current-app` | | `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | @@ -84,6 +85,30 @@ may run concurrently, lease the device first** (see Phase 2a) so two runs don't fails, see [references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, boot hang). +### Phase 2a — Lease the device (avoid collisions with other tests) + +Multiple E2E tests can run at once (different agents/sessions). To stop two runs from driving the same +device, **lease it** before use and **release it** when done: + +```powershell +# Acquire (reaps abandoned leases, picks a free device, or boots one if the pool cap allows). +# Use YOUR agent/session id as the owner so a dead agent's lease can be reclaimed. +$serial = (./scripts/devicelease.ps1 acquire -Owner $AgentId -Feature ` + -RequireGoogleApis -ApiLevel 30 -Wait | + Select-String '^SERIAL=(.+)$').Matches[0].Groups[1].Value +``` +- **Owner = your agent/session id** (or set `$env:E2E_AGENT_ID`). Liveness is heartbeat-based: refresh + during the run so a long test isn't reclaimed out from under you: + `./scripts/devicelease.ps1 heartbeat -Owner $AgentId -Serial $serial` (call it at each phase). +- **Pool cap:** `-MaxPoolSize N` (default 4) limits concurrent devices. If the pool is full and every + device is held by a **live** owner, acquire fails with guidance (wait, raise the cap, or connect a + device). If a holder's agent/session is **dead** (heartbeat older than `-StaleMinutes`, default 30), + its device is reclaimed automatically on the next `acquire`/`reap`. +- `acquire` will **boot/create an emulator** to add a device when the pool has no free one and the cap + allows — so it subsumes Phase 2's `emulator.ps1 ensure` (pass the same requirement flags). Use + `emulator.ps1 ensure` directly only when you deliberately don't want leasing. +- Do the whole run against the leased `$serial` (pass it as `-Serial` everywhere). + ### Phase 3 — Deploy the app-under-test 1. Decide the test surface and whether a **broker** must also be installed (brokered flows need a @@ -160,6 +185,11 @@ Summarize: scenario tested, target app/emulator, final verdict, iterations taken (success signal + correlation_id, or the blocker), and links to the run-folder artifacts (logs, screenshots). If blocked, state exactly what you need from the user to proceed. +**Release the device lease** so the next test can use it (do this even on failure/blocked): +```powershell +./scripts/devicelease.ps1 release -Owner $AgentId -Serial $serial +``` + ## When to ask the user Ask (don't guess) when: diff --git a/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 new file mode 100644 index 00000000..7489eb9d --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 @@ -0,0 +1,307 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +<# +.SYNOPSIS + Lease a device from the pool so concurrent E2E tests don't collide on the same emulator/real + device. Records who holds each device; reclaims a device automatically when its owning + agent/session is no longer alive; caps how many devices the pool may use at once. + +.DESCRIPTION + A lease is one small JSON file per device under: + $env:USERPROFILE\android-e2e-runs\.leases\.json + containing { serial, owner, feature, host, pid, startTime, heartbeat }. + + "owner" identifies the holder — pass your agent/session id so a dead agent's lease can be + reclaimed. Liveness is heartbeat-based: refresh the heartbeat at each test phase; a lease whose + heartbeat is older than -StaleMinutes is considered abandoned and is reclaimed on the next + acquire/reap. (A PID is also recorded and, when the owner is on this host, a dead PID makes the + lease immediately reclaimable.) + + `acquire` is one-stop: it reaps stale leases, picks a free device from the pool + (emulator.ps1 pool), respects -MaxPoolSize, and — if the pool has no free device but the cap + allows — boots/creates an emulator via emulator.ps1 to add one, then leases it. Lease creation is + atomic (create-new file) so two agents racing for the same free device can't both win. + +.PARAMETER Command + acquire | release | heartbeat | list | reap + +.EXAMPLE + # Get a leased, booted device for this agent (same reqs you'd pass emulator.ps1 ensure): + ./devicelease.ps1 acquire -Owner $AgentId -Feature mam-resume -RequireGoogleApis -ApiLevel 30 -Wait + # -> prints SERIAL=; use it for the whole run. + + ./devicelease.ps1 heartbeat -Owner $AgentId -Serial emulator-5554 # keep the lease fresh (each phase) + ./devicelease.ps1 release -Owner $AgentId -Serial emulator-5554 # when the test finishes + ./devicelease.ps1 list # who holds what + free devices + ./devicelease.ps1 reap # drop abandoned leases now + +.NOTES + Leases live OUTSIDE the repo (never committed). Always release when done; if the agent dies + without releasing, the lease self-expires after -StaleMinutes and becomes reclaimable. +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('acquire', 'release', 'heartbeat', 'list', 'reap')] + [string]$Command = 'list', + + [string]$Owner, # agent/session id holding the lease (identifies the holder) + [string]$Serial, # pin/target a specific device serial + [string]$Feature, # free-text label for what's being tested + [int]$MaxPoolSize = 4, # cap on how many devices may be leased at once + [int]$StaleMinutes = 30, # a lease older than this (no heartbeat) is reclaimable + [int]$ApiLevel = 0, # passed through to emulator.ps1 when booting a new device + [switch]$RequireGoogleApis, + [switch]$RequirePlayStore, + [switch]$PreferPhysical, + [switch]$NoPhysical, + [string]$Avd, + [switch]$NoBoot, # don't boot a new emulator if the pool has no free device + [switch]$Wait, # wait for boot when a device is (re)used + [switch]$Force, # release/reap even if owner doesn't match + [int]$TimeoutSec = 300 +) + +$ErrorActionPreference = 'Stop' + +$LeaseDir = Join-Path $env:USERPROFILE 'android-e2e-runs\.leases' +$EmulatorScript = Join-Path $PSScriptRoot 'emulator.ps1' + +function Ensure-LeaseDir { if (-not (Test-Path $LeaseDir)) { New-Item -ItemType Directory -Force -Path $LeaseDir | Out-Null } } +function Get-LeaseFile { param([string]$S) Join-Path $LeaseDir (($S -replace '[:/\\.]', '_') + '.json') } + +function Resolve-Owner { + param([string]$Explicit) + if ($Explicit) { return $Explicit } + if ($env:E2E_AGENT_ID) { return $env:E2E_AGENT_ID } + return "$env:COMPUTERNAME-$PID" +} + +function Get-Adb { + foreach ($v in @($env:ANDROID_HOME, $env:ANDROID_SDK_ROOT)) { + if ($v) { + $p = Join-Path ([Environment]::ExpandEnvironmentVariables($v)) 'platform-tools\adb.exe' + if (Test-Path $p) { return $p } + $p2 = Join-Path ([Environment]::ExpandEnvironmentVariables($v)) 'platform-tools/adb' + if (Test-Path $p2) { return $p2 } + } + } + foreach ($root in @((Join-Path $env:LOCALAPPDATA 'Android\Sdk'), (Join-Path $HOME 'AppData\Local\Android\Sdk'), (Join-Path $HOME 'Library/Android/sdk'), (Join-Path $HOME 'Android/Sdk'))) { + foreach ($exe in @('platform-tools\adb.exe', 'platform-tools/adb')) { + $p = Join-Path $root $exe + if ($root -and (Test-Path $p)) { return $p } + } + } + $cmd = Get-Command adb -ErrorAction SilentlyContinue + if ($cmd) { return $cmd.Source } + throw "adb not found. Set ANDROID_HOME to your SDK root." +} + +function Get-Pool { + # Ask emulator.ps1 for the current pool (running emulators + connected real devices). + if (-not (Test-Path $EmulatorScript)) { throw "emulator.ps1 not found next to devicelease.ps1 ($EmulatorScript)." } + $json = & $EmulatorScript pool -Json -NoPhysical:$NoPhysical 2>$null | Out-String + if (-not $json.Trim()) { return @() } + $parsed = $json | ConvertFrom-Json + if ($null -eq $parsed) { return @() } + if ($parsed -isnot [array]) { $parsed = @($parsed) } + return $parsed +} + +function Get-Leases { + Ensure-LeaseDir + Get-ChildItem $LeaseDir -Filter '*.json' -ErrorAction SilentlyContinue | ForEach-Object { + try { $o = Get-Content $_.FullName -Raw | ConvertFrom-Json; $o | Add-Member -NotePropertyName _File -NotePropertyValue $_.FullName -Force; $o } + catch { } # ignore a half-written lease file; it'll be rewritten/reaped + } +} + +function Test-OwnerAlive { + # Heartbeat is the authority. On the same host we can also treat a dead recorded PID as dead. + param($Lease) + $ageMin = ((Get-Date) - [datetime]$Lease.heartbeat).TotalMinutes + if ($ageMin -gt $StaleMinutes) { return $false } + if ($Lease.host -eq $env:COMPUTERNAME -and $Lease.pid) { + if (-not (Get-Process -Id $Lease.pid -ErrorAction SilentlyContinue)) { + # PID gone, but only call it dead if the heartbeat is also not fresh (< 2 min protects a + # brand-new lease taken by a short-lived shell whose PID already exited). + if ($ageMin -gt 2) { return $false } + } + } + return $true +} + +function Reap-StaleLeases { + $pool = Get-Pool + $poolSerials = @($pool | ForEach-Object { $_.Serial }) + $reaped = @() + foreach ($lease in (Get-Leases)) { + $dead = -not (Test-OwnerAlive $lease) + $gone = ($poolSerials -notcontains $lease.serial) # device no longer connected -> lease moot + if ($dead -or $gone) { + Remove-Item $lease._File -Force -ErrorAction SilentlyContinue + $reason = if ($dead) { 'stale/dead-owner' } else { 'device-disconnected' } + $reaped += [pscustomobject]@{ Serial = $lease.serial; Owner = $lease.owner; Reason = $reason } + } + } + return $reaped +} + +function Try-WriteLease { + # Atomically create the lease file; return $true only if WE created it (won the race). + param([string]$S, [string]$OwnerId, [string]$Feat) + $file = Get-LeaseFile $S + $payload = [pscustomobject]@{ + serial = $S + owner = $OwnerId + feature = $Feat + host = $env:COMPUTERNAME + pid = $PID + startTime = (Get-Date).ToString('o') + heartbeat = (Get-Date).ToString('o') + } | ConvertTo-Json + try { + $fs = [System.IO.File]::Open($file, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None) + try { + $bytes = [System.Text.Encoding]::UTF8.GetBytes($payload) + $fs.Write($bytes, 0, $bytes.Length) + } + finally { $fs.Dispose() } + return $true + } + catch [System.IO.IOException] { + return $false # someone else holds it (file already exists) + } +} + +$adb = Get-Adb +function Wait-Booted { param([string]$S) & $adb -s $S wait-for-device 2>$null; $b = (& $adb -s $S shell getprop sys.boot_completed 2>$null | Out-String).Trim(); return ($b -eq '1') } + +switch ($Command) { + 'list' { + $leases = @(Get-Leases) + $pool = Get-Pool + Write-Host "== Active leases ($($leases.Count)) ==" + if ($leases) { + $leases | ForEach-Object { + $ageMin = [math]::Round(((Get-Date) - [datetime]$_.heartbeat).TotalMinutes, 1) + [pscustomobject]@{ Serial = $_.serial; Owner = $_.owner; Feature = $_.feature; Host = $_.host; AgeMin = $ageMin; Alive = (Test-OwnerAlive $_) } + } | Format-Table -AutoSize | Out-String | Write-Host + } + else { Write-Host "(none)`n" } + $leasedSerials = @($leases | ForEach-Object { $_.serial }) + $free = @($pool | Where-Object { $leasedSerials -notcontains $_.Serial }) + Write-Host "== Free devices in pool ($($free.Count)) ==" + if ($free) { $free | Format-Table -AutoSize Type, Serial, Name, Booted | Out-String | Write-Host } else { Write-Host "(none)`n" } + } + + 'reap' { + $reaped = Reap-StaleLeases + if ($reaped) { Write-Host "Reclaimed $($reaped.Count) lease(s):"; $reaped | Format-Table -AutoSize | Out-String | Write-Host } + else { Write-Host "No stale/abandoned leases to reclaim." } + } + + 'heartbeat' { + if (-not $Serial) { throw "Provide -Serial." } + $owner = Resolve-Owner $Owner + $file = Get-LeaseFile $Serial + if (-not (Test-Path $file)) { throw "No lease exists for $Serial (nothing to heartbeat). Re-acquire." } + $lease = Get-Content $file -Raw | ConvertFrom-Json + if ($lease.owner -ne $owner -and -not $Force) { throw "Lease on $Serial is held by '$($lease.owner)', not '$owner'. Use -Force to override." } + $lease.heartbeat = (Get-Date).ToString('o') + $lease | Select-Object serial, owner, feature, host, pid, startTime, heartbeat | ConvertTo-Json | Set-Content -Path $file -Encoding utf8 + Write-Host "Heartbeat updated for $Serial (owner=$owner)." + } + + 'release' { + if (-not $Serial) { throw "Provide -Serial." } + $owner = Resolve-Owner $Owner + $file = Get-LeaseFile $Serial + if (-not (Test-Path $file)) { Write-Host "No lease for $Serial (already released)."; exit 0 } + $lease = Get-Content $file -Raw | ConvertFrom-Json + if ($lease.owner -ne $owner -and -not $Force) { throw "Lease on $Serial is held by '$($lease.owner)', not '$owner'. Use -Force to override." } + Remove-Item $file -Force + Write-Host "Released $Serial (owner=$owner)." + } + + 'acquire' { + $owner = Resolve-Owner $Owner + Ensure-LeaseDir + + # 1) Reclaim anything abandoned before we count active leases. + $reaped = Reap-StaleLeases + if ($reaped) { Write-Host "Reclaimed $($reaped.Count) abandoned lease(s) before acquiring: $((($reaped | ForEach-Object { "$($_.Serial)[$($_.Reason)]" }) -join ', '))" } + + $leases = @(Get-Leases) + $leasedSerials = @($leases | ForEach-Object { $_.serial }) + $pool = Get-Pool + + # 2) If this owner already holds a lease and no specific serial was requested, reuse it (idempotent). + if (-not $Serial) { + $mine = $leases | Where-Object { $_.owner -eq $owner } | Select-Object -First 1 + if ($mine) { + & (Join-Path $PSScriptRoot 'devicelease.ps1') heartbeat -Serial $mine.serial -Owner $owner | Out-Null + if ($Wait) { Wait-Booted $mine.serial | Out-Null } + Write-Host "Reusing existing lease for this owner." + Write-Host "SERIAL=$($mine.serial)" + exit 0 + } + } + + # 3) Choose a candidate free device, or boot one if the cap allows. + $target = $null + if ($Serial) { + $held = $leases | Where-Object { $_.serial -eq $Serial } | Select-Object -First 1 + if ($held -and $held.owner -ne $owner) { throw "Requested device $Serial is leased by '$($held.owner)'." } + $target = $Serial + } + else { + $free = @($pool | Where-Object { $leasedSerials -notcontains $_.Serial }) + # Prefer an already-booted device; among those, honor -PreferPhysical. + $booted = @($free | Where-Object { $_.Booted }) + if ($PreferPhysical) { $booted = @($booted | Sort-Object { if ($_.Type -eq 'physical') { 0 } else { 1 } }) } + if ($booted.Count -gt 0) { $target = $booted[0].Serial } + elseif ($free.Count -gt 0) { $target = $free[0].Serial } + } + + # 4) If nothing free, either boot a new emulator (cap permitting) or fail with guidance. + if (-not $target) { + if ($leases.Count -ge $MaxPoolSize) { + throw "Device pool is at capacity ($($leases.Count)/$MaxPoolSize) and every device is held by a live owner. Wait and retry, raise -MaxPoolSize, connect another device, or release a lease. Current holders:`n" + (($leases | ForEach-Object { " $($_.serial) <- $($_.owner) ($($_.feature))" }) -join "`n") + } + if ($NoBoot) { throw "No free device in the pool and -NoBoot was set. Connect/boot a device or drop -NoBoot." } + Write-Host "No free device; booting one via emulator.ps1 ensure (active leases $($leases.Count)/$MaxPoolSize)..." + $ensureArgs = @('ensure', '-Wait') + if ($Avd) { $ensureArgs += @('-Avd', $Avd) } + if ($ApiLevel -gt 0) { $ensureArgs += @('-ApiLevel', $ApiLevel) } + if ($RequireGoogleApis) { $ensureArgs += '-RequireGoogleApis' } + if ($RequirePlayStore) { $ensureArgs += '-RequirePlayStore' } + if ($PreferPhysical) { $ensureArgs += '-PreferPhysical' } + if ($NoPhysical) { $ensureArgs += '-NoPhysical' } + $out = & $EmulatorScript @ensureArgs 2>&1 + $out | Write-Host + $serialLine = ($out | Select-String '^SERIAL=(.+)$' | Select-Object -Last 1) + if (-not $serialLine) { throw "emulator.ps1 ensure did not yield a SERIAL=. Cannot acquire a device." } + $target = $serialLine.Matches[0].Groups[1].Value.Trim() + } + + # 5) Claim it atomically. If we lost the race, retry the whole acquire once. + if (-not (Try-WriteLease $target $owner $Feature)) { + Write-Host "Lost the race for $target (another agent claimed it); retrying acquire..." + $retryArgs = @('acquire', '-Owner', $owner, '-MaxPoolSize', $MaxPoolSize, '-StaleMinutes', $StaleMinutes, '-TimeoutSec', $TimeoutSec) + if ($Feature) { $retryArgs += @('-Feature', $Feature) } + if ($ApiLevel -gt 0) { $retryArgs += @('-ApiLevel', $ApiLevel) } + if ($RequireGoogleApis) { $retryArgs += '-RequireGoogleApis' } + if ($RequirePlayStore) { $retryArgs += '-RequirePlayStore' } + if ($PreferPhysical) { $retryArgs += '-PreferPhysical' } + if ($NoPhysical) { $retryArgs += '-NoPhysical' } + if ($NoBoot) { $retryArgs += '-NoBoot' } + if ($Wait) { $retryArgs += '-Wait' } + & (Join-Path $PSScriptRoot 'devicelease.ps1') @retryArgs + exit $LASTEXITCODE + } + + if ($Wait) { Wait-Booted $target | Out-Null } + Write-Host "Leased $target for owner=$owner (feature=$Feature). Refresh with 'heartbeat', free with 'release'." + Write-Host "SERIAL=$target" + } +} From f9b26fe149512e5c86786a17b0497636e3773afa Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:16:49 -0700 Subject: [PATCH 05/29] e2e skill: delegate the run to a sub-agent and supervise it Adds an execution-model section: the parent launches ONE sub-agent to own the long device-driving run, on the SAME model as the parent (e.g. parent on Claude Opus 4.8 -> sub-agent on Claude Opus 4.8), with a self-contained prompt that must end in a PASS/FAIL/BLOCKED verdict + evidence. The parent then WAITS for that verdict (block on completion) and never reports 'done' before it - directly fixing the failure we hit where the parent returned early and the test result was never shown. A watchdog restarts a hung sub-agent (no progress within a timeout and test not done), terminates it once the result is in, caps restarts, and escalates with evidence. Only the parent talks to the user. Reinforced in Guardrails. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index 0223c19d..328c5b0c 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -39,6 +39,37 @@ All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` | `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `screenshot`, `current-app` | | `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | +## Execution model — run inside a sub-agent, and supervise it + +An E2E run is long and chatty (boot, install, dozens of UI dumps/taps, log scans). **Delegate the actual +device-driving to a sub-agent** so the parent's context stays clean and the parent supervises rather than +performs. + +1. **Launch one sub-agent to own the run, on the SAME model as the parent.** If the parent is running on + Claude Opus 4.8, request the sub-agent on Claude Opus 4.8 (match whatever model the parent is on). + Give it a **self-contained** prompt: the scenario + explicit success criterion (Phase 1 output), the + leased `$serial` (or tell it to lease one with **its own agent id** as owner), the app/module + + package, credential-handling rules (type on device, never print), the run-folder path, any + mocks/flights to apply, and an instruction to **finish with a `PASS` / `FAIL` / `BLOCKED` verdict plus + evidence** (success signal + correlation_id, or the exact blocker). +2. **The parent WAITS for the sub-agent to finish.** Start it in the background, then block on its + completion (read its result with wait=true). **Do not end your turn, and do not report to the user, + until the sub-agent's verdict is in.** This is the #1 real failure mode we hit: the parent returned + early, the sub-agent's result and analysis were never surfaced to the user. +3. **Supervise with a watchdog while waiting:** + - **Hang → restart.** The sub-agent should emit a progress line and refresh its device lease + (`devicelease.ps1 heartbeat`) at each phase. If there's no progress within a timeout (e.g. ~10–15 min + for an install-heavy flow) **and the test isn't done**, treat it as hung: nudge once; if still stuck, + **restart** it (fresh sub-agent, resume from the last known state) rather than waiting forever. + - **Result in → stop.** If the verdict/evidence is already available, **stop waiting / terminate** the + sub-agent and move to reporting — don't let a finished run linger. + - **Cap restarts** (e.g. 2). If still unresolved, escalate to the user with the full evidence trail. +4. **Only the parent reports to the user.** Collect the sub-agent's verdict + artifact links and present + the Phase 7 report yourself. + +If the harness can't spawn a sub-agent (or you're already the deepest agent), run the phases inline — but +keep the same discipline: **don't declare done until the verdict is in, and don't loop forever on a hang.** + ## Workflow Execute in order. Announce a one-line status at each phase. @@ -219,6 +250,10 @@ Load these as needed (don't preload all): - **Never** print, log, or commit real or lab credentials, tokens, or secrets. Type them onto the device only. - **Artifacts stay outside the repo** (the run folder). Never commit logs/screenshots. +- **When delegating to a sub-agent, wait for its verdict** before reporting or ending the turn; restart it + if it hangs and the test isn't done, terminate it once the result is in. Never surface "done" without the + sub-agent's PASS/FAIL/BLOCKED evidence. +- **Lease the device before use and release it after** (even on failure) so concurrent runs don't collide. - **Environment problems are not code defects** — fix setup and re-run; only hand real defects to the fix loop. - **Cap the loop** (3–5 iterations) and escalate with evidence rather than looping indefinitely. - **Require a positive success signal** matching the scenario before declaring PASS. From b189b1827ece9c80260fe3fa7171aac18d7dc04c Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:18:54 -0700 Subject: [PATCH 06/29] e2e skill: mock unavailable data, set flights, and test in segments New reference mocking-flights-and-segments.md plus SKILL.md wiring so a run isn't abandoned as 'blocked' when the block isn't a defect. Covers: setting feature flags (runtime override first, else flip the source default temporarily and re-publish); mocking unavailable data/dependencies at the highest-fidelity boundary (local mock server for an undeployed API, inject a not-yet-shipped response field, mock brokers, adb intent extras) matched to the real contract; and segment-testing a flow when a middle piece can't be mocked (prove up to the boundary, then feed boundary inputs and prove the rest). Enforces a revert discipline: temp code changes stay uncommitted, carry a TODO REVERT marker, and are reverted with a clean-tree check. Softens the 'implementation incomplete' blocker to try mock/segment first, and reports what was real vs mocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 14 ++- .../mocking-flights-and-segments.md | 93 +++++++++++++++++++ 2 files changed, 106 insertions(+), 1 deletion(-) create mode 100644 .github/skills/android-emulator-e2e-tester/references/mocking-flights-and-segments.md diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index 328c5b0c..9d130c7e 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -171,6 +171,12 @@ expected element → act (`tap-text` / `input-text` / `key` / `finger`) → re-` Allow/Yes, pick an account, grant permissions, simulate a fingerprint (`finger`), enter a TOTP if the seed is known, set/enter a device PIN. **Do not** print or commit credentials. +**Set flags & mock what's missing (don't fake a pass).** If the scenario needs a feature flag on, or a +step depends on data/a dependency you can't produce naturally (a server API not deployed yet, a +collaborator app you can't drive), set the flag and mock the missing piece — including **temporary code +changes** that you revert afterward. If a middle piece genuinely can't be mocked, test the flow in +**segments**. See [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md). + **Stop and ask the user** only for genuine blockers (see the consolidated list below). ### Phase 5 — Verify from logs @@ -228,7 +234,10 @@ Ask (don't guess) when: - A **blocker** needs a human: real push-notification MFA on another device, SMS/phone OTP, a hardware key/NFC/QR/camera, CAPTCHA, a credential the AI doesn't have, or a tenant/CA policy it can't provision. - The **implementation is incomplete** for the path under test (stubs, `TODO`, missing wiring, feature - flag off with no way to enable) — report the gap instead of forcing a fake pass. + flag off with no way to enable) — **first try** to set the flag (temp code change) and mock/segment + around a missing middle piece (see + [mocking-flights-and-segments.md](references/mocking-flights-and-segments.md)); report the gap and ask + only if the feature genuinely can't be exercised even in segments. - **Environment setup** is missing and only the user can fix it (Maven creds/PAT, no sub-repo checkout, no system image, no lab account). @@ -244,6 +253,7 @@ Load these as needed (don't preload all): | [references/app-and-module-map.md](references/app-and-module-map.md) | Choosing/deploying the test app, package discovery, broker pairing, credentials, emulator requirements | | [references/log-signals.md](references/log-signals.md) | Interpreting logcat, success/failure patterns, AADSTS codes, per-flow pass criteria, eSTS correlation | | [references/ui-interaction.md](references/ui-interaction.md) | Driving auth screens, AI-vs-human inputs, FLAG_SECURE gotcha, selector strategy | +| [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md) | A flag must be set, a dependency/server data is unavailable, or the flow can't run fully E2E (mock it or test in segments) | | [references/troubleshooting.md](references/troubleshooting.md) | Emulator/build/install/uiautomator/broker failures; env-vs-defect triage | ## Guardrails @@ -254,6 +264,8 @@ Load these as needed (don't preload all): if it hangs and the test isn't done, terminate it once the result is in. Never surface "done" without the sub-agent's PASS/FAIL/BLOCKED evidence. - **Lease the device before use and release it after** (even on failure) so concurrent runs don't collide. +- **Temp changes for mocks/flags stay uncommitted and get reverted** — never commit/push a flag flip or a + mock; leave a `TODO: REVERT` marker and confirm the tree is clean before finishing. - **Environment problems are not code defects** — fix setup and re-run; only hand real defects to the fix loop. - **Cap the loop** (3–5 iterations) and escalate with evidence rather than looping indefinitely. - **Require a positive success signal** matching the scenario before declaring PASS. diff --git a/.github/skills/android-emulator-e2e-tester/references/mocking-flights-and-segments.md b/.github/skills/android-emulator-e2e-tester/references/mocking-flights-and-segments.md new file mode 100644 index 00000000..7f3425f0 --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/references/mocking-flights-and-segments.md @@ -0,0 +1,93 @@ +# Mocking, Flights & Segment Testing — Getting an E2E Run Unblocked + +Table of contents: +- [Principle: make it testable, don't fake a pass](#principle-make-it-testable-dont-fake-a-pass) +- [Temporary code changes — the revert discipline](#temporary-code-changes--the-revert-discipline) +- [Setting feature flags / flights](#setting-feature-flags--flights) +- [Mocking unavailable data or dependencies](#mocking-unavailable-data-or-dependencies) +- [Segment testing when the flow can't run end to end](#segment-testing-when-the-flow-cant-run-end-to-end) +- [Reporting mocked / segmented runs](#reporting-mocked--segmented-runs) + +When a run is blocked by something that isn't a defect in the code under test — a feature flag that's +off, a server API that isn't deployed yet, a piece in the middle that isn't implemented — **don't stop at +"blocked" and don't fake a pass.** Get as much of the feature under real test as you can by setting the +flag, mocking the missing input, or testing the flow in segments. Temporary code changes are allowed for +this, provided they're reverted and never committed. + +## Principle: make it testable, don't fake a pass + +- Prefer the **most real** option available: real data > a faithful mock at the network boundary > + a stub at the client boundary > a hardcoded value. The closer to real, the more the test proves. +- A mock must match the **actual contract** (field names, types, status codes) you're standing in for — + a mock that doesn't match the agreed server/API shape proves nothing. +- Never let a mock silently turn a red flow green. State clearly (in the report) which parts were real + and which were mocked, and what that means for confidence. + +## Temporary code changes — the revert discipline + +Making a temp change (flip a flag default, stub a response, inject a value) is fine **only** with this +discipline: + +1. Keep the change **in the working tree only** — never `git commit`/`git push` it, and never stage it + into a feature commit. +2. Leave a searchable marker at the edit site, e.g. `// TODO: REVERT — E2E mock, do not commit`. +3. **Revert after the run** — `git restore ` (or `git stash` the change while you commit real + work, then drop it). Verify with `git status` that the tree is clean of test-only edits before you + finish. +4. If the change is in a **library** the app consumes as a dependency, remember you must re-publish + + rebuild for it to take effect — see + [troubleshooting.md → Testing a local library change](troubleshooting.md#testing-a-local-library-change-publish-to-mavenlocal). + +## Setting feature flags / flights + +Get the flag into the state the scenario needs, cheapest option first: + +1. **A runtime override the app already exposes** — a test-app UI toggle, a config/spinner, `adb shell + setprop`, or a flights provider you can set at startup. Use this if it exists; no code change needed. +2. **Flip the flag's source default (temp).** If the app never installs a flights provider so every + lookup returns the coded default (a real gap we hit — see + [troubleshooting.md → Feature flags / flights](troubleshooting.md#feature-flags--flights-not-taking-effect)), + the only lever is the **default** in the library: `MyFlight("Key", true)`. Flip it, re-publish + + rebuild, run, then **revert**. Never commit the flip; the shipped default must stay as designed. +3. **Multiple flags:** set every flag the path reads, and confirm from logs that the flag-gated branch + actually executes (don't assume the flip took — verify the code path ran). + +## Mocking unavailable data or dependencies + +When a step needs data you can't produce naturally (a server response for an API that isn't deployed, a +service you can't trigger from the device), mock it at the **highest-fidelity** point you can: + +| Situation | Mock approach | +|---|---| +| Server API / endpoint not deployed yet | Point the client at a **local mock server** returning the agreed JSON, or use the repo's existing `MockWebServer`/interceptor test infra if present. Match the real contract exactly. | +| A single response **field** the server will add later | Inject it at the client's response-parsing boundary (temp code), using the agreed field name/shape, so the downstream path runs. | +| A collaborating app/broker you can't drive | Use a **mock broker** (`:mockcp`, `:mockauthapp`, `:mockltw` — see the app-and-module map) instead of the real one. | +| Data normally fetched at runtime (account, config) | Feed it via **adb intent extras** or a test hook the app exposes, or preload cache/prefs the app reads. | +| A push/callback you can't originate | Simulate the inbound intent/broadcast with `adb shell am start`/`am broadcast` carrying the expected payload. | + +Rules: the mock must be faithful to the real contract; gate it behind the feature flag or a debug hook +where possible; mark and revert it (above); and record in the report that the segment ran against a mock. + +## Segment testing when the flow can't run end to end + +If a piece in the middle is genuinely missing and **can't** be mocked faithfully, don't declare the whole +feature blocked — **test the segments that can run**, each with its own success criterion: + +1. **Segment before the gap:** drive the flow up to the boundary and assert the correct outputs at that + boundary (the right request was made, the right state/telemetry was produced, the parked/queued item + exists). That proves everything up to the gap. +2. **Segment after the gap:** feed the boundary inputs the missing piece *would* have produced (via a + mock/fixture) and assert the rest of the flow proceeds correctly. That proves everything after the gap. +3. Together the two segments cover the feature minus the exact missing piece; name the gap precisely and + what still needs an end-to-end pass once it's available. + +This is strictly better than "blocked": you localize exactly what's untested (the gap) and prove the rest. + +## Reporting mocked / segmented runs + +In the Phase 7 report, always state: +- **What was real vs mocked** (which flag was flipped, which data/dependency was mocked, at which boundary). +- **Which segments passed** and the success signal for each (with correlation_ids where relevant). +- **The remaining gap** — the exact piece that still needs a true end-to-end pass, and what unblocks it + (server API deployed, feature implemented, real credential/policy). +- **Confirmation that all temp changes were reverted** and the tree is clean. From ceccec30bb034f6ec2f060e392d48ba359960d47 Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:25:27 -0700 Subject: [PATCH 07/29] e2e skill: fingerprint enroll/status + prompt-user fallback deviceui.ps1 gains finger-status (best-effort enrollment detection, incl. the modern dumpsys JSON prints/count shape) and finger-enroll (emulator: set a screen-lock PIN, launch the enroll flow, drive it with repeated emu finger touches; if it can't confirm, print clear manual steps and exit 5 to prompt the user). finger (touch) now detects emulator vs real device: it simulates a touch on emulators and, on a real device where emu finger can't help, prompts the user to press the physical sensor. Verified on an emulator: status detection returns no->(after) and the enroll path degrades to the user prompt when the wizard can't be fully driven. Docs updated in ui-interaction.md and the SKILL.md script table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 2 +- .../references/ui-interaction.md | 6 +- .../scripts/deviceui.ps1 | 105 +++++++++++++++++- 3 files changed, 104 insertions(+), 9 deletions(-) diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index 9d130c7e..062efc07 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -36,7 +36,7 @@ All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` | `emulator.ps1` | Device pool: emulators **and** real devices | `ensure`, `status`, `list`, `pool`, `list-images`, `create`, `start` | | `devicelease.ps1` | Lease a device so concurrent tests don't collide | `acquire`, `heartbeat`, `release`, `list`, `reap` | | `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | -| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `screenshot`, `current-app` | +| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `finger-status`, `finger-enroll`, `screenshot`, `current-app` | | `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | ## Execution model — run inside a sub-agent, and supervise it diff --git a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md index b4186276..2312018c 100644 --- a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md +++ b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md @@ -41,7 +41,7 @@ pick among duplicates). When text is empty (icon-only buttons), use `tap-desc`. | **Consent / permissions requested** | `tap-text "Accept"` (safe for test apps/accounts). | | **"Stay signed in?" / KMSI** | `tap-text "Yes"` (or `No` if the scenario needs a fresh session). | | **"Approve sign in request" (push MFA)** | Usually a **blocker** — see below, unless a TOTP/seed is available. | -| **Fingerprint / biometric prompt** | On emulator, `deviceui.ps1 finger -Text 1` simulates an enrolled fingerprint. Enroll first if needed (Settings → Security), or fall back to PIN. | +| **Fingerprint / biometric prompt** | On an **emulator**, enroll once if needed (`deviceui.ps1 finger-enroll`), then `deviceui.ps1 finger -Text 1` simulates a touch. On a **real device** `emu finger` can't help — the user must have a fingerprint enrolled and press the sensor (the script will prompt). Check state with `deviceui.ps1 finger-status`. Fall back to PIN if biometrics can't be satisfied. | | **Device PIN/pattern (keyguard)** | Set a known PIN via adb during setup, then enter it; or `emulator.ps1` dismisses a no-secure keyguard automatically. | ## Inputs the AI handles automatically @@ -53,7 +53,9 @@ Do these without asking: - Selecting an account from a picker; choosing `Add account`. - Granting runtime permissions (`appcontrol.ps1 grant` or tapping the dialog). - Dismissing benign system dialogs/ANRs (`Wait`), closing bottom sheets (`key BACK`). -- **Simulating a fingerprint** on the emulator (`finger`). +- **Simulating a fingerprint** on the emulator (`finger` to touch; `finger-enroll` to enroll one first, + `finger-status` to check). On a real device, enrolling needs the physical sensor — the script will + prompt the user for that setup. - Entering a **TOTP** code when the authenticator seed/secret is known to the session (compute it). - Setting a device PIN during environment setup and re-entering it later. - Toggling in-app test switches (feature flags exposed in the test app UI). diff --git a/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 index e4e83a6a..ed268a1e 100644 --- a/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 +++ b/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 @@ -7,26 +7,30 @@ .PARAMETER Command dump | find-text | tap-text | tap-desc | input-text | wait-text | screenshot | - key | finger | current-app | tap-xy + key | finger | finger-status | finger-enroll | current-app | tap-xy .EXAMPLE ./deviceui.ps1 wait-text -Text "Sign in" -TimeoutSec 30 ./deviceui.ps1 tap-text -Text "Sign in" ./deviceui.ps1 input-text -Text "user@contoso.com" ./deviceui.ps1 key -Text ENTER - ./deviceui.ps1 finger -Text 1 # simulate enrolled fingerprint id 1 + ./deviceui.ps1 finger-status # is a fingerprint enrolled? + ./deviceui.ps1 finger-enroll # enroll one on an emulator (or prompt if a real device) + ./deviceui.ps1 finger -Text 1 # simulate a touch of enrolled fingerprint id 1 ./deviceui.ps1 screenshot -Out C:\runs\step1.png ./deviceui.ps1 current-app # resolved/focused package + activity .NOTES Text matching is case-insensitive substring by default. Use -Exact for equality. Always re-`dump` after an action; the UI tree changes between steps. + Fingerprint: `emu finger` only works on EMULATORS. On a real device a fingerprint must be enrolled + by the user against the physical sensor — `finger-enroll` will print the steps and ask. #> [CmdletBinding()] param( [Parameter(Position = 0)] [ValidateSet('dump', 'find-text', 'tap-text', 'tap-desc', 'input-text', 'wait-text', - 'screenshot', 'key', 'finger', 'current-app', 'tap-xy')] + 'screenshot', 'key', 'finger', 'finger-status', 'finger-enroll', 'current-app', 'tap-xy')] [string]$Command = 'dump', [string]$Serial, @@ -36,6 +40,7 @@ param( [int]$Index = 0, [switch]$Exact, [int]$TimeoutSec = 20, + [string]$Pin = '1234', [string]$Out ) @@ -114,6 +119,40 @@ function Encode-Input { return $s } +function Test-IsEmulator { + # An explicit emulator-XXXX serial is an emulator; otherwise probe the running device. + if ($Serial) { return ($Serial -match '^emulator-') } + $qemu = (Adb shell getprop ro.kernel.qemu 2>$null | Out-String).Trim() + if ($qemu -eq '1') { return $true } + $chars = (Adb shell getprop ro.build.characteristics 2>$null | Out-String).Trim() + return ($chars -match 'emulator') +} + +function Get-FingerprintStatus { + # Best-effort: 'yes' | 'no' | 'unknown'. dumpsys layout varies by API level, so match a few shapes. + $dump = (Adb shell dumpsys fingerprint 2>$null | Out-String) + if (-not $dump.Trim()) { return 'unknown' } + # Modern JSON shape: {"prints":[{"id":,"count":,...}]} — enrolled if any count >= 1. + if ($dump -match '"prints"\s*:\s*\[') { + $counts = [regex]::Matches($dump, '"count"\s*:\s*(\d+)') | ForEach-Object { [int]$_.Groups[1].Value } + if (@($counts | Where-Object { $_ -ge 1 }).Count -gt 0) { return 'yes' } + if ($counts.Count -gt 0) { return 'no' } + } + # Common markers of an enrolled template across older versions. + if ($dump -match 'Fingerprint\s*\(.*id=\d+' -or + $dump -match 'enrolledTemplates?=\s*\[?\s*[1-9]' -or + $dump -match 'mEnrolledFingerprints=\[[^\]]+\]' -or + $dump -match 'numEnrolled=\s*[1-9]' -or + $dump -match 'templates?:\s*[1-9]') { + return 'yes' + } + # Explicit "none enrolled" shapes. + if ($dump -match 'enrolledTemplates?=\s*\[\s*\]' -or $dump -match 'numEnrolled=\s*0' -or $dump -match 'mEnrolledFingerprints=\[\]') { + return 'no' + } + return 'unknown' +} + switch ($Command) { 'dump' { Get-Nodes (Get-UiXml) | Where-Object { $_.Text -or $_.Desc } | @@ -176,9 +215,63 @@ switch ($Command) { } 'finger' { $id = if ($Text) { $Text } else { '1' } - $ser = if ($Serial) { $Serial } else { '' } - if ($ser) { & $adb -s $ser emu finger touch $id | Out-Null } else { & $adb -e emu finger touch $id | Out-Null } - Write-Host "Simulated fingerprint id=$id" + if (Test-IsEmulator) { + if ($Serial) { & $adb -s $Serial emu finger touch $id | Out-Null } else { & $adb -e emu finger touch $id | Out-Null } + Write-Host "Simulated fingerprint touch id=$id on emulator." + } + else { + Write-Host "PROMPT-USER: This is a real device — `emu finger` cannot simulate a touch." + Write-Host "Please press the enrolled finger on the device's sensor now, then continue." + exit 5 + } + } + 'finger-status' { + $status = Get-FingerprintStatus + Write-Host "Fingerprint enrolled: $status" + if ($status -eq 'yes') { exit 0 } elseif ($status -eq 'no') { exit 2 } else { exit 3 } + } + 'finger-enroll' { + $id = if ($Text) { $Text } else { '1' } + if ((Get-FingerprintStatus) -eq 'yes') { Write-Host "A fingerprint is already enrolled; nothing to do."; exit 0 } + + if (-not (Test-IsEmulator)) { + Write-Host "PROMPT-USER: Real device — a fingerprint must be enrolled against the physical sensor." + Write-Host " 1) Settings > Security > Fingerprint (set a screen lock/PIN first if asked)" + Write-Host " 2) Add a fingerprint and follow the prompts on the sensor." + Write-Host " 3) Re-run the test once enrolled." + exit 5 + } + + Write-Host "Enrolling a fingerprint on the emulator (id=$id)..." + # A screen lock is a prerequisite for fingerprint enrollment. + Adb shell locksettings set-pin $Pin 2>$null | Out-Null + # Open the enrollment flow (action exists API 28+); fall back to Security settings. + Adb shell am start -a android.settings.FINGERPRINT_ENROLL 2>$null | Out-Null + Start-Sleep -Seconds 2 + # If it asks to confirm the existing PIN, enter it. + try { Adb shell input text $Pin 2>$null | Out-Null; Adb shell input keyevent 66 2>$null | Out-Null } catch { } + Start-Sleep -Seconds 1 + # The enroll wizard advances on each sensor touch; emulate several with brief pauses, + # tapping Next/Done/Confirm/OK when they appear. + for ($i = 0; $i -lt 8; $i++) { + if ($Serial) { & $adb -s $Serial emu finger touch $id | Out-Null } else { & $adb -e emu finger touch $id | Out-Null } + Start-Sleep -Milliseconds 700 + foreach ($label in @('Next', 'Done', 'Confirm', 'OK', 'Got it')) { + $m = Find-ByField 'Text' $label + if ($m) { $t = @($m)[0]; if ($t.Cx) { Adb shell input tap $t.Cx $t.Cy | Out-Null }; break } + } + } + Start-Sleep -Seconds 1 + $status = Get-FingerprintStatus + if ($status -eq 'yes') { + Write-Host "Fingerprint enrolled (id=$id). Use `finger -Text $id` to simulate a touch." + exit 0 + } + Write-Host "PROMPT-USER: Could not confirm automatic fingerprint enrollment on this emulator." + Write-Host "Please enroll one manually, then re-run:" + Write-Host " Settings > Security > Fingerprint > Add fingerprint; when it asks for a touch, run:" + Write-Host " ./deviceui.ps1 finger -Text $id -Serial $Serial (repeat until it completes)" + exit 5 } 'current-app' { $win = (Adb shell dumpsys window 2>$null | Out-String) From 9d03613a6dc432223bb11ff732f7bea7ba8307fb Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:26:56 -0700 Subject: [PATCH 08/29] e2e skill: fix TOC anchor for renamed requirements heading Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/app-and-module-map.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md b/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md index c240939a..dcc2805a 100644 --- a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md +++ b/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md @@ -7,7 +7,7 @@ Table of contents: - [Two E2E execution modes](#two-e2e-execution-modes) - [Broker pairing rules](#broker-pairing-rules) - [Test credentials (Lab API) and when they block](#test-credentials-lab-api-and-when-they-block) -- [Emulator requirements per feature](#emulator-requirements-per-feature) +- [Emulator / real-device requirements per feature](#emulator--real-device-requirements-per-feature) The repo root (`android-complete`) aggregates sub-repos cloned via `git droidSetup`: `msal/`, `common/`, `broker/`, `adal/`, and optionally `authenticator/`. Gradle modules are From 169e5913536b80211932dcf3eaa1e09d5c4e6edb Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:50:04 -0700 Subject: [PATCH 09/29] e2e skill: keep only generic lessons, drop test-case specifics The lessons captured in fc7b556 mixed generic patterns with details specific to one feature under test (MAM broker-install-resume). Per review, the skill should stay generic; feature-specific steps belong in a test case, not the skill. Removed: the sink-wait/TTL 'parked request / resume path' guidance and the Company-Portal/MAM framing in the Play Store and AADSTS530021 sections (generalized to 'installing an app mid-flow' and 'an unapproved app config blocks the flow'). Also generalized a devicelease example (-Feature mam-resume -> signin-e2e). Kept the genuinely generic fixes: publish-to-mavenLocal, NDK/CMake, AADSTS50011 signing, flights-manager-not-initialized, WebView field focus/verify, BACK-vs-ESCAPE, special-char passwords, Play Store Install-button targeting, and the broker telemetry keys. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/troubleshooting.md | 6 +++--- .../references/ui-interaction.md | 13 ++++++------- .../scripts/devicelease.ps1 | 2 +- 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md index 4050cf84..c855d5f7 100644 --- a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md +++ b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md @@ -142,9 +142,9 @@ build signed with the wrong keystore is rejected even though the package name is Access policy blocked the app **before** the flow you're trying to test could run. This happens when the `clientId` / app configuration you signed in with is **not on the tenant's approved-client-app list**. -- **Why it bites E2E:** you may be testing a MAM / CA-driven flow, but if the test app's configured - client id isn't CA-approved, eSTS returns `530021` and you never reach the step under test (e.g. the - install-broker prompt). +- **Why it bites E2E:** if the test app's configured client id isn't CA-approved, eSTS returns `530021` + and blocks the flow *before* the step you're trying to test — so it looks like the feature failed when + the app config is the real cause. - **Fix:** run with an **approved** app configuration. Many test apps let you pick the app config / client id at runtime (a spinner or a build flag). A well-known first-party client id with an OOB redirect (e.g. an Office client id) is CA-approved and reaches CA-gated flows, whereas a custom test diff --git a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md index 2312018c..d9d36b5e 100644 --- a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md +++ b/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md @@ -107,24 +107,23 @@ a few ways — these fixes came from real runs: - **Type passwords with special characters literally.** `~ ! # $ & * ( )` are shell metacharacters. `deviceui.ps1 input-text` escapes them, but if you call adb directly, single-quote the whole string **device-side**: `adb shell "input text 'P@ss~!word'"`. Never echo the password into the transcript. -- **Some CP/broker WebViews reject programmatic submit entirely.** With `FLAG_SECURE`, typing may work +- **Some broker WebViews reject programmatic submit entirely.** With `FLAG_SECURE`, typing may work but tapping **Sign in** / pressing Enter doesn't register on an emulator. If a secure broker screen won't submit, note it as a **harness limitation** (not a defect) and, if the segment under test is already proven, stop there — see [mocking-flights-and-segments.md](mocking-flights-and-segments.md). -## Play Store installs (referrer / broker-install flows) +## Installing an app from the Play Store mid-flow -Installing an app from the Play Store during a flow (e.g. installing Company Portal for a MAM/CA flow): +Some flows install another app from the Play Store partway through (e.g. a broker or a dependency): - **Target the real Install button, not the rating chip.** `tap-text "Install"` can hit the "Everyone" content-rating label or an unrelated element. Read the button's bounds from `dump` and `tap-xy` its center, and dismiss the interstitial **"Got it"** age-rating dialog first if it appears. - **Installs are slow and interstitials vary.** Poll for completion by package presence (`appcontrol.ps1 is-installed -Package `) in a loop rather than assuming a fixed delay. -- **Mind time-boxed waits.** If the feature parks a request with a TTL (e.g. a sink-wait that expires - after N minutes) and a slow emulator install blows past it, the original request may lapse. That's a - harness-timing artifact, not a defect: re-trigger the flow with the app already installed to exercise - the same resume path, and note the timing in the report. +- **A slow install can exceed a feature's time-boxed step.** If the flow has a bounded wait and a slow + emulator install overruns it, that's a harness-timing artifact, not a defect — note the timing and, if + the scenario allows, re-run the step with the app already installed. ## Keyboard & navigation gotchas diff --git a/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 index 7489eb9d..1f2c25c4 100644 --- a/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 +++ b/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 @@ -26,7 +26,7 @@ .EXAMPLE # Get a leased, booted device for this agent (same reqs you'd pass emulator.ps1 ensure): - ./devicelease.ps1 acquire -Owner $AgentId -Feature mam-resume -RequireGoogleApis -ApiLevel 30 -Wait + ./devicelease.ps1 acquire -Owner $AgentId -Feature signin-e2e -RequireGoogleApis -ApiLevel 30 -Wait # -> prints SERIAL=; use it for the whole run. ./devicelease.ps1 heartbeat -Owner $AgentId -Serial emulator-5554 # keep the lease fresh (each phase) From f45167bcf3d14c5bf5f59a3c4805365f78115a90 Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Sun, 19 Jul 2026 22:52:09 -0700 Subject: [PATCH 10/29] e2e skill: source test-case-specific steps from ADO or a known file Establishes where feature-specific steps come from so the skill stays generic. Phase 1 now lists an ADO Test Case work item and a known test-steps file as first-class sources, with a concrete fetch (az boards work-item show ... --fields Microsoft.VSTS.TCM.Steps) + parse (each = action + expected result -> success criterion), and notes the test-planner -> ADO -> this-skill loop. Adds an explicit principle: anything specific to one feature (app config/client id, account, broker, exact tap sequence, expected markers) belongs in the test case, never hardcoded in the skill or its scripts; read it at run time. Frontmatter description updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 33 ++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index 062efc07..377277d9 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -1,6 +1,6 @@ --- name: android-emulator-e2e-tester -description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator or a connected real device. Builds a device pool from emulators and any adb-connected hardware, finds or creates a suitable AVD (or reuses a running emulator / real device), leases the device so concurrent tests don't collide, builds/installs the right test app, drives the UI, and verifies via logcat. Delegates the actual run to a sub-agent (same model as the parent) and waits for its verdict. Use when asked to 'test this feature on the emulator', 'run the E2E test', 'verify the feature end to end', 'does this work on a device', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from the session's design spec, implementation diff, or user-provided test steps. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint), mocks unavailable dependencies and sets feature flags via temporary code changes when needed, and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail and drives a fix-and-retest loop until it passes." +description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator or a connected real device. Builds a device pool from emulators and any adb-connected hardware, finds or creates a suitable AVD (or reuses a running emulator / real device), leases the device so concurrent tests don't collide, builds/installs the right test app, drives the UI, and verifies via logcat. Delegates the actual run to a sub-agent (same model as the parent) and waits for its verdict. Use when asked to 'test this feature on the emulator', 'run the E2E test', 'verify the feature end to end', 'does this work on a device', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from user-provided steps, an ADO Test Case work item, a known test-steps file, the session's design spec, or the implementation diff — feature-specific steps live in the test case, not the skill. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint), mocks unavailable dependencies and sets feature flags via temporary code changes when needed, and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail and drives a fix-and-retest loop until it passes." --- # Android Emulator E2E Tester @@ -79,9 +79,12 @@ Execute in order. Announce a one-line status at each phase. Synthesize a concrete E2E scenario from the session context, in priority order: 1. **User-provided test steps** in the session (explicit steps or a scenario the user described). -2. **A test plan** (from the `test-planner` skill or a linked ADO test case). -3. **The design spec** (`design-docs/`) — its acceptance criteria and flows. -4. **The implementation diff** — `git -C diff` across changed repos to see what actually +2. **An ADO Test Case work item** — the primary home for feature-specific steps. Fetch and parse it (see + "Sourcing test-case-specific steps" below); the `test-planner` skill can author/export these. +3. **A known test-steps file** the session points to (an exported plan, a markdown checklist, or a path + the user names). +4. **The design spec** (`design-docs/`) — its acceptance criteria and flows. +5. **The implementation diff** — `git -C diff` across changed repos to see what actually changed and which app/module it affects. Produce: the **feature summary**, the **target app/module** (see @@ -92,6 +95,28 @@ with an explicit, observable **success criterion** (e.g. "AcquireTokenSilent ret **Ask the user if** the scenario is ambiguous, multiple flows could be meant, or you cannot tell what "working" looks like. Do not guess a scenario when the intent is unclear. +**Keep specifics out of the skill.** This skill is generic. Anything specific to one feature — which app +configuration / client id to select, which lab account, which broker to pair, the exact tap sequence, the +expected success markers — belongs in the **test case**, not here. Read those specifics from the sources +above at run time; never hardcode a particular feature's steps into the skill or its scripts. + +#### Sourcing test-case-specific steps + +- **From ADO** (preferred for anything repeatable): a **Test Case** work item stores its steps in the + `Microsoft.VSTS.TCM.Steps` field (HTML/XML — each `` has an action and an expected result). Fetch it: + ```powershell + az boards work-item show --id --org ` + --fields "System.Title,Microsoft.VSTS.TCM.Steps" --output json + ``` + Parse the steps into an ordered action/expected list, drive them in Phase 4, and use each step's + **expected result** as its success criterion. A linked **Shared Steps** work item is fetched the same + way. (The `test-planner` skill authors and pushes these test cases, so the loop is: + test-planner → ADO → this skill.) +- **From a known file**: if the session provides a test-steps file (an exported plan, a markdown + checklist, or a path the user names), read the steps from there and treat them like ADO steps. +- If neither exists and the scenario isn't otherwise clear, **ask the user** rather than inventing + feature-specific steps. + ### Phase 2 — Provision the device (emulator or real device) The **device pool** is every usable target: emulators **and** any real device connected over adb. From 8b60ed331c6efcd13690730153417803ea1b09f2 Mon Sep 17 00:00:00 2001 From: Zhipan Wang Date: Mon, 20 Jul 2026 12:30:14 -0700 Subject: [PATCH 11/29] e2e skill: emulator performance + Android Studio integration Addresses two real pain points from a Cloud PC run: (1) very slow emulator, (2) emulator not visible in Android Studio's Running Devices. emulator.ps1 now profiles the host and picks fast, correct flags: auto -gpu (host GPU when a real GPU exists, else SwiftShader software), auto -cores/-memory sized from the host (e.g. 6/6144 on a big box vs a stock AVD's 4/1536), with -Gpu/-Cores/-Memory overrides. It detects GPU-less VM/RDP/Cloud-PC hosts and, since a software-rendered emulator is painfully slow there, auto-prefers a connected physical device in ensure (override -NoPhysical) and prints a clear perf note. resolve-sdk now reports the host GPU/perf profile. New reference emulator-performance.md explains why the emulator is slow (no GPU on Cloud PC/VM/RDP -> SwiftShader), the physical-device fast path, emulator tuning, slow Play Store/NAT downloads, and Android Studio integration: AVDs already appear in Device Manager (shared ~/.android/avd); to see one in Running Devices, start it from Studio and let the skill reuse the running emulator. Wired into SKILL.md Phase 2 + troubleshooting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../android-emulator-e2e-tester/SKILL.md | 9 ++ .../references/emulator-performance.md | 116 ++++++++++++++++++ .../references/troubleshooting.md | 2 + .../scripts/emulator.ps1 | 96 ++++++++++++++- 4 files changed, 221 insertions(+), 2 deletions(-) create mode 100644 .github/skills/android-emulator-e2e-tester/references/emulator-performance.md diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-emulator-e2e-tester/SKILL.md index 377277d9..b77c1b52 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-emulator-e2e-tester/SKILL.md @@ -141,6 +141,14 @@ may run concurrently, lease the device first** (see Phase 2a) so two runs don't fails, see [references/troubleshooting.md](references/troubleshooting.md) (missing images, hypervisor, boot hang). +> **Speed check (do this if the run feels slow).** Run `./scripts/emulator.ps1 resolve-sdk` — it prints the +> host GPU/perf profile. On a **Cloud PC / VM / RDP host there is no GPU**, so the emulator uses slow +> **software rendering** (SwiftShader) and its NAT makes Play Store downloads crawl — nothing makes an +> emulator fast there. **Prefer a connected physical device** (the skill auto-prefers one on a GPU-less +> host; force with `-PreferPhysical`). AVDs the skill creates already appear in **Android Studio's Device +> Manager** (shared `~/.android/avd`); to see one in Studio's **Running Devices**, start it from Studio and +> let the skill reuse it. Details: [references/emulator-performance.md](references/emulator-performance.md). + ### Phase 2a — Lease the device (avoid collisions with other tests) Multiple E2E tests can run at once (different agents/sessions). To stop two runs from driving the same @@ -276,6 +284,7 @@ Load these as needed (don't preload all): | File | Read it when | |---|---| | [references/app-and-module-map.md](references/app-and-module-map.md) | Choosing/deploying the test app, package discovery, broker pairing, credentials, emulator requirements | +| [references/emulator-performance.md](references/emulator-performance.md) | The run is slow, you're on a Cloud PC/VM/RDP (software rendering), or you want the emulator to show in Android Studio's Device Manager / Running Devices | | [references/log-signals.md](references/log-signals.md) | Interpreting logcat, success/failure patterns, AADSTS codes, per-flow pass criteria, eSTS correlation | | [references/ui-interaction.md](references/ui-interaction.md) | Driving auth screens, AI-vs-human inputs, FLAG_SECURE gotcha, selector strategy | | [references/mocking-flights-and-segments.md](references/mocking-flights-and-segments.md) | A flag must be set, a dependency/server data is unavailable, or the flow can't run fully E2E (mock it or test in segments) | diff --git a/.github/skills/android-emulator-e2e-tester/references/emulator-performance.md b/.github/skills/android-emulator-e2e-tester/references/emulator-performance.md new file mode 100644 index 00000000..3602f80a --- /dev/null +++ b/.github/skills/android-emulator-e2e-tester/references/emulator-performance.md @@ -0,0 +1,116 @@ +# Emulator Performance & Android Studio Integration + +Table of contents: +- [Why the emulator is slow (and the #1 fix)](#why-the-emulator-is-slow-and-the-1-fix) +- [Diagnose your host in one command](#diagnose-your-host-in-one-command) +- [Fast-path: use a physical device](#fast-path-use-a-physical-device) +- [Making the emulator as fast as it can be](#making-the-emulator-as-fast-as-it-can-be) +- [Slow Play Store / network downloads](#slow-play-store--network-downloads) +- [Android Studio: Device Manager & Running Devices](#android-studio-device-manager--running-devices) + +## Why the emulator is slow (and the #1 fix) + +The dominant factor is **GPU rendering mode**: + +- **With a real GPU** the emulator uses **host (hardware) rendering** — UI, `screencap`, and `uiautomator dump` are fast. +- **Without a usable GPU** — typically a **Cloud PC / VM / RDP session** (e.g. a `CPC-*` Windows 365 box, which reports only a "Microsoft Hyper-V Video" / "Remote Display" adapter) — the emulator falls back to **SwiftShader software rendering** (the CPU draws every frame). Everything graphical is slow, and there is **no flag that makes a GPU appear**; `-gpu host` can't help when the host has no GPU. + +CPU virtualization (WHPX/HAXM/AEHD) is separate and usually still works on a VM (nested virtualization), so the CPU isn't the bottleneck — the **GPU is**. + +**#1 fix: run the scenario on a physical device instead of the emulator.** The skill's device pool includes +connected real devices, and on a GPU-less host `emulator.ps1 ensure` / `devicelease.ps1 acquire` will +**auto-prefer a connected physical device** for you. A phone renders on its own GPU and downloads over real +Wi-Fi, so flows that crawl on a software-rendered emulator (installing Company Portal, driving WebViews) +run in seconds. + +## Diagnose your host in one command + +```powershell +./scripts/emulator.ps1 resolve-sdk +``` +It prints the host GPU/perf profile, e.g. on a Cloud PC: +``` +host GPU: NONE -> emulator uses SLOW software (SwiftShader) rendering +host type: VM (Virtual Machine), RDP session, Cloud PC +emu defaults: -gpu swiftshader_indirect -cores 6 -memory 6144 +TIP: use a physical device for a fast run (emulator.ps1 ensure -PreferPhysical). +``` +If `host GPU` says NONE, expect a slow emulator and prefer a real device. + +## Fast-path: use a physical device + +1. Connect the device over USB (enable **USB debugging**) or Wi-Fi (`adb connect :5555`), and accept the + RSA prompt. Confirm it's in the pool: `./scripts/emulator.ps1 pool`. +2. Acquire/target it — on a GPU-less host it's automatic, or force it explicitly: + ```powershell + ./scripts/devicelease.ps1 acquire -Owner $AgentId -PreferPhysical -Wait # lease a real device + # or, without leasing: + ./scripts/emulator.ps1 ensure -PreferPhysical -Wait + ``` +3. Drive everything against that `$serial` exactly as with an emulator. + +**Caveats for real devices:** brokered flows with a **production** broker (Company Portal / Authenticator) +won't release tokens to a **debug-signed** calling app (caller-trust) — that's a signing/environment limit, +not a defect (see [troubleshooting.md](troubleshooting.md#signing--redirect-uri-mismatch-aadsts50011)). +Biometric prompts need a **real enrolled fingerprint** (the emulator-only `emu finger` simulation doesn't +apply) — see [ui-interaction.md](ui-interaction.md). Never print credentials; type them on the device. + +## Making the emulator as fast as it can be + +When you must use the emulator (no device available), these help — but won't beat a GPU-less ceiling: + +- **More cores/RAM.** `emulator.ps1` now auto-sizes `-cores`/`-memory` from the host (e.g. 6 cores / 6144 MB + on a big Cloud PC, up from a stock AVD's 4/1536). Override with `-Cores` / `-Memory`. +- **Keep the warm snapshot.** Don't pass `-ColdBoot` unless a snapshot is corrupt — a warm boot from + snapshot is much faster than a cold boot. +- **Explicit GPU mode.** The script auto-picks `host` when a GPU exists, else `swiftshader_indirect`. + Force with `-Gpu host` on a machine that has a GPU but defaulted to software. +- **Headless where possible.** `-NoWindow` skips the Qt UI (uiautomator/screencap still work); saves some + overhead, especially over RDP. +- **Use x86_64 images on x86 hosts.** ARM images on an x86 host run under slow translation. (On a physical + ARM phone, native arm64 is fine — build the matching ABI.) +- **Raise timeouts, don't fight slowness.** On a slow host, prefer longer `wait-text` / boot / install + timeouts and poll for completion (e.g. `is-installed` in a loop) instead of fixed sleeps, so a slow-but- + correct step isn't misread as a failure. + +## Slow Play Store / network downloads + +Large downloads (e.g. Company Portal ~30 MB) can take **minutes** on an emulator that finishes in **seconds** +on a phone. Causes and mitigations: + +- The emulator routes through a **user-mode NAT** with its own DNS; throughput is limited and unrelated to + your real link speed. `-netspeed full -netdelay none` (already set) removes artificial throttling but can't + make the NAT fast. +- Software rendering also slows the Play Store UI itself. +- **Mitigation:** poll for install completion by package presence rather than a fixed wait; mind any + **time-boxed** feature step (a slow download can outrun a TTL — re-trigger with the app already installed). + For download-heavy flows, **use a physical device** — this is the clearest win. + +## Android Studio: Device Manager & Running Devices + +Two different things, and the skill already lines up with the first: + +- **Device Manager (the AVD list).** The skill creates AVDs in the **default AVD home** (`~/.android/avd`, + unless you set `ANDROID_AVD_HOME`) — the **same** place Android Studio reads. So **AVDs the skill creates + already appear in Android Studio's Device Manager**, and you can start/stop/wipe/delete them there. (Verify + with `./scripts/emulator.ps1 list` vs. Device Manager — same names.) + +- **Running Devices (the embedded mirror window).** This tool window mirrors an emulator that Android Studio + **launched into it**. The skill starts the emulator as a **standalone process** (its own window / gRPC + endpoint), so it shows up in `adb devices` and Device Manager but **not automatically inside the Running + Devices window**. Two ways to get parity: + + 1. **Start from Studio, let the skill reuse it (recommended).** In Studio: *Settings → Tools → Emulator → + "Launch in the Running Devices tool window"* (enable), then start the AVD from **Device Manager** — it + opens inside **Running Devices**. Now run the skill **without** `-Avd`/`-Serial`; `ensure` **prefers an + already-running emulator**, so it drives the exact device you see mirrored in Studio. (Confirm the skill + picked it: its `SERIAL=` matches `adb devices`.) + 2. **Mirror the skill's emulator.** With that same setting enabled, recent Android Studio can also mirror an + externally-started emulator in Running Devices once it detects it (it may prompt). If it doesn't appear, + use option 1 — it's the reliable path. + + Either way the skill and Studio share one emulator, one AVD list, and one adb — no duplication. + +> Net: if you're on a Cloud PC/VM (no GPU), the fastest, least-friction setup is a **connected physical +> device** (the skill auto-prefers it). If you want the emulator visible in Studio's **Running Devices**, +> start it from Studio and let the skill reuse it. diff --git a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md index c855d5f7..daf09bf0 100644 --- a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md +++ b/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md @@ -49,6 +49,8 @@ Table of contents: | Stuck snapshot / corrupted state | `-ColdBoot` (`-no-snapshot-load`) to skip the stale snapshot. | | Transient "System UI isn't responding" ANR after boot | `deviceui.ps1 tap-text -Text "Wait"`; not a test failure. | | Need it faster / CI | `-NoWindow` (headless) — uiautomator still works. | +| **Everything is slow** (UI, screencap, downloads) | Likely **no host GPU** (Cloud PC / VM / RDP) → software rendering. Run `emulator.ps1 resolve-sdk` to confirm; prefer a **physical device** (`ensure -PreferPhysical`). See [emulator-performance.md](emulator-performance.md). | +| Emulator not in Android Studio **Running Devices** | The skill starts it standalone; it's still in **Device Manager** (shared AVD home). Start the AVD from Studio and let the skill reuse it. See [emulator-performance.md](emulator-performance.md#android-studio-device-manager--running-devices). | ## adb device issues diff --git a/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 b/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 index e436867f..43189d61 100644 --- a/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 +++ b/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 @@ -16,6 +16,12 @@ favor one, or `-NoPhysical` to stick to emulators). Real devices are never created/booted — they're only used if already connected and booted. + PERFORMANCE: emulator launches auto-pick the fastest usable `-gpu` mode (host GPU when present, + else SwiftShader software) and generous `-cores`/`-memory` from the host. On a GPU-less host + (Cloud PC / VM / RDP) the emulator can only do slow software rendering, so `ensure` will + automatically prefer a connected physical device when one is available (override with + `-NoPhysical`). Run `resolve-sdk` to see the host GPU/perf profile. + .PARAMETER Command resolve-sdk | list | list-images | status | pool | create | start | ensure @@ -24,8 +30,10 @@ ./emulator.ps1 ensure -ApiLevel 34 -RequireGoogleApis -Wait .EXAMPLE + ./emulator.ps1 resolve-sdk # SDK paths + host GPU/perf profile ./emulator.ps1 ensure -Avd Pixel_7 -Wait # use/create+start a named AVD ./emulator.ps1 ensure -PreferPhysical -Wait # prefer a connected real device if one fits + ./emulator.ps1 ensure -Cores 6 -Memory 6144 -Gpu host -Wait # tune emulator resources/GPU ./emulator.ps1 pool # the whole device pool (emulators + real devices) ./emulator.ps1 status # what is running right now ./emulator.ps1 list-images # installed system images (for create) @@ -41,6 +49,9 @@ param( [int]$ApiLevel = 0, # minimum API level requirement (0 = any) [string]$Image, # explicit system-image package id for create [string]$Device = 'pixel_7', # device profile for create + [int]$Cores = 0, # vCPU cores for the emulator (0 = auto: min(6, host/2)) + [int]$Memory = 0, # RAM MB for the emulator (0 = auto: 4096, or 6144 when host RAM allows) + [string]$Gpu, # emulator -gpu mode (host|swiftshader_indirect|auto). Empty = auto-detect. [switch]$RequireGoogleApis, # require google_apis or google_apis_playstore tag (GMS/push) [switch]$RequirePlayStore, # require google_apis_playstore tag [switch]$PreferPhysical, # prefer a connected real device over booting an emulator @@ -228,6 +239,56 @@ function Get-InstalledImages { return $images } +# --------------------------------------------------------------------------- +# Host performance profiling (GPU / virtualization) — drives fast, correct emulator flags +# --------------------------------------------------------------------------- +# Returns $true if the host has a real, emulator-usable GPU. On a Cloud PC / VM / RDP session there is +# typically only a "Microsoft Basic/Hyper-V/Remote Display" adapter with no dedicated memory, in which +# case the emulator can only do SLOW SwiftShader software rendering — and a physical device is far better. +function Test-HostGpuAvailable { + try { + $vc = Get-CimInstance Win32_VideoController -ErrorAction SilentlyContinue + foreach ($g in $vc) { + $n = "$($g.Name)" + if ($n -match 'Microsoft (Basic|Hyper-V|Remote)') { continue } # virtual/remote adapters + if ($n -match 'RDP|Citrix|VMware|VirtualBox|Parsec') { continue } # remoting adapters + if ($g.AdapterRAM -and $g.AdapterRAM -gt 0) { return $true } # a real GPU with VRAM + if ($n -match 'NVIDIA|AMD|Radeon|Intel|Arc|GeForce|Quadro') { return $true } + } + } catch { } + return $false +} + +function Test-VirtualOrRemoteHost { + $reasons = @() + try { + $cs = Get-CimInstance Win32_ComputerSystem -ErrorAction SilentlyContinue + if ("$($cs.Model)" -match 'Virtual Machine|VMware|VirtualBox' -or "$($cs.Manufacturer)" -match 'QEMU|Xen|innotek') { $reasons += "VM ($($cs.Model))" } + } catch { } + if ($env:SESSIONNAME -and $env:SESSIONNAME -match '^rdp') { $reasons += "RDP session" } + if ("$env:COMPUTERNAME" -match '^CPC-') { $reasons += "Cloud PC" } + return $reasons +} + +# Pick the fastest emulator -gpu mode the host can actually use. +function Get-BestGpuMode { + param([switch]$Headless) + if ($Gpu) { return $Gpu } # explicit override wins + if ($Headless) { return 'swiftshader_indirect' } # headless/CI: software is the safe choice + if (Test-HostGpuAvailable) { return 'host' } # real GPU: hardware acceleration + return 'swiftshader_indirect' # GPU-less VM/RDP: software (auto would pick this anyway) +} + +# Resolve auto cores/memory from the host (plenty of headroom on a 16-vCPU/64-GB Cloud PC). +function Resolve-EmuResources { + $cs = $null; try { $cs = Get-CimInstance Win32_ComputerSystem -ErrorAction SilentlyContinue } catch { } + $hostCores = if ($cs) { [int]$cs.NumberOfLogicalProcessors } else { 4 } + $hostGb = if ($cs) { [math]::Round($cs.TotalPhysicalMemory / 1GB, 0) } else { 8 } + $cores = if ($Cores -gt 0) { $Cores } else { [Math]::Max(4, [Math]::Min(6, [int]($hostCores / 2))) } + $mem = if ($Memory -gt 0) { $Memory } else { if ($hostGb -ge 32) { 6144 } elseif ($hostGb -ge 16) { 4096 } else { 3072 } } + return @{ Cores = $cores; Memory = $mem } +} + # --------------------------------------------------------------------------- # Actions # --------------------------------------------------------------------------- @@ -273,9 +334,17 @@ function New-Avd { function Start-Emu { param($Tools, [string]$Name, [switch]$Cold, [switch]$Headless, [int]$Timeout, [switch]$DoWait) $before = (Get-RunningEmulators $Tools).Serial - $emuArgs = @('-avd', $Name, '-no-boot-anim', '-netdelay', 'none', '-netspeed', 'full') + $res = Resolve-EmuResources + $gpuMode = Get-BestGpuMode -Headless:$Headless + $emuArgs = @('-avd', $Name, '-no-boot-anim', '-netdelay', 'none', '-netspeed', 'full', + '-gpu', $gpuMode, '-cores', "$($res.Cores)", '-memory', "$($res.Memory)") if ($Cold) { $emuArgs += '-no-snapshot-load' } - if ($Headless) { $emuArgs += '-no-window'; $emuArgs += '-gpu'; $emuArgs += 'swiftshader_indirect' } + if ($Headless) { $emuArgs += '-no-window' } + if ($gpuMode -eq 'swiftshader_indirect' -and -not $Headless) { + $why = (Test-VirtualOrRemoteHost) -join ', ' + Write-Host "PERF NOTE: no host GPU detected$(if ($why) { " ($why)" }); the emulator will use SLOW software rendering (SwiftShader)." -ForegroundColor Yellow + Write-Host " For a fast run, use a connected physical device: emulator.ps1 ensure -PreferPhysical (or devicelease.ps1 acquire -PreferPhysical)." -ForegroundColor Yellow + } Write-Host "Starting emulator: $($Tools.Emulator) $($emuArgs -join ' ')" Start-Process -FilePath $Tools.Emulator -ArgumentList $emuArgs -WindowStyle Minimized | Out-Null # Wait for a NEW emulator serial to appear that reports the target AVD name. @@ -307,6 +376,14 @@ switch ($Command) { Write-Host "emulator: $($tools.Emulator)" Write-Host "avdmanager: $($tools.AvdManager)" Write-Host "sdkmanager: $($tools.SdkManager)" + # Host perf profile — explains emulator speed and drives the fast defaults. + $gpuOk = Test-HostGpuAvailable + $virt = (Test-VirtualOrRemoteHost) -join ', ' + $res = Resolve-EmuResources + Write-Host "host GPU: $(if ($gpuOk) { 'available -> emulator uses HARDWARE (host) rendering' } else { 'NONE -> emulator uses SLOW software (SwiftShader) rendering' })" + if ($virt) { Write-Host "host type: $virt" } + Write-Host "emu defaults: -gpu $(Get-BestGpuMode) -cores $($res.Cores) -memory $($res.Memory)" + if (-not $gpuOk) { Write-Host "TIP: use a physical device for a fast run (emulator.ps1 ensure -PreferPhysical)." -ForegroundColor Yellow } } 'list' { Write-Host "== AVDs ==" @@ -386,6 +463,21 @@ switch ($Command) { Write-Host "No connected real device meets the requirements; falling back to an emulator." } + # 0c) On a GPU-less host (Cloud PC / VM / RDP) the emulator is painfully slow (software rendering). + # If a suitable real device is already connected, auto-prefer it — that's what the developer wants. + if (-not $PreferPhysical -and -not $NoPhysical -and -not (Test-HostGpuAvailable)) { + $physMatch = Get-PhysicalDevices $tools | + Where-Object { $_.Booted -and (Test-PhysicalMeetsReqs $_ -MinApi $ApiLevel -NeedGoogle:$RequireGoogleApis -NeedPlay:$RequirePlayStore) } | + Select-Object -First 1 + $why = (Test-VirtualOrRemoteHost) -join ', ' + if ($physMatch) { + Write-Host "No host GPU$(if ($why) { " ($why)" }) => emulator would be slow; using the connected real device instead: $($physMatch.Serial) (model=$($physMatch.Model)). Pass -NoPhysical to force the emulator." -ForegroundColor Yellow + Write-Host "SERIAL=$($physMatch.Serial)" + exit 0 + } + Write-Host "PERF NOTE: no host GPU$(if ($why) { " ($why)" }); the emulator will use SLOW software rendering. Connect a physical device for a fast run (it will be used automatically)." -ForegroundColor Yellow + } + # 1) Pick or create an AVD that satisfies the feature's requirements. $target = $null $avds = Get-Avds $tools From d6fef44deb27cf2c2aaf4cd25781c5dc9751195f Mon Sep 17 00:00:00 2001 From: wzhipan Date: Tue, 21 Jul 2026 22:36:50 -0700 Subject: [PATCH 12/29] e2e skill: rename to android-e2e-tester, mandatory ADO reports, LAB API + blocker refs Rework the E2E testing skill based on lessons from the AAD MFA sign-in run: - Rename skill android-emulator-e2e-tester -> android-e2e-tester (real device or emulator). - Make an HTML+Markdown test report MANDATORY for ADO test cases on every outcome; add scripts/report.ps1 and references/test-reporting.md. - Fix non-test-specific blockers: deviceui.ps1 input-text gains -Clear/-CharByChar/-PerCharDelayMs/-Secret to beat the Chrome autofill/passkey overlay; add scripts/labapi.ps1 (create-user/reset/enable-policy/disable-policy/delete-device) via the EasyAuth WAM-SSO workaround. - Add references/common-blockers.md (emulator-vs-device decision, fingerprint/App-Lock, number-match MFA, FLAG_SECURE, session timeouts) and references/lab-api.md (endpoints, usertypes/policies, auth, entitlements). - Add references/run-speed.md analyzing per-step latency (fixed sleeps, per-call process startup, redundant dumps/screenshots) with a faster wait-text/batch-per-screen recipe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../SKILL.md | 64 ++++++- .../references/app-and-module-map.md | 10 +- .../references/common-blockers.md | 158 ++++++++++++++++ .../references/emulator-performance.md | 0 .../android-e2e-tester/references/lab-api.md | 133 +++++++++++++ .../references/log-signals.md | 0 .../mocking-flights-and-segments.md | 0 .../references/run-speed.md | 116 ++++++++++++ .../references/test-reporting.md | 107 +++++++++++ .../references/troubleshooting.md | 5 + .../references/ui-interaction.md | 8 +- .../scripts/appcontrol.ps1 | 0 .../scripts/authlogs.ps1 | 0 .../scripts/devicelease.ps1 | 0 .../scripts/deviceui.ps1 | 37 +++- .../scripts/emulator.ps1 | 0 .../android-e2e-tester/scripts/labapi.ps1 | 174 +++++++++++++++++ .../android-e2e-tester/scripts/report.ps1 | 176 ++++++++++++++++++ 18 files changed, 970 insertions(+), 18 deletions(-) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/SKILL.md (77%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/references/app-and-module-map.md (93%) create mode 100644 .github/skills/android-e2e-tester/references/common-blockers.md rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/references/emulator-performance.md (100%) create mode 100644 .github/skills/android-e2e-tester/references/lab-api.md rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/references/log-signals.md (100%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/references/mocking-flights-and-segments.md (100%) create mode 100644 .github/skills/android-e2e-tester/references/run-speed.md create mode 100644 .github/skills/android-e2e-tester/references/test-reporting.md rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/references/troubleshooting.md (97%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/references/ui-interaction.md (92%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/scripts/appcontrol.ps1 (100%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/scripts/authlogs.ps1 (100%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/scripts/devicelease.ps1 (100%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/scripts/deviceui.ps1 (86%) rename .github/skills/{android-emulator-e2e-tester => android-e2e-tester}/scripts/emulator.ps1 (100%) create mode 100644 .github/skills/android-e2e-tester/scripts/labapi.ps1 create mode 100644 .github/skills/android-e2e-tester/scripts/report.ps1 diff --git a/.github/skills/android-emulator-e2e-tester/SKILL.md b/.github/skills/android-e2e-tester/SKILL.md similarity index 77% rename from .github/skills/android-emulator-e2e-tester/SKILL.md rename to .github/skills/android-e2e-tester/SKILL.md index b77c1b52..b4e91f27 100644 --- a/.github/skills/android-emulator-e2e-tester/SKILL.md +++ b/.github/skills/android-e2e-tester/SKILL.md @@ -1,13 +1,14 @@ --- -name: android-emulator-e2e-tester -description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator or a connected real device. Builds a device pool from emulators and any adb-connected hardware, finds or creates a suitable AVD (or reuses a running emulator / real device), leases the device so concurrent tests don't collide, builds/installs the right test app, drives the UI, and verifies via logcat. Delegates the actual run to a sub-agent (same model as the parent) and waits for its verdict. Use when asked to 'test this feature on the emulator', 'run the E2E test', 'verify the feature end to end', 'does this work on a device', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from user-provided steps, an ADO Test Case work item, a known test-steps file, the session's design spec, or the implementation diff — feature-specific steps live in the test case, not the skill. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint), mocks unavailable dependencies and sets feature flags via temporary code changes when needed, and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail and drives a fix-and-retest loop until it passes." +name: android-e2e-tester +description: "Execute and iterate end-to-end (E2E) tests for an in-development Android Auth feature (MSAL, Broker, Common, ADAL, Authenticator) on an Android emulator or a connected real device. Builds a device pool from emulators and any adb-connected hardware, finds or creates a suitable AVD (or reuses a running emulator / real device), leases the device so concurrent tests don't collide, builds/installs the right test app, drives the UI, and verifies via logcat. Delegates the actual run to a sub-agent (same model as the parent) and waits for its verdict. Use when asked to 'run the E2E test', 'test this feature end to end', 'verify the feature on a device or emulator', 'does this work on a device', 'run this ADO test case', or 'try the sign-in flow' — or automatically once a feature finishes development and is ready for E2E testing. Discovers what to test from user-provided steps, an ADO Test Case work item, a known test-steps file, the session's design spec, or the implementation diff — feature-specific steps live in the test case, not the skill. Auto-performs inputs the AI can handle (typing lab test credentials, tapping buttons, granting permissions, simulating a fingerprint, provisioning lab accounts via the LAB API), mocks unavailable dependencies and sets feature flags via temporary code changes when needed, and asks the user when the intent is unclear or a step is a real blocker (push MFA, hardware, missing credentials, implementation gaps). Checks logs to decide pass/fail, drives a fix-and-retest loop until it passes, and — for ADO test cases — always writes an HTML + Markdown test report at the end. Prefers an emulator when a step needs an injectable fingerprint/biometric (App Lock, number-match with biometric gate)." --- -# Android Emulator E2E Tester +# Android E2E Tester -Take an in-development Android Auth feature and prove it works on a real emulator: provision the -device, deploy the app, run the scenario (auto-handling every input the AI reasonably can), read the +Take an in-development Android Auth feature and prove it works on a real device or an emulator: provision +the device, deploy the app, run the scenario (auto-handling every input the AI reasonably can), read the logs to decide pass/fail, and loop fix-and-retest until it passes — or stop and ask when genuinely blocked. +For a test case driven from Azure DevOps, always finish by writing an HTML + Markdown test report. ## When this runs @@ -36,8 +37,10 @@ All under `scripts/` (PowerShell — the team's cross-platform shell). Run `-?` | `emulator.ps1` | Device pool: emulators **and** real devices | `ensure`, `status`, `list`, `pool`, `list-images`, `create`, `start` | | `devicelease.ps1` | Lease a device so concurrent tests don't collide | `acquire`, `heartbeat`, `release`, `list`, `reap` | | `appcontrol.ps1` | App build/install/state | `build`, `install`, `launch`, `clear`, `uninstall`, `is-installed`, `grant`, `list-apks` | -| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text`, `key`, `finger`, `finger-status`, `finger-enroll`, `screenshot`, `current-app` | +| `deviceui.ps1` | AI-driven UI I/O | `dump`, `wait-text`, `tap-text`, `tap-desc`, `input-text` (`-Clear`, `-CharByChar`), `key`, `finger`, `finger-status`, `finger-enroll`, `screenshot`, `current-app` | | `authlogs.ps1` | Log capture + verdict | `clear`, `scan`, `snapshot`, `grep`, `watch` | +| `labapi.ps1` | Provision/reset LAB test accounts (EasyAuth via WAM SSO) | `create-user`, `reset`, `enable-policy`, `disable-policy`, `delete-device`, `open` | +| `report.ps1` | Render the HTML + Markdown test report (mandatory for ADO test cases) | `render` (from a run JSON) | ## Execution model — run inside a sub-agent, and supervise it @@ -204,6 +207,26 @@ expected element → act (`tap-text` / `input-text` / `key` / `finger`) → re-` Allow/Yes, pick an account, grant permissions, simulate a fingerprint (`finger`), enter a TOTP if the seed is known, set/enter a device PIN. **Do not** print or commit credentials. +**Typing into eSTS/WebView credential fields (hard-won).** The email/password pages are a WebView and +Chrome's autofill/passkey overlay silently swallows a bulk `input text` (the value lands in the wrong +field or is dropped, producing "Enter a valid email"). Type reliably: +```powershell +./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial # username +./scripts/deviceui.ps1 input-text -Text $pw -Clear -CharByChar -Secret -Serial # password (never echoed) +``` +`-Clear` empties the field first, `-CharByChar` defeats the overlay, `-Secret` keeps the value out of the +transcript. Dismiss a passkey/"Save password" bottom sheet with `key ESCAPE` before typing if one appears. +See [references/common-blockers.md](references/common-blockers.md). + +**Provision / repair the lab account with the LAB API** when the scenario needs a fresh account or the +account state is stuck (e.g. MFA already registered from a previous run, a CA policy blocking the step): +```powershell +./scripts/labapi.ps1 create-user -UserType GlobalMFA # temp user, auto-deletes in 60 min +./scripts/labapi.ps1 reset -Upn $upn -Operation mfa # clear stale MFA registration +./scripts/labapi.ps1 disable-policy -Upn $upn -Policy GlobalMFA # unblock a CA-gated segment +``` +See [references/lab-api.md](references/lab-api.md) for all endpoints, usertypes, and the auth workaround. + **Set flags & mock what's missing (don't fake a pass).** If the scenario needs a feature flag on, or a step depends on data/a dependency you can't produce naturally (a server API not deployed yet, a collaborator app you can't drive), set the flag and mock the missing piece — including **temporary code @@ -255,6 +278,21 @@ Summarize: scenario tested, target app/emulator, final verdict, iterations taken (success signal + correlation_id, or the blocker), and links to the run-folder artifacts (logs, screenshots). If blocked, state exactly what you need from the user to proceed. +**For an ADO Test Case, a written test report is MANDATORY — always generate it, on every outcome +(PASS / FAIL / BLOCKED / PARTIAL), not only on success.** Render both an HTML and a Markdown report into +the run folder with `report.ps1`: +```powershell +# Build a run JSON (verdict, device/app/account metadata, per-step action/expected/result, evidence, +# blockers, artifact paths — UPN only, never a password), then render: +./scripts/report.ps1 render -In \run.json # writes TestReport.html + TestReport.md next to it +``` +The step list should mirror the ADO test case's steps, and each step's **result** should be judged +against that step's **expected result**. Include the ADO ids (`testCaseId`/`planId`/`suiteId`) and a link +so the report ties back to the test plan. See [references/test-reporting.md](references/test-reporting.md) +for the JSON schema and a worked example. (For an automation-test run, also attach the +`build/reports/androidTests/` output.) Present the verdict and the report paths to the user; optionally +publish the outcome back to the ADO Test Run. + **Release the device lease** so the next test can use it (do this even on failure/blocked): ```powershell ./scripts/devicelease.ps1 release -Owner $AgentId -Serial $serial @@ -284,6 +322,10 @@ Load these as needed (don't preload all): | File | Read it when | |---|---| | [references/app-and-module-map.md](references/app-and-module-map.md) | Choosing/deploying the test app, package discovery, broker pairing, credentials, emulator requirements | +| [references/lab-api.md](references/lab-api.md) | Provisioning/resetting a lab test account, LAB API endpoints/usertypes/policies, the EasyAuth auth workaround, `labapi.ps1` | +| [references/common-blockers.md](references/common-blockers.md) | Recurring hiccups & when to switch to an emulator (fingerprint/App-Lock, number-match MFA, session timeouts, FLAG_SECURE, autofill/passkey overlay, screenshot corruption) | +| [references/test-reporting.md](references/test-reporting.md) | Writing the mandatory ADO test report — run-JSON schema, `report.ps1`, worked example | +| [references/run-speed.md](references/run-speed.md) | The run feels slow; understanding per-step latency sources and how to shorten them | | [references/emulator-performance.md](references/emulator-performance.md) | The run is slow, you're on a Cloud PC/VM/RDP (software rendering), or you want the emulator to show in Android Studio's Device Manager / Running Devices | | [references/log-signals.md](references/log-signals.md) | Interpreting logcat, success/failure patterns, AADSTS codes, per-flow pass criteria, eSTS correlation | | [references/ui-interaction.md](references/ui-interaction.md) | Driving auth screens, AI-vs-human inputs, FLAG_SECURE gotcha, selector strategy | @@ -292,8 +334,14 @@ Load these as needed (don't preload all): ## Guardrails -- **Never** print, log, or commit real or lab credentials, tokens, or secrets. Type them onto the device only. -- **Artifacts stay outside the repo** (the run folder). Never commit logs/screenshots. +- **Never** print, log, or commit real or lab credentials, tokens, or secrets. Type them onto the device + only (use `input-text -Secret` so the value never hits the transcript). +- **Artifacts stay outside the repo** (the run folder). Never commit logs/screenshots/reports. +- **For an ADO test case, always generate the HTML + Markdown report** (`report.ps1`) on every outcome — + PASS, FAIL, BLOCKED, or PARTIAL. A run driven from a test case is not "done" until the report exists. +- **Prefer an emulator when a step needs an injectable fingerprint/biometric** (App Lock, biometric-gated + number-match). `adb emu finger touch` works only on emulators; a physical device needs a human at the + sensor. See [references/common-blockers.md](references/common-blockers.md). - **When delegating to a sub-agent, wait for its verdict** before reporting or ending the turn; restart it if it hangs and the test isn't done, terminate it once the result is in. Never surface "done" without the sub-agent's PASS/FAIL/BLOCKED evidence. diff --git a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md b/.github/skills/android-e2e-tester/references/app-and-module-map.md similarity index 93% rename from .github/skills/android-emulator-e2e-tester/references/app-and-module-map.md rename to .github/skills/android-e2e-tester/references/app-and-module-map.md index dcc2805a..7f4a9bb8 100644 --- a/.github/skills/android-emulator-e2e-tester/references/app-and-module-map.md +++ b/.github/skills/android-e2e-tester/references/app-and-module-map.md @@ -109,10 +109,14 @@ Brokered auth requires **a calling app + a broker app**, both installed: - Interactive tests (Mode B) need a test account. Preferred sources, in order: 1. A lab account the user provides in the session. 2. A credential the automation harness/Lab API exposes. +- **Provision or repair accounts on demand with `scripts/labapi.ps1`** (`create-user`, `reset`, + `enable-policy`/`disable-policy`, `delete-device`). Full endpoint list, usertypes/policies, the EasyAuth + auth workaround, and entitlements are in **[lab-api.md](lab-api.md)**. - **Blocker to raise with the user** if none is available, or if the scenario needs a real MSA/consumer - account, a specific tenant/CA policy, or a federated account that the AI cannot provision. -- **Never** hardcode, print, or commit real credentials. Type them into the device only; never echo them - into logs or the transcript. + account, a specific tenant/CA policy, or a federated account that the AI cannot provision (try + `labapi.ps1 disable-policy` first if it's a lab CA policy). +- **Never** hardcode, print, or commit real credentials. Type them into the device only (`input-text + -Secret`); never echo them into logs or the transcript. ## Emulator / real-device requirements per feature diff --git a/.github/skills/android-e2e-tester/references/common-blockers.md b/.github/skills/android-e2e-tester/references/common-blockers.md new file mode 100644 index 00000000..f89ce16e --- /dev/null +++ b/.github/skills/android-e2e-tester/references/common-blockers.md @@ -0,0 +1,158 @@ +# Common Scenarios, Hiccups & Blockers + +Recurring friction from real E2E runs, what causes it, and how to get past it — plus **when to switch +from a physical device to an emulator**. Read this before driving an auth flow so you don't lose time +rediscovering a known trap. See also [ui-interaction.md](ui-interaction.md) (screen-by-screen driving) +and [troubleshooting.md](troubleshooting.md) (env/build/install). + +Table of contents: +- [Decision: emulator vs physical device](#decision-emulator-vs-physical-device) +- [Steps that need a fingerprint / biometric / App Lock](#steps-that-need-a-fingerprint--biometric--app-lock) +- [Number-match MFA](#number-match-mfa) +- [Session timeouts & SSO resets mid-flow](#session-timeouts--sso-resets-mid-flow) +- [Chrome autofill / passkey overlay steals input](#chrome-autofill--passkey-overlay-steals-input) +- [FLAG_SECURE black screenshots](#flag_secure-black-screenshots) +- [Screenshot corruption via redirection](#screenshot-corruption-via-redirection) +- [Single-use pairing / setup links](#single-use-pairing--setup-links) +- [Stale account state between runs](#stale-account-state-between-runs) +- [Genuine blockers (stop and ask)](#genuine-blockers-stop-and-ask) +- [Quick reference table](#quick-reference-table) + +## Decision: emulator vs physical device + +The skill can run on either, but some steps are **only automatable on an emulator** because they need an +input you can inject programmatically. Choose up front from the test case's steps: + +**Prefer an emulator when the scenario includes any of:** +- A **fingerprint / biometric** prompt you must satisfy (enroll + inject a touch). +- **App Lock** in Microsoft Authenticator (it re-prompts for device credential/biometric). +- A **biometric-gated number-match** approval. +- Anything that needs `adb emu ...` (finger, sensors, GSM, battery) — emulator-only console commands. + +**A physical device is fine (or better) when:** +- No biometric is required (pure password + KMSI + token acquisition). +- You're on a **GPU-less host** (Cloud PC / VM / RDP) where the emulator is painfully slow — a physical + device is much faster there (see [emulator-performance.md](emulator-performance.md)). +- The feature needs real hardware (real FCM push, real SIM) — though real push MFA is still a human step. + +> Rule of thumb: **biometric/App-Lock → emulator; everything else → whatever is fastest** (usually a +> connected physical device on a Cloud PC). If you start on a physical device and hit an unavoidable +> fingerprint gate, that's the signal to re-run the biometric segment on an emulator. + +## Steps that need a fingerprint / biometric / App Lock + +`adb emu finger touch ` injects a fingerprint **only on an emulator**. A physical device has no adb +path to press the sensor — a human must do it. Microsoft Authenticator makes this bite because it +**enables App Lock by default** right after you register an account, so the *next* action (re-opening the +app, approving a browser number-match) is gated behind the device PIN/biometric. + +Options, best first: +1. **Run on an emulator.** Enroll once, then inject touches: + ```powershell + ./scripts/deviceui.ps1 finger-enroll -Serial # sets a PIN + enrolls (idempotent) + ./scripts/deviceui.ps1 finger -Text 1 -Serial # inject a touch when the prompt appears + ./scripts/deviceui.ps1 finger-status -Serial # verify one is enrolled + ``` +2. **Turn App Lock off** so no biometric is required: Authenticator → Settings → toggle **App Lock** off + (drive it with `tap-text`), then proceed with password-only steps. Do this early if the scenario + doesn't specifically test App Lock. +3. **Fall back to a device PIN.** If a keyguard/biometric prompt also accepts a PIN, set a known PIN during + setup and enter it with `deviceui.ps1` (many biometric prompts have a "Use PIN" path). +4. **Physical device + human.** If the scenario *must* run on hardware, pause and ask the user to press the + sensor, then resume — see [Genuine blockers](#genuine-blockers-stop-and-ask). + +## Number-match MFA + +Modern MFA shows a **number in the browser** that you must select/enter in Authenticator. It's +automatable **only if** getting into Authenticator isn't biometric-gated: +- If App Lock is on → satisfy the biometric first (see above) → then read the number from the browser + (`deviceui.ps1 dump` on the Chrome page — it's **not** FLAG_SECURE) and tap the matching tile in + Authenticator. +- If the number appears as a **push you must approve on a *separate* physical phone**, it's a genuine + blocker (no TOTP seed) — ask the user. + +## Session timeouts & SSO resets mid-flow + +Switching between the browser and Authenticator (or a slow manual segment) can **time out the eSTS web +session**. Symptoms: you return to the browser and it's back on the password page, or SSO silently +re-prompts. Mitigations: +- Keep the biometric/PIN setup done **before** you start the timed web segment so you don't stall on it. +- If you get bounced to sign-in, just re-drive the email/password screens (SSO often re-completes quickly). +- Don't leave the flow parked while doing long setup on the side — provision the account/emulator first, + then run the web segment in one go. + +## Chrome autofill / passkey overlay steals input + +On eSTS **WebView** email/password fields, Chrome's autofill / **passkey** overlay can intercept a bulk +`input text` — the value lands in the wrong field or is dropped, and eSTS shows "Enter a valid email". +Fix (baked into `deviceui.ps1`): +```powershell +./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial +./scripts/deviceui.ps1 input-text -Text $pw -Clear -CharByChar -Secret -Serial +``` +`-Clear` empties the field, `-CharByChar` types one character at a time (defeats the overlay), `-Secret` +keeps the value out of the transcript. If a "Save password / use passkey" bottom sheet pops, dismiss it +with `key ESCAPE` (not `BACK`, which can exit the app) before typing. **Verify by whether the page +advances**, not by reading the field's `text` — a WebView often doesn't reflect typed content back in the +accessibility tree (the password field `i0118` is an exception and does show a length). + +## FLAG_SECURE black screenshots + +eSTS login pages, some broker screens, and **Microsoft Authenticator** set `FLAG_SECURE`, so `screencap` +returns an all-black image and some nodes are hidden. Don't rely on screenshots to verify these: +- Verify state with `uiautomator dump` + XML parse instead (the account list, a specific `resource-id`, + the focused activity via `deviceui.ps1 current-app`). Save the **XML** as your evidence artifact. +- Chrome pages are **not** FLAG_SECURE and screenshot fine — capture those normally. +- Full detail: [ui-interaction.md](ui-interaction.md#the-flag_secure-gotcha). + +## Screenshot corruption via redirection + +`adb ... screencap -p > file.png` through PowerShell **corrupts the PNG** (newline translation on the +binary stream). Always capture on-device then pull — which is exactly what `deviceui.ps1 screenshot` does: +```powershell +adb shell screencap -p /sdcard/_sc.png ; adb pull /sdcard/_sc.png # never `screencap -p > file` +``` + +## Single-use pairing / setup links + +MFA-setup "pair your account to the app" deep-links (e.g. from `aka.ms/mfasetup`) and QR codes are often +**single-use** — a second attempt shows "code already used" or a stale QR. If a pairing step fails on a +retry, **re-generate** the link/QR from the setup page rather than reusing the old one, or provision a +fresh user (`labapi.ps1 create-user`). + +## Stale account state between runs + +A lab account that already completed first-time MFA setup will **skip** the very registration step you +want to test, making the run look like it "passed" without exercising anything. Reset the state: +```powershell +./scripts/labapi.ps1 reset -Upn $upn -Operation mfa # clear MFA registration +./scripts/labapi.ps1 create-user -UserType GlobalMFA # or just get a brand-new temp user +``` +See [lab-api.md](lab-api.md). + +## Genuine blockers (stop and ask) + +These can't be produced by the AI — report them and ask the user (see SKILL "When to ask the user"): +- **Real push-notification MFA** approval on a *separate* physical device with no TOTP seed. +- **SMS / phone-call OTP** to a real number. +- **Hardware** FIDO2 key, NFC, real camera/QR the emulator can't provide. +- **CAPTCHA** / "prove you're human". +- A **fingerprint on a physical device** that the scenario insists must run on that hardware. +- A credential/tenant/CA policy the AI can't provision (try `labapi.ps1 disable-policy` first if it's a + lab CA policy). + +## Quick reference table + +| Scenario / step | Automatable? | Where | How / workaround | +|---|---|---|---| +| Fingerprint / biometric prompt | ✅ emulator · ❌ physical | **Emulator** | `finger-enroll` + `finger`; on physical, human presses sensor | +| Authenticator **App Lock** re-prompt | ✅ emulator · ⚠️ physical | **Emulator** | inject biometric, or turn App Lock **off**, or use device PIN | +| Number-match MFA (same device) | ✅ if not biometric-gated | Either | read number from Chrome dump → tap tile in Authenticator | +| Real push MFA on another phone | ❌ | — | **Blocker** — ask the user | +| eSTS password typing | ✅ | Either | `input-text -Clear -CharByChar -Secret` | +| Autofill/passkey overlay | ✅ | Either | char-by-char + `key ESCAPE` to dismiss sheet | +| Verify state on FLAG_SECURE screen | ✅ | Either | `uiautomator dump` + XML (screenshot is black) | +| Session timed out mid-flow | ✅ | Either | re-drive sign-in; set up biometric/PIN before timed segment | +| Single-use pairing link reused | ✅ | Either | re-generate the link / fresh temp user | +| Stale MFA already registered | ✅ | Either | `labapi.ps1 reset -Operation mfa` or new temp user | +| SMS / phone / hardware-key / CAPTCHA | ❌ | — | **Blocker** — ask the user | diff --git a/.github/skills/android-emulator-e2e-tester/references/emulator-performance.md b/.github/skills/android-e2e-tester/references/emulator-performance.md similarity index 100% rename from .github/skills/android-emulator-e2e-tester/references/emulator-performance.md rename to .github/skills/android-e2e-tester/references/emulator-performance.md diff --git a/.github/skills/android-e2e-tester/references/lab-api.md b/.github/skills/android-e2e-tester/references/lab-api.md new file mode 100644 index 00000000..ac78703b --- /dev/null +++ b/.github/skills/android-e2e-tester/references/lab-api.md @@ -0,0 +1,133 @@ +# LAB API — Provisioning and Managing Test Accounts + +The **MSID LAB user-manager API** (`https://labusermanagerapi.azurewebsites.net`) creates and manages +lab test accounts for E2E runs. Drive it with `scripts/labapi.ps1`. The canonical, always-current list of +endpoints/parameters is the LAB **URL generator** web app: +. + +Table of contents: +- [Authentication (why a normal token fails)](#authentication-why-a-normal-token-fails) +- [Endpoints](#endpoints) +- [`labapi.ps1` usage](#labapips1-usage) +- [Response shape (create-user)](#response-shape-create-user) +- [When to use which endpoint](#when-to-use-which-endpoint) +- [Entitlements](#entitlements) + +## Authentication (why a normal token fails) + +The API sits behind **Azure App Service Authentication (EasyAuth)**. A service token from +`az account get-access-token` is rejected with `consent_required` — EasyAuth wants an **interactive** +Entra session, not a raw bearer token. The LAB URL-generator page works because your browser is already +signed in; the endpoint validates against that existing session. + +`labapi.ps1` reproduces that: it opens the endpoint in **headless Microsoft Edge** +(`--headless=new --dump-dom`), which reuses your Entra **WAM SSO** session, then parses the JSON the +endpoint returns. It caches an isolated Edge profile under `%TEMP%\labapi_edge_profile` so the first call +may do a silent WAM handshake and later calls are fast. If a call comes back needing sign-in, run it once +in a visible browser to seed the session: +```powershell +./scripts/labapi.ps1 open -Url "https://labusermanagerapi.azurewebsites.net/api/CreateTempUserID4SLab2?usertype=GlobalMFA" +``` + +> This is a **harness auth workaround**, not a code path under test. It only works on a machine where the +> operator is interactively signed in to Edge with an entitled account. + +## Endpoints + +Base URL: `https://labusermanagerapi.azurewebsites.net/api/` + +| Endpoint | Params | What it does | +|---|---|---| +| `CreateTempUserID4SLab2` | `usertype` | Creates a **temp cloud user** in the ID4SLab2 lab. **Auto-deletes after ~60 min.** Returns UPN + metadata. | +| `ResetID4SLab2` | `upn`, `operation` | Resets `mfa` or `password` for a user. **Password reset = temp users only.** | +| `EnablePolicyID4SLab2` | `upn`, `policy` | Enables a CA/special policy for a locked user. | +| `DisablePolicyID4SLab2` | `upn`, `policy` | Disables a CA/special policy for a locked user. | +| `DeleteDeviceID4SLab2` | `upn`, `deviceid` | Removes a device registration from Entra ID. | +| `List of Test Accounts` | `team` | KeyVault **deep-link** to the pre-created test-account JSON secret (team: `Android`, `JS`, `iOS`, `OneAuth`). Opens in the Azure Portal — use `labapi.ps1 open`. | +| `Fetch Password for Tenant` | `testTenant` | KeyVault **deep-link** to a tenant's password secret (`ID4SLAB2`, `ID4SLAB1`, `ARLMSIDLAB1`, `MNCMSIDLAB1`, `MSIDLAB4`, `MSIDLAB3`, `MSIDLAB8`). Opens in the Azure Portal. | + +**`usertype` values:** `Basic`, `GlobalMFA`, `MAMCA`, `MDMCA`, `MFAONSPO`, `MFAONEXO`, `FIDOBasic`, +`FIDOMDM`, `AuthappLBAC`, `AuthappRichContext`. + +**`policy` values:** `GlobalMFA`, `MAMCA`, `MDMCA`, `MFAONSPO`, `MFAONEXO`, `AuthappLBAC`, +`AuthappRichContext`. + +**`operation` values:** `mfa`, `password`. + +The two KeyVault items return **portal deep-links to a secret**, not a JSON API — they need an interactive +browser (`labapi.ps1 open`), and reading the secret needs the DevKV entitlement below. + +## `labapi.ps1` usage + +```powershell +# Create a fresh temp user of a given type (prints UPN=... on success): +./scripts/labapi.ps1 create-user -UserType GlobalMFA + +# Clear a stale MFA registration so a first-time-setup flow can be re-run cleanly: +./scripts/labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation mfa + +# Reset the password of a temp user: +./scripts/labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation password + +# Temporarily disable a CA policy that blocks a segment you're not testing, then re-enable it after: +./scripts/labapi.ps1 disable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA +./scripts/labapi.ps1 enable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA + +# Remove a device registration (e.g. clean up after a device-registration test): +./scripts/labapi.ps1 delete-device -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -DeviceId + +# Open a KeyVault deep-link (test-account list / tenant password) in a visible browser: +./scripts/labapi.ps1 open -Url "https://labusermanagerapi.azurewebsites.net/api/WebApp" + +# Debug: dump the raw DOM Edge returned instead of parsing JSON: +./scripts/labapi.ps1 create-user -UserType Basic -Raw +``` + +Flags: `-TimeoutSec ` (Edge virtual-time budget, default 30), `-Fresh` (throwaway Edge profile instead +of the cached one). + +## Response shape (create-user) + +`CreateTempUserID4SLab2` returns JSON like: +```json +{ + "title": "User Creation Successful", + "userType": "GlobalMFA", + "upn": "Locked_5b335908a3@ID4SLab2.onmicrosoft.com", + "passwordUri": "https://ms.portal.azure.com/#@.../Microsoft_Azure_KeyVault/Secret/https://msidlabs.vault.azure.net/secrets/ID4SLAB2", + "credentialVaultKeyName": "https://msidlabs.vault.azure.net:443/secrets/ID4SLab2", + "tenantId": "c7cef333-42af-492c-afb0-21f74a661133", + "tenantName": "ID4SLab2.onmicrosoft.com", + "labName": "ID4SLab2", + "authority": "https://login.microsoftonline.com/", + "objectId": "2007370f-74dc-4e03-b4b6-148838cd4323", + "userObject": { "UserPrincipalName": "Locked_5b335908a3@ID4SLab2.onmicrosoft.com", "...": "..." } +} +``` +`labapi.ps1 create-user` echoes the full JSON and a convenience `UPN=` line. The **password** for a +temp user is the shared lab password (kept in the KeyVault the `passwordUri` points to). In a run, the +user usually supplies the shared password directly — **never print it into the transcript**; type it with +`deviceui.ps1 input-text -Secret`. + +## When to use which endpoint + +- **Need a clean account** → `create-user`. Remember it self-destructs in ~60 min; provision it just + before the run, not at the start of a long setup. +- **A first-time-registration flow already registered on a prior attempt** (so the app skips the very step + you want to test) → `reset -Operation mfa` to clear MFA, or provision a brand-new user. +- **A CA policy blocks a segment you're not testing** (e.g. you want to test token acquisition but MFA + keeps interrupting) → `disable-policy`, run the segment, then `enable-policy` to restore state. +- **Device-registration test left a stale device** → `delete-device` to clean up so the next run starts fresh. +- **You need the exact UPNs of the durable, pre-created accounts** (not temp) → open the **List of Test + Accounts** KeyVault link for your team; get the tenant password via **Fetch Password for Tenant**. + +## Entitlements + +Request/manage at : + +- **`TM-MSIDLabs-Ext`** — required for **all** LAB APIs. Needs both **RO** and **RW**. +- **`TM-MSIDLABS-DevKV`** — required to read the **Mobile Build Vault** (the KeyVault deep-links). Needs + both **RO** and **RW**. + +If a call returns a sign-in/consent page instead of data, you're either not signed into Edge with an +entitled account or you're missing one of the above — that's a **user/setup blocker**, not a defect. diff --git a/.github/skills/android-emulator-e2e-tester/references/log-signals.md b/.github/skills/android-e2e-tester/references/log-signals.md similarity index 100% rename from .github/skills/android-emulator-e2e-tester/references/log-signals.md rename to .github/skills/android-e2e-tester/references/log-signals.md diff --git a/.github/skills/android-emulator-e2e-tester/references/mocking-flights-and-segments.md b/.github/skills/android-e2e-tester/references/mocking-flights-and-segments.md similarity index 100% rename from .github/skills/android-emulator-e2e-tester/references/mocking-flights-and-segments.md rename to .github/skills/android-e2e-tester/references/mocking-flights-and-segments.md diff --git a/.github/skills/android-e2e-tester/references/run-speed.md b/.github/skills/android-e2e-tester/references/run-speed.md new file mode 100644 index 00000000..e91c22b7 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/run-speed.md @@ -0,0 +1,116 @@ +# Run-Speed Analysis — why steps are slow and how to shorten them + +An analysis of where wall-clock time goes in a UI-driven run, using the AAD MFA sign-in run +(`20260721_184540`) as the reference, plus concrete speed-ups. This is guidance, not a hard gate — but the +default driving pattern should follow the "fast path" below. + +Table of contents: +- [Measured timeline](#measured-timeline) +- [Where the time actually goes](#where-the-time-actually-goes) +- [Root causes](#root-causes) +- [Speed-ups (what to do differently)](#speed-ups-what-to-do-differently) +- [Fast-path recipe](#fast-path-recipe) +- [Expected savings](#expected-savings) + +## Measured timeline + +Timestamps are screenshot capture times; the gap is the time that segment took. + +| Segment | From → To | Elapsed | What happened | +|---|---|---:|---| +| App first-run | — → 01_firstrun 18:48:16 | (setup) | launch + accept privacy | +| Add account → sign-in page | 01 → 02_chrome 18:53:30 | **5m14s** | navigate menus, load eSTS WebView, type UPN | +| Enter password → post-sign-in | 02 → 03_after_signin 18:57:35 | **4m05s** | type password (char-by-char + retries), eSTS round-trips | +| MFA wizard | 03 → 04_mfa_wizard 18:58:11 | 36s | "More info required" wizard | +| Pairing | 04 → 05_pairing 18:58:54 | 43s | pair account | +| MFA challenge | 05 → 07_mfa_challenge 19:03:33 | **4m39s** | switch to browser, re-auth, number-match appears | +| Number match | 07 → 08_number_match 19:04:55 | 1m22s | read number, attempt approval (blocked by App Lock) | + +**UI total ≈ 16m40s**, plus **≈3–5m** of setup (provision account, install/verify APK, device lease). +A human does the same flow in **~3–4 minutes**. The gap is almost entirely **harness overhead**, not the +device or the network. + +## Where the time actually goes + +The three fat segments (5m14s, 4m05s, 4m39s) share the same shape — they're dominated by the +**observe→act loop overhead**, not by anything the app is doing: + +1. **Per-action adb round-trips.** A single "tap the button labeled X" is really: `uiautomator dump` → + `exec-out cat` the XML → parse → compute center → `input tap x y` → **fixed `Start-Sleep`** → often a + re-`dump` to confirm. That's 4–6 adb invocations and a hard sleep for **one** logical action, and each + screen has several actions. +2. **Fixed sleeps instead of polling.** Waiting a flat `Start-Sleep -Seconds 3–5` after every tap "to be + safe" is the single biggest tax. Screens that were ready in 300 ms still cost the full sleep; multiply + by dozens of actions across a run. +3. **Fresh process per tool call.** Every `powershell` tool call is a brand-new process that re-resolves + `adb`, re-reads env, and re-establishes the adb client each time — hundreds of ms of pure startup, paid + on every micro-step because steps were issued as separate calls. +4. **Char-by-char typing.** The autofill/passkey overlay forces one-character-at-a-time input at ~55–60 ms + per character; a 20-char password + a UPN is a couple of seconds *just typing*, before any verification. +5. **Screenshot capture + pull + view.** `screencap` on-device → `adb pull` → open the PNG is ~1–2 s each; + taking one after every step to "see what happened" adds up fast (and is wasted on FLAG_SECURE screens + that come back black). +6. **Retries when input didn't land.** WebView typing that silently dropped (autofill) triggered re-typing + and re-verification — the 4m05s password segment is mostly this. +7. **Genuinely slow bits (unavoidable-ish).** eSTS WebView first paint, the switch to the browser for the + challenge, and any re-auth after a session timeout are real seconds — but they're the minority. + +## Root causes + +- **Chatty, one-action-per-call driving** with a fresh shell each time → startup + round-trip cost paid + hundreds of times. +- **Pessimistic fixed sleeps** substituting for readiness signals. +- **Verify-by-re-dump / verify-by-screenshot** after every action instead of only at decision points. +- **Overlay-forced char-by-char** typing on every field, even when bulk would have worked. +- **No reuse** of the uiautomator dump: we re-dump for the next action instead of reusing the XML we just + fetched. + +## Speed-ups (what to do differently) + +Ordered by payoff: + +1. **Keep one long-lived shell for a whole screen/segment.** Batch the dump→parse→tap(s) for a screen into + a *single* `powershell` call (an async session you reuse) so you pay process/adb startup once per + segment, not once per tap. This alone removes most of the fresh-process tax (root cause 3). +2. **Replace fixed sleeps with `wait-text` polling.** `deviceui.ps1 wait-text -Text ""` + returns the instant the screen is ready instead of always waiting N seconds. Use it after every + navigation instead of `Start-Sleep`. Biggest single win (root cause 2). +3. **Reuse one dump for multiple actions.** When a screen has several fields/buttons, `dump` once, then + compute all the tap targets from that one XML rather than re-dumping per element. +4. **Try bulk input first, fall back to char-by-char.** Attempt `input-text` (bulk) once; only switch to + `-CharByChar` if verification shows it didn't land. Don't pay per-char cost on fields where the overlay + isn't present (root cause 4). `input-text -Clear` already clears in **one** adb call (MOVE_END + bulk + DEL) instead of many. +5. **Verify at decision points, not after every micro-action.** Confirm state only where a wrong turn is + costly (did the password page advance? is the account in the list?). Skip the reflexive re-dump after + taps you're confident about. +6. **Stop screenshotting FLAG_SECURE screens.** They come back black — capturing them is pure waste. Use a + single `uiautomator dump` as evidence instead, and reserve screenshots for the non-secure screens that + actually render (root cause 5). See [common-blockers.md](common-blockers.md#flag_secure-black-screenshots). +7. **Provision with cached SSO.** `labapi.ps1` reuses a cached Edge profile so the account-creation call + isn't paying an interactive sign-in each time; provision the account **just before** the run (temp users + expire in ~60 min) so setup overlaps nothing. +8. **Prefer a connected physical device on GPU-less hosts.** On a Cloud PC/VM the emulator renders in + software and is far slower; unless a step needs an injectable fingerprint, a physical device removes a + whole class of slowness (see [emulator-performance.md](emulator-performance.md)). + +## Fast-path recipe + +Per screen: +1. `wait-text` on an anchor for the *expected* screen (no fixed sleep). +2. `dump` **once**; compute every tap/field target from that XML. +3. Do the taps / `input-text` (bulk first) in the **same** shell call. +4. Verify **once** by the next screen's anchor (`wait-text`) — that both confirms success *and* is the wait + for the following screen. One signal does double duty. +5. Screenshot only if the screen is non-secure and you need it as evidence. + +## Expected savings + +The three fat segments are ~80% overhead. Realistic targets: +- **Fixed-sleep → poll** and **batch-per-screen** together typically cut those segments by **50–70%**. +- Dropping FLAG_SECURE screenshots and redundant re-dumps trims another chunk. +- A ~16m40s UI run should land around **5–7 minutes** — within ~2× of a human instead of ~5×, with the + remainder being genuine eSTS/WebView load and any real re-auths. + +None of these change *what* is tested — they only remove harness idle time. Apply them by default; fall +back to the slower, more defensive pattern only on a screen that's proving flaky. diff --git a/.github/skills/android-e2e-tester/references/test-reporting.md b/.github/skills/android-e2e-tester/references/test-reporting.md new file mode 100644 index 00000000..64ccd099 --- /dev/null +++ b/.github/skills/android-e2e-tester/references/test-reporting.md @@ -0,0 +1,107 @@ +# Test Reporting (mandatory for ADO test cases) + +Every run driven from an **Azure DevOps test case** must end with a written report — on **every** outcome +(PASS, FAIL, BLOCKED, PARTIAL), not just success. Generate it with `scripts/report.ps1`, which renders a +polished **`TestReport.html`** plus a **`TestReport.md`** from a small run-metadata JSON. This is Phase 7 +of the [SKILL](../SKILL.md) and a hard guardrail. + +Table of contents: +- [Why mandatory](#why-mandatory) +- [How it works](#how-it-works) +- [Run-JSON schema](#run-json-schema) +- [Worked example](#worked-example) +- [Rendering the report](#rendering-the-report) +- [Rules](#rules) + +## Why mandatory + +- ADO test execution needs an auditable artifact you can attach to the test run / share with the team. +- A **BLOCKED** or **PARTIAL** outcome is still a result — the report is how you communicate *what* was + verified, *where* it stopped, and *why* (env constraint vs product defect). Skipping the report on a + non-PASS hides exactly the information the team needs. +- It forces you to collect evidence (screenshots, XML dumps, logcat scans) as you go instead of + reconstructing them afterward. + +## How it works + +`report.ps1 render -In ` reads a JSON file describing the run and writes `TestReport.html` and +`TestReport.md` next to it (or to `-OutDir`). The renderer is **defensive**: any missing field is simply +omitted, so a half-finished run still produces a valid report. Build the JSON incrementally during the run +(append to the `steps` array as you complete each step) so that even an early failure has a report to emit. + +## Run-JSON schema + +All fields are optional **except `title` and `verdict`**. Never put passwords/tokens in the JSON — include +the account **UPN only**. + +| Field | Type | Notes | +|---|---|---| +| `title` | string | **Required.** Test-case title. | +| `verdict` | string | **Required.** `PASS` \| `FAIL` \| `BLOCKED` \| `PARTIAL`. Drives the colored banner. | +| `verdictNote` | string | One-line justification (esp. for BLOCKED/PARTIAL — say if it's an env constraint). | +| `feature` | string | Human name of the flow under test. | +| `ado` | object | `{ testCaseId, planId, suiteId, url }`. | +| `device` | object | `{ model, serial, os, resolution, type }` (`type`: `physical`/`emulator`). | +| `app` | object | `{ package, version }`. | +| `account` | object | `{ upn, usertype, tenant }` — **UPN only, no password**. | +| `started` / `finished` | string | Timestamps (any readable format). | +| `iterations` | number | How many attempts the fix-loop took. | +| `steps` | array | `{ n, action, expected, result, notes, screenshot }` per step. `result`: PASS/FAIL/BLOCKED/SKIPPED. | +| `evidence` | array of string | Positive success signals (e.g. "account appears in list — 06_list.xml"). | +| `blockers` | array of string | What stopped a full E2E pass. | +| `artifacts` | array of string | Relative paths to logs/screenshots/XML in the run folder. | + +## Worked example + +`run.json` (mirrors the AAD MFA sign-in run): +```json +{ + "title": "Register AAD MFA cloud account via Sign in flow", + "verdict": "PARTIAL", + "verdictNote": "Core objective met (account registered); browser number-match blocked by Authenticator App Lock — an environment constraint on a physical device, not a product defect.", + "feature": "AAD MFA sign-in + first-time MFA setup", + "ado": { "testCaseId": 1579381, "planId": 714514, "suiteId": 3503165, + "url": "https://identitydivision.visualstudio.com/Engineering/_testPlans/define?planId=714514&suiteId=3503165" }, + "device": { "model": "Samsung SM-F741U1", "serial": "R5CXB0P430X", "os": "Android 16 (SDK 36)", + "resolution": "1080x2640", "type": "physical" }, + "app": { "package": "com.azure.authenticator", "version": "6.2607.4584" }, + "account": { "upn": "Locked_5b335908a3@ID4SLab2.onmicrosoft.com", "usertype": "GlobalMFA", + "tenant": "ID4SLab2.onmicrosoft.com" }, + "started": "2026-07-21 18:45", "finished": "2026-07-21 19:05", "iterations": 1, + "steps": [ + { "n": 1, "action": "Launch Authenticator, accept first-run", "expected": "First-run/privacy screen", + "result": "PASS", "screenshot": "iter1/01_firstrun.png" }, + { "n": 2, "action": "Add account > Work/School > Sign in", "expected": "eSTS sign-in WebView", + "result": "PASS", "screenshot": "iter1/02_chrome.png" }, + { "n": 3, "action": "Enter UPN + password (char-by-char, secret)", "expected": "Password accepted", + "result": "PASS", "notes": "bulk input swallowed by autofill; -CharByChar worked", + "screenshot": "iter1/03_after_signin.png" }, + { "n": 4, "action": "Complete first-time MFA setup wizard", "expected": "Account paired", + "result": "PASS", "screenshot": "iter1/05_pairing.png" }, + { "n": 5, "action": "Approve browser number-match", "expected": "Number entered in Authenticator", + "result": "BLOCKED", "notes": "App Lock gates entry behind PIN/biometric — not injectable on a physical device", + "screenshot": "iter1/08_number_match.png" } + ], + "evidence": [ "Account 'Locked_5b33...' appears in Authenticator account list (06_authenticator_accountlist.xml)" ], + "blockers": [ "Authenticator App Lock requires device biometric/PIN to re-enter and approve the number-match; run this segment on an emulator to inject a fingerprint (see common-blockers.md)." ], + "artifacts": [ "iter1/01_firstrun.png", "iter1/06_authenticator_accountlist.xml", "iter1/logcat_scan.txt" ] +} +``` + +## Rendering the report + +```powershell +./scripts/report.ps1 render -In C:\Users\\android-e2e-runs\aad-mfa-signin-\run.json +# → writes TestReport.html and TestReport.md into that run folder +``` +Optionally target a different folder with `-OutDir`. Reports live in the **run folder outside the repo** — +never commit them. + +## Rules + +- **Mandatory for ADO cases, every outcome.** No report ⇒ the run isn't done. +- **UPN only, never the password/token** in the JSON or report. +- Distinguish **environment constraint** from **product defect** in `verdictNote`/`blockers` — a BLOCKED + due to a physical-device biometric gate is not a bug; say so and point at the emulator workaround. +- Keep the run folder self-contained: reference screenshots/XML by **relative path** so the HTML links work + when the folder is zipped and shared. diff --git a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md b/.github/skills/android-e2e-tester/references/troubleshooting.md similarity index 97% rename from .github/skills/android-emulator-e2e-tester/references/troubleshooting.md rename to .github/skills/android-e2e-tester/references/troubleshooting.md index daf09bf0..eba28186 100644 --- a/.github/skills/android-emulator-e2e-tester/references/troubleshooting.md +++ b/.github/skills/android-e2e-tester/references/troubleshooting.md @@ -1,5 +1,10 @@ # Troubleshooting — Environment, Build, Install, and Test Failures +> For **in-flow UI hiccups** (fingerprint/App-Lock, number-match MFA, session timeouts, autofill overlay, +> FLAG_SECURE black screenshots, single-use pairing links) see +> [common-blockers.md](common-blockers.md). This file covers **environment/build/install/tooling** +> failures — the things that go wrong *before or around* the flow rather than *inside* it. + Table of contents: - [SDK / tooling not found](#sdk--tooling-not-found) - [Emulator won't start or boot](#emulator-wont-start-or-boot) diff --git a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md b/.github/skills/android-e2e-tester/references/ui-interaction.md similarity index 92% rename from .github/skills/android-emulator-e2e-tester/references/ui-interaction.md rename to .github/skills/android-e2e-tester/references/ui-interaction.md index d9d36b5e..7300a44e 100644 --- a/.github/skills/android-emulator-e2e-tester/references/ui-interaction.md +++ b/.github/skills/android-e2e-tester/references/ui-interaction.md @@ -12,6 +12,10 @@ Table of contents: Drive the UI with `scripts/deviceui.ps1`. The goal: perform every input a human tester *could* reasonably do, automatically — and stop to ask only when an input genuinely cannot be produced by the AI. +> **See also:** [common-blockers.md](common-blockers.md) for recurring hiccups and the emulator-vs-device +> decision (fingerprint/App-Lock, number-match, session timeouts, autofill overlay), and +> [run-speed.md](run-speed.md) for the fast driving pattern (poll instead of fixed sleeps; batch per screen). + ## Interaction loop For each step: @@ -36,8 +40,8 @@ pick among duplicates). When text is empty (icon-only buttons), use `tap-desc`. | **App first-run / runtime permissions** | Tap `Allow` / `While using the app` / `Continue`; or pre-grant with `appcontrol.ps1 grant`. | | **Test app main screen** | Tap the acquire-token control (e.g. `Acquire Token`, `Sign in`, `AcquireTokenSilent`). Set scopes/authority fields first if the scenario needs them. | | **Broker account picker** | If the target account is listed, `tap-text` it (SSO). If `Add account` / `Use another account`, tap it to start interactive sign-in. | -| **eSTS email page** | `input-text` the username, then `tap-text "Next"`. | -| **eSTS password page** | `input-text` the password, then `tap-text "Sign in"`. Never log the value. | +| **eSTS email page** | `input-text` the username (add `-Clear -CharByChar` if autofill steals bulk input), then `tap-text "Next"`. | +| **eSTS password page** | `input-text -Secret` the password (add `-Clear -CharByChar`), then `tap-text "Sign in"`. Never log the value. Verify by whether the page advances (the field text often doesn't reflect back). See [common-blockers.md](common-blockers.md#chrome-autofill--passkey-overlay-steals-input). | | **Consent / permissions requested** | `tap-text "Accept"` (safe for test apps/accounts). | | **"Stay signed in?" / KMSI** | `tap-text "Yes"` (or `No` if the scenario needs a fresh session). | | **"Approve sign in request" (push MFA)** | Usually a **blocker** — see below, unless a TOTP/seed is available. | diff --git a/.github/skills/android-emulator-e2e-tester/scripts/appcontrol.ps1 b/.github/skills/android-e2e-tester/scripts/appcontrol.ps1 similarity index 100% rename from .github/skills/android-emulator-e2e-tester/scripts/appcontrol.ps1 rename to .github/skills/android-e2e-tester/scripts/appcontrol.ps1 diff --git a/.github/skills/android-emulator-e2e-tester/scripts/authlogs.ps1 b/.github/skills/android-e2e-tester/scripts/authlogs.ps1 similarity index 100% rename from .github/skills/android-emulator-e2e-tester/scripts/authlogs.ps1 rename to .github/skills/android-e2e-tester/scripts/authlogs.ps1 diff --git a/.github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 b/.github/skills/android-e2e-tester/scripts/devicelease.ps1 similarity index 100% rename from .github/skills/android-emulator-e2e-tester/scripts/devicelease.ps1 rename to .github/skills/android-e2e-tester/scripts/devicelease.ps1 diff --git a/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 b/.github/skills/android-e2e-tester/scripts/deviceui.ps1 similarity index 86% rename from .github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 rename to .github/skills/android-e2e-tester/scripts/deviceui.ps1 index ed268a1e..b027e8f5 100644 --- a/.github/skills/android-emulator-e2e-tester/scripts/deviceui.ps1 +++ b/.github/skills/android-e2e-tester/scripts/deviceui.ps1 @@ -13,6 +13,8 @@ ./deviceui.ps1 wait-text -Text "Sign in" -TimeoutSec 30 ./deviceui.ps1 tap-text -Text "Sign in" ./deviceui.ps1 input-text -Text "user@contoso.com" + ./deviceui.ps1 input-text -Text "user@contoso.com" -Clear -CharByChar # defeat autofill/passkey overlays + ./deviceui.ps1 input-text -Text $pw -Clear -CharByChar -Secret # never echoes the value ./deviceui.ps1 key -Text ENTER ./deviceui.ps1 finger-status # is a fingerprint enrolled? ./deviceui.ps1 finger-enroll # enroll one on an emulator (or prompt if a real device) @@ -23,8 +25,12 @@ .NOTES Text matching is case-insensitive substring by default. Use -Exact for equality. Always re-`dump` after an action; the UI tree changes between steps. + input-text options: -Clear empties the focused field first (one adb round-trip); -CharByChar types one + character at a time (defeats Chrome autofill/passkey overlays that swallow a bulk `input text`); + -Secret suppresses echoing the value to the transcript (prints length only) — always use it for passwords. Fingerprint: `emu finger` only works on EMULATORS. On a real device a fingerprint must be enrolled - by the user against the physical sensor — `finger-enroll` will print the steps and ask. + by the user against the physical sensor — `finger-enroll` will print the steps and ask. When a step + needs an injectable fingerprint/biometric (App Lock, biometric-gated number-match), prefer an emulator. #> [CmdletBinding()] param( @@ -41,7 +47,11 @@ param( [switch]$Exact, [int]$TimeoutSec = 20, [string]$Pin = '1234', - [string]$Out + [string]$Out, + [switch]$Clear, + [switch]$CharByChar, + [int]$PerCharDelayMs = 60, + [switch]$Secret ) $ErrorActionPreference = 'Stop' @@ -185,9 +195,26 @@ switch ($Command) { Write-Host "Tapped ($X,$Y)" } 'input-text' { - $enc = Encode-Input $Text - Adb shell input text $enc | Out-Null - Write-Host "Typed: $Text" + # Optionally clear the focused field first: MOVE_END then a burst of DEL — all in ONE adb call. + if ($Clear) { + $clearCodes = @('input', 'keyevent', '123') + (1..80 | ForEach-Object { '67' }) + Adb shell $clearCodes | Out-Null + } + if ($CharByChar) { + # Type one character at a time. Chrome autofill / passkey overlays swallow a bulk `input text` + # (the whole string lands in an unexpected field or is dropped); per-char typing lands reliably. + foreach ($ch in $Text.ToCharArray()) { + Adb shell input text (Encode-Input ([string]$ch)) | Out-Null + if ($PerCharDelayMs -gt 0) { Start-Sleep -Milliseconds $PerCharDelayMs } + } + if ($Secret) { Write-Host "Typed (char-by-char): [$($Text.Length) chars]" } + else { Write-Host "Typed (char-by-char): $Text" } + } + else { + Adb shell input text (Encode-Input $Text) | Out-Null + if ($Secret) { Write-Host "Typed: [$($Text.Length) chars]" } + else { Write-Host "Typed: $Text" } + } } 'wait-text' { $deadline = (Get-Date).AddSeconds($TimeoutSec) diff --git a/.github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 b/.github/skills/android-e2e-tester/scripts/emulator.ps1 similarity index 100% rename from .github/skills/android-emulator-e2e-tester/scripts/emulator.ps1 rename to .github/skills/android-e2e-tester/scripts/emulator.ps1 diff --git a/.github/skills/android-e2e-tester/scripts/labapi.ps1 b/.github/skills/android-e2e-tester/scripts/labapi.ps1 new file mode 100644 index 00000000..839560bf --- /dev/null +++ b/.github/skills/android-e2e-tester/scripts/labapi.ps1 @@ -0,0 +1,174 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +<# +.SYNOPSIS + Call the MSID LAB user-manager API (https://labusermanagerapi.azurewebsites.net) to provision and + manage lab test accounts for E2E runs. The API is EasyAuth-protected (Azure App Service + Authentication), so a service token from `az account get-access-token` is rejected with + consent_required. This script authenticates the way the LAB "URL generator" web app does: it opens + the endpoint in **headless Edge**, which reuses your existing Entra WAM SSO session, then parses the + JSON the endpoint returns. + +.DESCRIPTION + Commands (function endpoints return JSON and are parsed automatically): + create-user -UserType CreateTempUserID4SLab2 (temp user, auto-deletes in 60 min) + reset -Upn -Operation mfa|password ResetID4SLab2 (password reset = temp users only) + enable-policy -Upn -Policy EnablePolicyID4SLab2 + disable-policy -Upn -Policy DisablePolicyID4SLab2 + delete-device -Upn -DeviceId DeleteDeviceID4SLab2 + open -Url launch an interactive browser (for KeyVault deep-links: + "List of Test Accounts" / "Fetch Password for Tenant") + + UserType : Basic | GlobalMFA | MAMCA | MDMCA | MFAONSPO | MFAONEXO | FIDOBasic | FIDOMDM | AuthappLBAC | AuthappRichContext + Policy : GlobalMFA | MAMCA | MDMCA | MFAONSPO | MFAONEXO | AuthappLBAC | AuthappRichContext + Operation: mfa | password + + Auth/entitlements: you must be signed into Edge with an account that holds TM-MSIDLabs-Ext (RO+RW) + and, for KeyVault deep-links, TM-MSIDLABS-DevKV (RO+RW). Manage at + https://coreidentity.microsoft.com/manage/entitlement . See references/lab-api.md. + +.EXAMPLE + ./labapi.ps1 create-user -UserType GlobalMFA + ./labapi.ps1 reset -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Operation mfa + ./labapi.ps1 disable-policy -Upn "Locked_xxx@ID4SLab2.onmicrosoft.com" -Policy GlobalMFA + +.NOTES + - Never print the fetched password into the transcript. `create-user` returns the UPN; the password + for temp users is the shared lab password the user supplies for the run. + - The first call in a session may pop a silent WAM prompt; subsequent calls reuse the cached profile + (a stable, isolated user-data-dir) so they're fast. Use -Fresh to force a throwaway profile. +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('create-user', 'reset', 'enable-policy', 'disable-policy', 'delete-device', 'open')] + [string]$Command = 'create-user', + + [string]$UserType = 'GlobalMFA', + [string]$Upn, + [ValidateSet('mfa', 'password')] + [string]$Operation, + [string]$Policy, + [string]$DeviceId, + [string]$Url, + [int]$TimeoutSec = 30, + [switch]$Fresh, + [switch]$Raw +) + +$ErrorActionPreference = 'Stop' +$ApiBase = 'https://labusermanagerapi.azurewebsites.net/api' + +function Get-Edge { + foreach ($p in @( + "$env:ProgramFiles\Microsoft\Edge\Application\msedge.exe", + "${env:ProgramFiles(x86)}\Microsoft\Edge\Application\msedge.exe")) { + if ($p -and (Test-Path $p)) { return $p } + } + $cmd = Get-Command msedge -ErrorAction SilentlyContinue + if ($cmd) { return $cmd.Source } + throw "Microsoft Edge not found. Install Edge, or use 'open' and complete the call in your own browser." +} + +function Invoke-LabApi { + param([string]$Endpoint) + $edge = Get-Edge + # Stable, isolated profile caches WAM SSO across calls (fast). -Fresh forces a throwaway dir. + $profile = if ($Fresh) { Join-Path $env:TEMP ("labapi_edge_" + [guid]::NewGuid().ToString('N')) } + else { Join-Path $env:TEMP 'labapi_edge_profile' } + $dom = Join-Path $env:TEMP ("labapi_dom_" + [guid]::NewGuid().ToString('N') + ".html") + $budget = [Math]::Max(5000, $TimeoutSec * 1000) + try { + & $edge --headless=new --disable-gpu --user-data-dir=$profile --no-first-run --no-default-browser-check ` + --dump-dom --virtual-time-budget=$budget $Endpoint 2>$null | + Out-File -FilePath $dom -Encoding utf8 + $html = if (Test-Path $dom) { Get-Content $dom -Raw } else { '' } + } + finally { + Remove-Item $dom -ErrorAction SilentlyContinue + if ($Fresh) { Remove-Item $profile -Recurse -Force -ErrorAction SilentlyContinue } + } + if ($Raw) { return [pscustomobject]@{ Raw = $html; Json = $null } } + + # 1) Prefer a 
 block (text/plain and application/json render inside 
).
+    $payload = $null
+    $m = [regex]::Match($html, ']*>(.*?)
', 'Singleline, IgnoreCase')
+    if ($m.Success) { $payload = [System.Net.WebUtility]::HtmlDecode(($m.Groups[1].Value -replace '<[^>]+>', '')) }
+    # 2) Fallback: first balanced-looking JSON object/array in the decoded body text.
+    if (-not $payload) {
+        $text = [System.Net.WebUtility]::HtmlDecode(([regex]::Replace($html, '<[^>]+>', ' ')))
+        $jm = [regex]::Match($text, '(\{.*\}|\[.*\])', 'Singleline')
+        if ($jm.Success) { $payload = $jm.Value }
+    }
+    if (-not $payload) {
+        if ($html -match 'AADSTS|Sign in|login\.microsoftonline|consent') {
+            throw "LAB API call did not return data — Edge appears to need an interactive sign-in/consent. Run '.\labapi.ps1 open -Url `"$Endpoint`"' once in a visible browser to establish the session, then retry."
+        }
+        throw "Could not extract a JSON payload from the LAB API response. Re-run with -Raw to inspect the DOM."
+    }
+    $json = $null
+    try { $json = $payload | ConvertFrom-Json } catch { }
+    return [pscustomobject]@{ Raw = $payload; Json = $json }
+}
+
+function Show-Result {
+    param($Result)
+    if ($Result.Json) {
+        $Result.Json | ConvertTo-Json -Depth 8
+    }
+    else {
+        Write-Host $Result.Raw
+    }
+}
+
+switch ($Command) {
+    'create-user' {
+        $ep = "$ApiBase/CreateTempUserID4SLab2?usertype=$([uri]::EscapeDataString($UserType))"
+        Write-Host "Creating temp $UserType user (auto-deletes in ~60 min)..."
+        $r = Invoke-LabApi $ep
+        Show-Result $r
+        # Surface the UPN prominently for the caller to reuse.
+        $upn = $null
+        if ($r.Json) {
+            foreach ($p in 'upn', 'Upn', 'UPN', 'userPrincipalName', 'username', 'User') {
+                if ($r.Json.PSObject.Properties.Name -contains $p -and $r.Json.$p) { $upn = $r.Json.$p; break }
+            }
+        }
+        if (-not $upn) { $m = [regex]::Match($r.Raw, '[\w.\-+]+@[\w.\-]+onmicrosoft\.com'); if ($m.Success) { $upn = $m.Value } }
+        if ($upn) { Write-Host "UPN=$upn" }
+    }
+    'reset' {
+        if (-not $Upn) { throw "-Upn is required." }
+        if (-not $Operation) { throw "-Operation mfa|password is required." }
+        $ep = "$ApiBase/ResetID4SLab2?upn=$([uri]::EscapeDataString($Upn))&operation=$([uri]::EscapeDataString($Operation))"
+        Write-Host "Resetting '$Operation' for $Upn ..."
+        Show-Result (Invoke-LabApi $ep)
+    }
+    'enable-policy' {
+        if (-not $Upn) { throw "-Upn is required." }
+        if (-not $Policy) { throw "-Policy is required (e.g. GlobalMFA)." }
+        $ep = "$ApiBase/EnablePolicyID4SLab2?upn=$([uri]::EscapeDataString($Upn))&policy=$([uri]::EscapeDataString($Policy))"
+        Write-Host "Enabling policy '$Policy' for $Upn ..."
+        Show-Result (Invoke-LabApi $ep)
+    }
+    'disable-policy' {
+        if (-not $Upn) { throw "-Upn is required." }
+        if (-not $Policy) { throw "-Policy is required (e.g. GlobalMFA)." }
+        $ep = "$ApiBase/DisablePolicyID4SLab2?upn=$([uri]::EscapeDataString($Upn))&policy=$([uri]::EscapeDataString($Policy))"
+        Write-Host "Disabling policy '$Policy' for $Upn ..."
+        Show-Result (Invoke-LabApi $ep)
+    }
+    'delete-device' {
+        if (-not $Upn) { throw "-Upn is required." }
+        if (-not $DeviceId) { throw "-DeviceId is required." }
+        $ep = "$ApiBase/DeleteDeviceID4SLab2?upn=$([uri]::EscapeDataString($Upn))&deviceid=$([uri]::EscapeDataString($DeviceId))"
+        Write-Host "Deleting device $DeviceId for $Upn ..."
+        Show-Result (Invoke-LabApi $ep)
+    }
+    'open' {
+        if (-not $Url) { throw "-Url is required (e.g. a List-of-Test-Accounts or Fetch-Password KeyVault deep-link)." }
+        $edge = Get-Edge
+        # Interactive (non-headless) so KeyVault/Azure Portal deep-links can render for the user.
+        & $edge $Url | Out-Null
+        Write-Host "Opened in Edge: $Url"
+    }
+}
diff --git a/.github/skills/android-e2e-tester/scripts/report.ps1 b/.github/skills/android-e2e-tester/scripts/report.ps1
new file mode 100644
index 00000000..014f27b2
--- /dev/null
+++ b/.github/skills/android-e2e-tester/scripts/report.ps1
@@ -0,0 +1,176 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+<#
+.SYNOPSIS
+    Render a standard E2E test report (TestReport.html + TestReport.md) from a run-metadata JSON file.
+    For ADO test cases this report is MANDATORY (Phase 7) — generate it on every outcome (PASS / FAIL /
+    BLOCKED / PARTIAL), not only on success.
+
+.DESCRIPTION
+    Feed it a JSON file describing the run; it writes TestReport.html and TestReport.md next to it (or to
+    -OutDir). The renderer is defensive: any missing field is simply omitted, so a partial run still
+    produces a report. NEVER put passwords/tokens in the JSON — include the account UPN only.
+
+    JSON schema (all fields optional except title + verdict):
+    {
+      "title":       "Register AAD MFA cloud account via Sign in flow",
+      "verdict":     "PASS",                 // PASS | FAIL | BLOCKED | PARTIAL
+      "verdictNote": "core objective met; browser number-match blocked by App Lock (env constraint)",
+      "feature":     "AAD MFA sign-in + first-time MFA setup",
+      "ado": { "testCaseId": 1579381, "planId": 714514, "suiteId": 3503165, "url": "https://..." },
+      "device": { "model": "Samsung SM-F741U1", "serial": "R5CX...", "os": "Android 16 (SDK 36)",
+                  "resolution": "1080x2640", "type": "physical" },
+      "app":     { "package": "com.azure.authenticator", "version": "6.2607.4584" },
+      "account": { "upn": "Locked_xxx@ID4SLab2.onmicrosoft.com", "usertype": "GlobalMFA", "tenant": "ID4SLab2.onmicrosoft.com" },
+      "started": "2026-07-21 18:45", "finished": "2026-07-21 19:05", "iterations": 1,
+      "steps": [ { "n":1, "action":"Launch Authenticator", "expected":"First-run screen",
+                   "result":"PASS", "notes":"...", "screenshot":"iter1/01_firstrun.png" } ],
+      "evidence":  [ "Account appears in Authenticator list (06_authenticator_accountlist.xml)" ],
+      "blockers":  [ "App Lock gates browser approval behind device PIN/biometric" ],
+      "artifacts": [ "iter1/logcat_scan.txt", "iter1/01_firstrun.png" ]
+    }
+
+.EXAMPLE
+    ./report.ps1 render -In C:\runs\aad-mfa\run.json
+    ./report.ps1 render -In C:\runs\aad-mfa\run.json -OutDir C:\runs\aad-mfa
+#>
+[CmdletBinding()]
+param(
+    [Parameter(Position = 0)][ValidateSet('render')][string]$Command = 'render',
+    [Parameter(Mandatory)][string]$In,
+    [string]$OutDir
+)
+
+$ErrorActionPreference = 'Stop'
+if (-not (Test-Path $In)) { throw "Run JSON not found: $In" }
+$run = Get-Content $In -Raw | ConvertFrom-Json
+if (-not $OutDir) { $OutDir = Split-Path (Resolve-Path $In) -Parent }
+if (-not (Test-Path $OutDir)) { New-Item -ItemType Directory -Force -Path $OutDir | Out-Null }
+
+function Val($o, $name) { if ($o -and ($o.PSObject.Properties.Name -contains $name)) { return $o.$name } return $null }
+function He([string]$s) { if ($null -eq $s) { return '' } [System.Net.WebUtility]::HtmlEncode([string]$s) }
+$verdict = ([string](Val $run 'verdict')).ToUpper(); if (-not $verdict) { $verdict = 'UNKNOWN' }
+$vColor = @{ PASS = '#107c10'; FAIL = '#d13438'; BLOCKED = '#c19c00'; PARTIAL = '#c19c00'; UNKNOWN = '#605e5c' }
+$vc = $vColor[$verdict]; if (-not $vc) { $vc = '#605e5c' }
+
+# ---------- Markdown ----------
+$md = New-Object System.Text.StringBuilder
+[void]$md.AppendLine("# E2E Test Report — $(Val $run 'title')")
+[void]$md.AppendLine("")
+[void]$md.AppendLine("**Verdict: $verdict**" + $(if (Val $run 'verdictNote') { " — $(Val $run 'verdictNote')" } else { '' }))
+[void]$md.AppendLine("")
+$ado = Val $run 'ado'
+if ($ado) {
+    $adoLine = @()
+    if (Val $ado 'testCaseId') { $adoLine += "Test Case #$(Val $ado 'testCaseId')" }
+    if (Val $ado 'planId') { $adoLine += "Plan $(Val $ado 'planId')" }
+    if (Val $ado 'suiteId') { $adoLine += "Suite $(Val $ado 'suiteId')" }
+    if ($adoLine.Count) { [void]$md.AppendLine("**ADO:** " + ($adoLine -join ' · ') + $(if (Val $ado 'url') { "  <$(Val $ado 'url')>" } else { '' })) }
+    [void]$md.AppendLine("")
+}
+function MdMeta($label, $val) { if ($val) { [void]$md.AppendLine("- **${label}:** $val") } }
+MdMeta 'Feature' (Val $run 'feature')
+$dev = Val $run 'device'
+if ($dev) { MdMeta 'Device' (@((Val $dev 'model'), (Val $dev 'os'), (Val $dev 'type'), (Val $dev 'resolution') | Where-Object { $_ }) -join ' · ') ; MdMeta 'Serial' (Val $dev 'serial') }
+$app = Val $run 'app'
+if ($app) { MdMeta 'App' (@((Val $app 'package'), (Val $app 'version') | Where-Object { $_ }) -join ' v') }
+$acct = Val $run 'account'
+if ($acct) { MdMeta 'Account' (@((Val $acct 'upn'), (Val $acct 'usertype') | Where-Object { $_ }) -join '  ·  ') }
+MdMeta 'Started' (Val $run 'started'); MdMeta 'Finished' (Val $run 'finished'); MdMeta 'Iterations' (Val $run 'iterations')
+[void]$md.AppendLine("")
+$steps = Val $run 'steps'
+if ($steps) {
+    [void]$md.AppendLine("## Steps")
+    [void]$md.AppendLine("")
+    [void]$md.AppendLine("| # | Action | Expected | Result | Notes |")
+    [void]$md.AppendLine("|---|--------|----------|--------|-------|")
+    foreach ($s in $steps) {
+        $n = Val $s 'n'; $a = (Val $s 'action') -replace '\|', '\|'; $e = (Val $s 'expected') -replace '\|', '\|'
+        $r = Val $s 'result'; $no = (Val $s 'notes') -replace '\|', '\|'
+        [void]$md.AppendLine("| $n | $a | $e | $r | $no |")
+    }
+    [void]$md.AppendLine("")
+}
+function MdList($label, $items) {
+    if ($items) { [void]$md.AppendLine("## $label"); [void]$md.AppendLine(""); foreach ($i in $items) { [void]$md.AppendLine("- $i") }; [void]$md.AppendLine("") }
+}
+MdList 'Evidence' (Val $run 'evidence')
+MdList 'Blockers' (Val $run 'blockers')
+MdList 'Artifacts' (Val $run 'artifacts')
+$mdPath = Join-Path $OutDir 'TestReport.md'
+$md.ToString() | Out-File -FilePath $mdPath -Encoding utf8
+
+# ---------- HTML ----------
+$rows = ''
+if ($steps) {
+    foreach ($s in $steps) {
+        $rc = @{ PASS = '#107c10'; FAIL = '#d13438'; BLOCKED = '#c19c00'; PARTIAL = '#c19c00' }[([string](Val $s 'result')).ToUpper()]
+        if (-not $rc) { $rc = '#605e5c' }
+        $shot = Val $s 'screenshot'
+        $shotCell = if ($shot) { "$(He (Split-Path $shot -Leaf))" } else { '' }
+        $rows += "$(He (Val $s 'n'))$(He (Val $s 'action'))$(He (Val $s 'expected'))" +
+        "$(He (Val $s 'result'))$(He (Val $s 'notes'))$shotCell"
+    }
+}
+function HtmlList($items) { if (-not $items) { return '' } $li = ($items | ForEach-Object { "
  • $(He $_)
    • " }) -join ''; return "
      $li
    " }
+function MetaRow($label, $val) { if ($val) { return "$(He $label)$(He $val)" } return '' }
+
+$adoRow = ''
+if ($ado) {
+    $bits = @()
+    if (Val $ado 'testCaseId') { $bits += "Test Case #$(Val $ado 'testCaseId')" }
+    if (Val $ado 'planId') { $bits += "Plan $(Val $ado 'planId')" }
+    if (Val $ado 'suiteId') { $bits += "Suite $(Val $ado 'suiteId')" }
+    $txt = $bits -join ' · '
+    if (Val $ado 'url') { $txt = "$(He $txt)" } else { $txt = He $txt }
+    $adoRow = "ADO$txt"
+}
+$devTxt = if ($dev) { (@((Val $dev 'model'), (Val $dev 'os'), (Val $dev 'type'), (Val $dev 'resolution') | Where-Object { $_ }) -join ' · ') } else { '' }
+$appTxt = if ($app) { (@((Val $app 'package'), (Val $app 'version') | Where-Object { $_ }) -join ' v') } else { '' }
+$acctTxt = if ($acct) { (@((Val $acct 'upn'), (Val $acct 'usertype'), (Val $acct 'tenant') | Where-Object { $_ }) -join '  ·  ') } else { '' }
+
+$html = @"
+
+
+E2E Test Report — $(He (Val $run 'title'))
+
    
+
    E2E Test Report — $(He (Val $run 'title'))
    
+
    $verdict
    
+$(if (Val $run 'verdictNote'){"
    $(He (Val $run 'verdictNote'))
    "})
+
    Run details
    
+
+$adoRow
+$(MetaRow 'Feature' (Val $run 'feature'))
+$(MetaRow 'Device' $devTxt)
+$(MetaRow 'Serial' $(if($dev){Val $dev 'serial'}))
+$(MetaRow 'App' $appTxt)
+$(MetaRow 'Account' $acctTxt)
+$(MetaRow 'Started' (Val $run 'started'))
+$(MetaRow 'Finished' (Val $run 'finished'))
+$(MetaRow 'Iterations' (Val $run 'iterations'))
+
    
+$(if($rows){"
    Steps
    $rows
    #ActionExpectedResultNotesScreenshot
    "})
+$(if(Val $run 'evidence'){"
    Evidence
    $(HtmlList (Val $run 'evidence'))"})
+$(if(Val $run 'blockers'){"
    Blockers
    $(HtmlList (Val $run 'blockers'))"})
+$(if(Val $run 'artifacts'){"
    Artifacts
    $(HtmlList (Val $run 'artifacts'))"})
+
    Generated by the android-e2e-tester skill · $(Get-Date -Format 'yyyy-MM-dd HH:mm')
    
+
    
+"@
+$htmlPath = Join-Path $OutDir 'TestReport.html'
+$html | Out-File -FilePath $htmlPath -Encoding utf8
+
+Write-Host "Wrote:"
+Write-Host "  $htmlPath"
+Write-Host "  $mdPath"
+Write-Host "Verdict: $verdict"

From 917d26b4fc7e681b5116f32d0f718214a4cf3b4a Mon Sep 17 00:00:00 2001
From: wzhipan 
Date: Tue, 21 Jul 2026 23:46:11 -0700
Subject: [PATCH 13/29] e2e skill: apply run-speed fast path (wait-text
 polling, tap -Then, bulk-first typing)

Section 5 of the skill-improvement request was documented in run-speed.md as
analysis only; this bakes those speed-ups into the scripts and prescribed workflow.

deviceui.ps1:
- Add shared Wait-ForText helper: immediate first check, then poll every -PollMs
  (default 600ms) until the -TimeoutSec deadline; returns the instant the anchor
  appears instead of paying a fixed sleep.
- wait-text now uses it (was a hardcoded Start-Sleep -Seconds 2 loop -> reaching a
  3rd probe drops from ~4s to ~1.2s).
- tap-text / tap-desc gain -Then : tap AND wait for the next screen in ONE
  process (removes the per-tap process/adb startup tax and the fixed-sleep tax),
  reusing wait-text's exit-4 timeout semantics.
- New params -Then / -PollMs; doc header documents them and the bulk-first typing rule.

SKILL.md + references/ui-interaction.md: replace the slow loop (act -> reflexive
re-dump -> screenshot every step; unconditional -CharByChar) with the fast path -
dump once per screen, verify by the next anchor via tap-text -Then, bulk input-text
first (add -CharByChar only if it didn't land), screenshot milestones only and skip
FLAG_SECURE screens.

references/run-speed.md: point the recommendations + fast-path recipe at the now-
shipped -Then / -PollMs mechanism.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
 .github/skills/android-e2e-tester/SKILL.md    | 20 +++++---
 .../references/run-speed.md                   | 18 +++++---
 .../references/ui-interaction.md              | 32 ++++++++-----
 .../android-e2e-tester/scripts/deviceui.ps1   | 46 +++++++++++++++----
 4 files changed, 82 insertions(+), 34 deletions(-)

diff --git a/.github/skills/android-e2e-tester/SKILL.md b/.github/skills/android-e2e-tester/SKILL.md
index b4e91f27..d968ed2b 100644
--- a/.github/skills/android-e2e-tester/SKILL.md
+++ b/.github/skills/android-e2e-tester/SKILL.md
@@ -199,19 +199,27 @@ Clear the log buffer first, then drive the flow:
 ./scripts/authlogs.ps1 clear -Serial 
 ```
 
-Loop per step using [references/ui-interaction.md](references/ui-interaction.md): `wait-text` for the
-expected element → act (`tap-text` / `input-text` / `key` / `finger`) → re-`dump` to confirm. Save a
-`screenshot` at each major step into the run folder.
+Loop per screen using [references/ui-interaction.md](references/ui-interaction.md), following the
+**fast path** (see [references/run-speed.md](references/run-speed.md)): `dump` **once** per screen and
+compute every target from that one XML → act → verify by the **next** screen's anchor with
+`tap-text -Then ""` (tap + wait in one call) or `wait-text`, **never a fixed `Start-Sleep`**.
+Batch the dump→tap(s) for a screen into a **single** shell call so process/adb startup is paid once per
+screen, not once per tap. Re-`dump` only when you must read genuinely new state; verify a navigation by its
+anchor, not a reflexive re-dump after every tap. Save a `screenshot` only at **milestones** and only on
+screens that actually render (skip FLAG_SECURE screens — they come back black; capture a `uiautomator dump`
+as evidence instead).
 
 **Auto-handle** everything the AI reasonably can: type lab test credentials, tap Next/Sign in/Accept/
 Allow/Yes, pick an account, grant permissions, simulate a fingerprint (`finger`), enter a TOTP if the
 seed is known, set/enter a device PIN. **Do not** print or commit credentials.
 
 **Typing into eSTS/WebView credential fields (hard-won).** The email/password pages are a WebView and
-Chrome's autofill/passkey overlay silently swallows a bulk `input text` (the value lands in the wrong
-field or is dropped, producing "Enter a valid email"). Type reliably:
+Chrome's autofill/passkey overlay can silently swallow a bulk `input text` (the value lands in the wrong
+field or is dropped, producing "Enter a valid email"). **Try bulk first** (it's fast); only fall back to
+`-CharByChar` if verification shows the value didn't land:
 ```powershell
-./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial           # username
+./scripts/deviceui.ps1 input-text -Text $upn -Clear -Serial                       # bulk first (fast)
+./scripts/deviceui.ps1 input-text -Text $upn -Clear -CharByChar -Serial           # fallback if it didn't land
 ./scripts/deviceui.ps1 input-text -Text $pw  -Clear -CharByChar -Secret -Serial   # password (never echoed)
 ```
 `-Clear` empties the field first, `-CharByChar` defeats the overlay, `-Secret` keeps the value out of the
diff --git a/.github/skills/android-e2e-tester/references/run-speed.md b/.github/skills/android-e2e-tester/references/run-speed.md
index e91c22b7..3f41aed4 100644
--- a/.github/skills/android-e2e-tester/references/run-speed.md
+++ b/.github/skills/android-e2e-tester/references/run-speed.md
@@ -72,9 +72,11 @@ Ordered by payoff:
 1. **Keep one long-lived shell for a whole screen/segment.** Batch the dump→parse→tap(s) for a screen into
    a *single* `powershell` call (an async session you reuse) so you pay process/adb startup once per
    segment, not once per tap. This alone removes most of the fresh-process tax (root cause 3).
-2. **Replace fixed sleeps with `wait-text` polling.** `deviceui.ps1 wait-text -Text ""`
-   returns the instant the screen is ready instead of always waiting N seconds. Use it after every
-   navigation instead of `Start-Sleep`. Biggest single win (root cause 2).
+2. **Replace fixed sleeps with anchor polling — and fuse tap+wait.** `deviceui.ps1 wait-text -Text
+   ""` returns the instant the screen is ready instead of always waiting N seconds.
+   Better still, `tap-text "