Ensure per-phase UIEvent creation in KIF tap simulation#1334
Merged
justinseanmartin merged 1 commit intokif-framework:masterfrom Sep 16, 2025
krayc425:master
Merged
Ensure per-phase UIEvent creation in KIF tap simulation#1334justinseanmartin merged 1 commit intokif-framework:masterfrom krayc425:master
justinseanmartin merged 1 commit intokif-framework:masterfrom
krayc425:master
Conversation
justinseanmartin
approved these changes
Sep 16, 2025
Contributor
justinseanmartin
left a comment
justinseanmartin
left a comment
There was a problem hiding this comment.
Seems like something that ought to be fairly benign on previous versions. I'll wait on CI results, but should be able to merge this if things looks good.
Contributor
|
Think that there are any other changes needed for iOS26 before cutting a release? It'd be nice to add CI support to validate it and see if anything else is broken. I might be able to get to that later this week. |
Contributor
Author
|
@justinseanmartin I think the Liquid Glass in the play would break some UI element testings, for example the TabBar / Alert / NavigationBar would have different UX or class hierarchy now. I haven't tested those since my use case doesn't enable Liquid Glass yet, but I think that part is worth an audit and fixes. |
Contributor
|
FYI - I cut a release for this fix. Feel free to start using |
andrzejchm
added a commit
to andrzejchm/fdb
that referenced
this pull request
Apr 28, 2026
…S 26 iOS 26 enforces stricter validation that the UIEvent's IOHIDEvent snapshot matches the current UITouch phase. The previous implementation reused the same UIEvent across Began/Ended phases, which UIKit silently dropped for non-UIControl responders (UIAlertController, UILabel, UIView). Reimplements iOS in-process tap to mirror KIF v3.12.2 (PR #1334): - Construct UITouch via [[alloc] init] then configure via private setters - Build a fresh UIEvent for EACH phase via _touchesEvent + _clearTouches - Attach a fresh IOHIDEvent (digitizer hand + finger sub-event) per phase Implementation moved to ObjC (FdbHelperNativeTap.h/m) because Swift's @convention(c) cannot represent the 17-argument C functions with by-value struct args required by IOKit's IOHIDEventCreateDigitizerEvent. Drops macOS support for fdb native-tap. cliclick path requires Accessibility permission, which macOS Sequoia/Tahoe only grants to signed .app bundles. Homebrew CLIs (cliclick, opencode, tmux) cannot receive it. Use fdb tap --at instead — it works without any system permissions via in-process fdb_helper. Verified working: - iOS 26 simulator: UIAlertController dismissal via fdb tap --at - iOS 26.3.1 physical (iPhone 17 Pro): same - Android (Android 16, real device): WebView HTML button via fdb tap --at - macOS: Flutter widget tap via fdb tap --at (counter increment verified) Renames test app bundle id from com.fdb.test.testApp to dev.andrzejchm.fdb.testApp following the flutter/packages convention. Removes hardcoded DEVELOPMENT_TEAM from project.pbxproj (also flutter/packages convention). Refs KIF PR #1334: kif-framework/KIF#1334
andrzejchm
added a commit
to andrzejchm/fdb
that referenced
this pull request
Apr 28, 2026
* feat: add fdb native-tap command for tapping native OS UI Adds `fdb native-tap --at x,y` to tap native (non-Flutter) UI elements such as iOS permission prompts, Android runtime-permission sheets, and macOS native dialogs — bypassing Flutter's GestureBinding entirely. Platform dispatch: - Android: `adb shell input tap` - iOS simulator: `cliclick` with CGWindowList-based window offset (no AX permission needed) - iOS physical: `idb ui tap` (requires idb from Meta) - macOS: `cliclick` with CGWindowList-based Flutter app window offset Closes #57 * fix: replace iOS sim cliclick+offset with idb, add macOS AX check - iOS simulator and physical now both use `idb ui tap` via IndigoHID (SimulatorKit), which correctly penetrates native OS dialogs and does not require fragile window-offset math or Accessibility permission - Drop all Simulator window CGWindowList lookup and hardcoded chrome offset constants — replaced by idb's own coordinate handling - macOS: detect missing Accessibility permission upfront via AXIsProcessTrusted() and print a clear, actionable error rather than silently dropping the CGEvent injection - Update README, agents doc, and skill file to reflect idb for both iOS targets and document the macOS AX requirement * feat: convert fdb_helper to Flutter plugin with native in-process tap injection Replaces GestureBinding-based coordinate taps with platform-native injection so that `fdb tap --at x,y` reaches native views overlaid on the Flutter surface — UIAlertController, WKWebView, AlertDialog, platform views — not just Flutter widgets. Implementation: - Pigeon (v26.3.4) defines the NativeTapApi interface, generating typed Dart/Swift/Kotlin bindings with zero hand-rolled codec code - iOS: UIApplication.sendEvent() with synthetic UITouch via UIKit private APIs (same pattern as EarlGrey/KIF); only used in debug builds - macOS: NSApplication.sendEvent() with synthetic NSEvent mouseDown/Up - Android: Activity.dispatchTouchEvent() with MotionEvent in physical pixels (Flutter logical → physical via displayMetrics.density); dispatched synchronously via CountDownLatch so the Pigeon reply is sent only after the tap completes - Selector-based taps (--key, --text, --type) are unchanged and still use GestureBinding after resolving widget coordinates - Unsupported platforms (web, Linux, Windows) fall back to GestureBinding * test: add native view test screen + fix iOS/macOS main-thread crash + use IndigoHIDMessageForMouseNSEvent Test app changes: - Add NativeViewTestScreen with embedded WebView (WKWebView/Android WebView) showing a native HTML button not in the Flutter widget tree - Wire to /native-view-test route and add navigation button - Add task test:native-view-tap Taskfile task verifying fdb tap --at reaches native platform views unreachable by --key - Add webview_flutter dependency to test app iOS / macOS fdb_helper fix: - Dispatch UIKit/AppKit sendEvent() on main thread via DispatchQueue.main.sync to prevent crash from UIKit access on Dart VM isolate thread - Use Thread.isMainThread check to avoid deadlock when already on main thread iOS native-tap fix (native_tap.dart): - Replace manual IndigoMessage struct construction (which used wrong phase values causing app crashes) with IndigoHIDMessageForMouseNSEvent from SimulatorKit — the same private function idb uses internally - This correctly constructs the Mach IPC message without corrupting uninitialized struct fields Status: - Android: fdb tap --at confirmed working via Activity.dispatchTouchEvent - iOS sim: fdb native-tap confirmed not crashing, reaches UIKit native views; WKWebView requires SimDeviceScreen registration (out of scope for v1) - macOS: fix applied, not yet tested end-to-end (no AX permission on this machine) * fix(ios): use KIF v3.12.2 approach for in-process tap injection on iOS 26 iOS 26 enforces stricter validation that the UIEvent's IOHIDEvent snapshot matches the current UITouch phase. The previous implementation reused the same UIEvent across Began/Ended phases, which UIKit silently dropped for non-UIControl responders (UIAlertController, UILabel, UIView). Reimplements iOS in-process tap to mirror KIF v3.12.2 (PR #1334): - Construct UITouch via [[alloc] init] then configure via private setters - Build a fresh UIEvent for EACH phase via _touchesEvent + _clearTouches - Attach a fresh IOHIDEvent (digitizer hand + finger sub-event) per phase Implementation moved to ObjC (FdbHelperNativeTap.h/m) because Swift's @convention(c) cannot represent the 17-argument C functions with by-value struct args required by IOKit's IOHIDEventCreateDigitizerEvent. Drops macOS support for fdb native-tap. cliclick path requires Accessibility permission, which macOS Sequoia/Tahoe only grants to signed .app bundles. Homebrew CLIs (cliclick, opencode, tmux) cannot receive it. Use fdb tap --at instead — it works without any system permissions via in-process fdb_helper. Verified working: - iOS 26 simulator: UIAlertController dismissal via fdb tap --at - iOS 26.3.1 physical (iPhone 17 Pro): same - Android (Android 16, real device): WebView HTML button via fdb tap --at - macOS: Flutter widget tap via fdb tap --at (counter increment verified) Renames test app bundle id from com.fdb.test.testApp to dev.andrzejchm.fdb.testApp following the flutter/packages convention. Removes hardcoded DEVELOPMENT_TEAM from project.pbxproj (also flutter/packages convention). Refs KIF PR #1334: kif-framework/KIF#1334 * chore(beads): add follow-up tickets from feature/native-tap PR * refactor: drop idb dependency from native-tap (physical iOS not yet supported) idb's 'ui tap' command is simulator-only — it errors on physical devices with 'Target doesn't conform to FBSimulatorLifecycleCommands protocol' (idb issue #853, open since Oct 2023, no fix planned). The physical iOS code path was therefore never functional. Removes the idb-based _tapIosPhysical implementation and replaces with a clear 'not yet supported' error pointing users to: - fdb tap --at (in-process via fdb_helper, works on physical iOS for in-app overlays like UIAlertController) - beads ticket fdb-6sz which tracks the WebDriverAgent-based replacement (the de-facto standard for physical iOS tap injection in mid-2026) Updates README, doc/README.agents.md, SKILL.md, and Taskfile to reflect the reduced platform support matrix: - Android (real + emulator) — adb shell input tap - iOS simulator — IndigoHID via SimulatorKit - iOS physical — not supported (use fdb tap --at) - macOS — not supported (use fdb tap --at) * chore(beads): recalibrate ticket priorities (P0–P4 scale) * chore(beads): refine ticket priorities after sanity check * chore(review): remove dead code and stale comments from native-tap - Drop unused FdbConfigureTouch helper (was never called). - Drop redundant setPhase/setTimestamp before Began event in FdbHelperNativeTapAtPoint — both are already set during touch setup, and FdbBuildEventForTouch updates the HID snapshot per phase. - Drop unused DEVICE var and STORED_DEVICE local in test:native-tap. - Rewrite stale comment in _simulatorScreenSize that promised a simctl io fallback that was never implemented. Verified iOS sim 26.2 smoke test still passes: task test:native-view-tap → UIAlertController dismissed via tap --at task test:native-tap → IndigoHID tap dispatched at 200,400 * fix(review): apply review findings 1, 2, 3, 4, 6, 7, 8 Findings from PR #63 review (sub-agent): 1. Restore long-press semantics for coordinate longpress. When rawDuration is provided (i.e. fdb longpress --at), bypass the native tap path (which is quick-tap only via Pigeon NativeTapApi) and use dispatchTap with holdDuration. Native long-press on overlays will be added in a follow-up ticket. 2. Remove leftover hardcoded DEVELOPMENT_TEAM = FZX57HL4C9 from the iOS Debug build config (the previous edit only cleaned the Release config). Matches the flutter/packages convention referenced in the PR description. 3. Update packages/fdb_helper/AGENTS.md to acknowledge gesture_dispatcher.dart as the home for all gesture-dispatch helpers regardless of caller count. The native-tap Pigeon dispatch belongs at the gesture-dispatch boundary. 4. Surface native-tap fallback to callers via WARNING= token. When the in-process native injection fails on a supported platform and we fall back to Flutter's GestureBinding, agents need to know native overlays were NOT actually reached. dispatchNativeTap now returns NativeTapResult (native | unsupportedPlatform | nativeFailedFallback); the handler propagates this to the JSON response; the CLI emits 'WARNING=native_tap_fallback'. 6. Fix lying return type on _simulatorScreenSize. The function always returns a value (falls back to iPhone 17 Pro dimensions); change return type from Future<(double,double)?> to Future<(double,double)> and remove the now-unreachable null check at the call site. 7. Surface Android dispatchTouchEvent timeout. The 2-second latch wait was silently ignored; now throws DISPATCH_TIMEOUT FlutterError if the UI thread doesn't dispatch within 2s. 8. iOS impl now validates required private selectors upfront via FdbMissingRequiredSelector. Returns a clear NSError naming the missing selector instead of throwing unrecognized-selector exceptions mid-tap when a future iOS version drops a private API. * chore(beads): file fdb-sip — native long-press by coordinate * docs(fdb_helper): document Linux/Windows fallback behavior on dispatchNativeTap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
-tapAtPoint:sometimes fails to trigger taps on non-UIControl views (e.g.UILabel,UIView).UIEventinstance for multiple touch phases (BeganandEnded).UIEventsnapshots.UITouchphase, the system may ignore the injected event, resulting in taps being dropped.Changes
-tapAtPoint:to create a newUIEventfor each touch phase (.beganand.ended).UITouchstate before sending.