feat: add fdb native-tap + native in-process tap for fdb tap --at#63
Merged
andrzejchm merged 13 commits intomainfrom Apr 28, 2026
Merged
feat: add fdb native-tap + native in-process tap for fdb tap --at#63andrzejchm merged 13 commits intomainfrom
andrzejchm merged 13 commits intomainfrom
Conversation
Adds `fdb native-tap --at x,y` to tap native (non-Flutter) UI elements such as iOS permission prompts, Android runtime-permission sheets, and macOS native dialogs — bypassing Flutter's GestureBinding entirely. Platform dispatch: - Android: `adb shell input tap` - iOS simulator: `cliclick` with CGWindowList-based window offset (no AX permission needed) - iOS physical: `idb ui tap` (requires idb from Meta) - macOS: `cliclick` with CGWindowList-based Flutter app window offset Closes #57
- iOS simulator and physical now both use `idb ui tap` via IndigoHID (SimulatorKit), which correctly penetrates native OS dialogs and does not require fragile window-offset math or Accessibility permission - Drop all Simulator window CGWindowList lookup and hardcoded chrome offset constants — replaced by idb's own coordinate handling - macOS: detect missing Accessibility permission upfront via AXIsProcessTrusted() and print a clear, actionable error rather than silently dropping the CGEvent injection - Update README, agents doc, and skill file to reflect idb for both iOS targets and document the macOS AX requirement
… injection Replaces GestureBinding-based coordinate taps with platform-native injection so that `fdb tap --at x,y` reaches native views overlaid on the Flutter surface — UIAlertController, WKWebView, AlertDialog, platform views — not just Flutter widgets. Implementation: - Pigeon (v26.3.4) defines the NativeTapApi interface, generating typed Dart/Swift/Kotlin bindings with zero hand-rolled codec code - iOS: UIApplication.sendEvent() with synthetic UITouch via UIKit private APIs (same pattern as EarlGrey/KIF); only used in debug builds - macOS: NSApplication.sendEvent() with synthetic NSEvent mouseDown/Up - Android: Activity.dispatchTouchEvent() with MotionEvent in physical pixels (Flutter logical → physical via displayMetrics.density); dispatched synchronously via CountDownLatch so the Pigeon reply is sent only after the tap completes - Selector-based taps (--key, --text, --type) are unchanged and still use GestureBinding after resolving widget coordinates - Unsupported platforms (web, Linux, Windows) fall back to GestureBinding
andrzejchm
changed the title
… use IndigoHIDMessageForMouseNSEvent Test app changes: - Add NativeViewTestScreen with embedded WebView (WKWebView/Android WebView) showing a native HTML button not in the Flutter widget tree - Wire to /native-view-test route and add navigation button - Add task test:native-view-tap Taskfile task verifying fdb tap --at reaches native platform views unreachable by --key - Add webview_flutter dependency to test app iOS / macOS fdb_helper fix: - Dispatch UIKit/AppKit sendEvent() on main thread via DispatchQueue.main.sync to prevent crash from UIKit access on Dart VM isolate thread - Use Thread.isMainThread check to avoid deadlock when already on main thread iOS native-tap fix (native_tap.dart): - Replace manual IndigoMessage struct construction (which used wrong phase values causing app crashes) with IndigoHIDMessageForMouseNSEvent from SimulatorKit — the same private function idb uses internally - This correctly constructs the Mach IPC message without corrupting uninitialized struct fields Status: - Android: fdb tap --at confirmed working via Activity.dispatchTouchEvent - iOS sim: fdb native-tap confirmed not crashing, reaches UIKit native views; WKWebView requires SimDeviceScreen registration (out of scope for v1) - macOS: fix applied, not yet tested end-to-end (no AX permission on this machine)
…S 26 iOS 26 enforces stricter validation that the UIEvent's IOHIDEvent snapshot matches the current UITouch phase. The previous implementation reused the same UIEvent across Began/Ended phases, which UIKit silently dropped for non-UIControl responders (UIAlertController, UILabel, UIView). Reimplements iOS in-process tap to mirror KIF v3.12.2 (PR #1334): - Construct UITouch via [[alloc] init] then configure via private setters - Build a fresh UIEvent for EACH phase via _touchesEvent + _clearTouches - Attach a fresh IOHIDEvent (digitizer hand + finger sub-event) per phase Implementation moved to ObjC (FdbHelperNativeTap.h/m) because Swift's @convention(c) cannot represent the 17-argument C functions with by-value struct args required by IOKit's IOHIDEventCreateDigitizerEvent. Drops macOS support for fdb native-tap. cliclick path requires Accessibility permission, which macOS Sequoia/Tahoe only grants to signed .app bundles. Homebrew CLIs (cliclick, opencode, tmux) cannot receive it. Use fdb tap --at instead — it works without any system permissions via in-process fdb_helper. Verified working: - iOS 26 simulator: UIAlertController dismissal via fdb tap --at - iOS 26.3.1 physical (iPhone 17 Pro): same - Android (Android 16, real device): WebView HTML button via fdb tap --at - macOS: Flutter widget tap via fdb tap --at (counter increment verified) Renames test app bundle id from com.fdb.test.testApp to dev.andrzejchm.fdb.testApp following the flutter/packages convention. Removes hardcoded DEVELOPMENT_TEAM from project.pbxproj (also flutter/packages convention). Refs KIF PR #1334: kif-framework/KIF#1334
…upported)
idb's 'ui tap' command is simulator-only — it errors on physical devices
with 'Target doesn't conform to FBSimulatorLifecycleCommands protocol'
(idb issue #853, open since Oct 2023, no fix planned). The physical iOS
code path was therefore never functional.
Removes the idb-based _tapIosPhysical implementation and replaces with a
clear 'not yet supported' error pointing users to:
- fdb tap --at (in-process via fdb_helper, works on physical iOS for
in-app overlays like UIAlertController)
- beads ticket fdb-6sz which tracks the WebDriverAgent-based replacement
(the de-facto standard for physical iOS tap injection in mid-2026)
Updates README, doc/README.agents.md, SKILL.md, and Taskfile to reflect
the reduced platform support matrix:
- Android (real + emulator) — adb shell input tap
- iOS simulator — IndigoHID via SimulatorKit
- iOS physical — not supported (use fdb tap --at)
- macOS — not supported (use fdb tap --at)
- Drop unused FdbConfigureTouch helper (was never called). - Drop redundant setPhase/setTimestamp before Began event in FdbHelperNativeTapAtPoint — both are already set during touch setup, and FdbBuildEventForTouch updates the HID snapshot per phase. - Drop unused DEVICE var and STORED_DEVICE local in test:native-tap. - Rewrite stale comment in _simulatorScreenSize that promised a simctl io fallback that was never implemented. Verified iOS sim 26.2 smoke test still passes: task test:native-view-tap → UIAlertController dismissed via tap --at task test:native-tap → IndigoHID tap dispatched at 200,400
Findings from PR #63 review (sub-agent): 1. Restore long-press semantics for coordinate longpress. When rawDuration is provided (i.e. fdb longpress --at), bypass the native tap path (which is quick-tap only via Pigeon NativeTapApi) and use dispatchTap with holdDuration. Native long-press on overlays will be added in a follow-up ticket. 2. Remove leftover hardcoded DEVELOPMENT_TEAM = FZX57HL4C9 from the iOS Debug build config (the previous edit only cleaned the Release config). Matches the flutter/packages convention referenced in the PR description. 3. Update packages/fdb_helper/AGENTS.md to acknowledge gesture_dispatcher.dart as the home for all gesture-dispatch helpers regardless of caller count. The native-tap Pigeon dispatch belongs at the gesture-dispatch boundary. 4. Surface native-tap fallback to callers via WARNING= token. When the in-process native injection fails on a supported platform and we fall back to Flutter's GestureBinding, agents need to know native overlays were NOT actually reached. dispatchNativeTap now returns NativeTapResult (native | unsupportedPlatform | nativeFailedFallback); the handler propagates this to the JSON response; the CLI emits 'WARNING=native_tap_fallback'. 6. Fix lying return type on _simulatorScreenSize. The function always returns a value (falls back to iPhone 17 Pro dimensions); change return type from Future<(double,double)?> to Future<(double,double)> and remove the now-unreachable null check at the call site. 7. Surface Android dispatchTouchEvent timeout. The 2-second latch wait was silently ignored; now throws DISPATCH_TIMEOUT FlutterError if the UI thread doesn't dispatch within 2s. 8. iOS impl now validates required private selectors upfront via FdbMissingRequiredSelector. Returns a clear NSError naming the missing selector instead of throwing unrecognized-selector exceptions mid-tap when a future iOS version drops a private API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two complementary commands for tapping native UI that Flutter's
GestureBindingcannot reach:fdb native-tap --at x,y— out-of-process tap via platform tools (Android:adb shell input tap; iOS sim: IndigoHID via SimulatorKit). Targets system-level UI that lives outside the app's window.fdb tap --at x,y(existing command, new behavior) — in-process tap via the newfdb_helperPigeonNativeTapApi. ReachesUIAlertController,AlertDialog, in-process WebView content, and platform views.Selector-based taps (
--key,--text,--type) continue to use Flutter'sGestureBindingfor backwards compatibility.The iOS 26 fix
The hardest part of this PR: iOS 26 broke the standard EarlGrey/KIF approach to
UITouch+UIEventinjection. The fix mirrors KIF v3.12.2 (PR #1334) which was released in Oct 2025:UIEvent's internalIOHIDEventsnapshot matches the currentUITouchphaseUIEventacross Began → Ended phases (the pre-iOS-26 idiom) is silently dropped for non-UIControlrespondersUIEventwith a freshIOHIDEventper phaseImplementation lives in
packages/fdb_helper/ios/Classes/FdbHelperNativeTap.{h,m}— ObjC because Swift's@convention(c)cannot represent IOKit's 17-argument digitizer event constructors with by-value struct args.Verified platforms
fdb tap --at(in-process)fdb native-tap(out-of-process)adb shell input tapfdb-6szWhat's NOT supported (intentionally)
fdb native-tapon physical iOS — idb'sui tapis simulator-only (idb #853, open since Oct 2023). The de-facto standard for physical iOS tap injection in 2026 is WebDriverAgent (signed XCUITest runner installed on device), which has higher setup burden than the current zero-setup paths. Usefdb tap --atinstead — it reaches in-app native overlays on physical iOS viafdb_helper. WDA-based replacement tracked infdb-6sz.fdb native-tapon macOS — cross-process click injection on macOS Sequoia/Tahoe requires Accessibility permission, which the system only grants to signed.appbundles. Homebrew CLIs (cliclick, opencode, tmux) cannot receive it. Usefdb tap --atinstead.Architecture
fdb_helperis now a plugin (was a pure-Dart package), with platform-specific Swift/Kotlin/ObjC underpackages/fdb_helper/{ios,android,macos}/.NativeTapApiprovides the typed Dart↔native bridge (packages/fdb_helper/pigeons/native_tap.dart).tap_handler.dartroute throughdispatchNativeTap→NativeTapApi.nativeTap. Selector-based taps unchanged.GestureBindingon platforms without native impl (web, Linux, Windows). Clear warning + fallback when supported platform's native call fails.Test app changes
NativeViewTestScreenwith embeddedWebViewWidgetfor verifying native injection on Android.MethodChannel("fdb_test/native_dialog").com.fdb.test.testApptodev.andrzejchm.fdb.testApp(matchesflutter/packagesconvention; old ID was claimed by another developer team).DEVELOPMENT_TEAMfromproject.pbxproj(also matchesflutter/packagesconvention — contributors set their own team via Xcode automatic signing).Smoke tests
task test:native-view-tapis now platform-aware:--at, asserts WebView title becomes "TAPPED"--at 269,488, asserts result text becomes "CONFIRMED"--at, asserts counter increasestask test:native-tapexercises the out-of-process path with sane error handling per platform (skips gracefully on iOS physical and macOS with clear messages).Follow-up tickets (filed)
fdb-oru(P2) — fdb launch: surface actionable errors instead of generic "process exited unexpectedly"fdb-9a8(P3) — fdb launch: add--verboseflag passing through toflutter run --verbosefdb-4e7(P3) — fdb devices: handle wireless/stale device discoveryfdb-dty(P3) — fdb doctor: physical iOS prerequisite checksfdb-6sz(P3) — fdb native-tap: implement physical iOS support via WebDriverAgentRefs
Sources/KIF/Additions/UIView-KIFAdditions.m,UITouch-KIFAdditions.m,IOHIDEvent+KIF.mui tapon physical iOS): iOS 17 support facebook/idb#853