INMO Air3 native client + ArUco fiducial alignment (phone validated, glasses WIP)#30
Open
IsiAzu wants to merge 4 commits into
Open
INMO Air3 native client + ArUco fiducial alignment (phone validated, glasses WIP)#30IsiAzu wants to merge 4 commits into
IsiAzu wants to merge 4 commits into
Conversation
Head orientation now tracks correctly on-device (IMA301, Android 14) using
INMO's pre-calibrated fusion quaternion (GyroRotation.vQuat), which bakes in
the true IMU mounting offset that remapCoordinateSystem could not represent
(see decision 011). Yaw/pitch/roll all correct; the old "yaw -> roll" bug is
gone. GlassesTracker (Game Rotation Vector) remains as a fallback.
Integrating the air3_core AAR outside Unity required three fixes, each found
by decompiling the AAR and reading on-device crashes:
- UnityPlayer stub: GyroRotation.start() reads UnityPlayer.currentActivity to
get the Activity for getSystemService("sensor"). The real Unity class isn't
bundled, so add a minimal com.unity3d.player.UnityPlayer with that one
static field; InmoFusionTracker sets it (now takes an Activity, not Context).
- System.loadLibrary("inmoair3"): the native fusion symbol
(AtwCore.nativeGyroFusion) lives in libinmoair3.so, normally loaded by
Air3Core's static initializer, which we bypass. Load it ourselves before
start() or the first sensor callback aborts with UnsatisfiedLinkError.
- tools:overrideLibrary: the AAR declares minSdk 34; override the manifest
merge so the app stays installable on the API-26 hub phone. The INMO classes
only load on the glasses, gated by InmoFusionTracker.isAvailable().
vQuat order [x,y,z,w] is correct as-is; axisFix stays identity. Added
tap-to-recenter (any key) to GlassesTestActivity to fix the static centering
offset left by auto-recenter-on-first-frame. AAR is gitignored (libs README
documents how to obtain it).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace decision-009's awkward "stand at the D435 and tap Align" with automatic fiducial detection: point the phone at an ArUco marker placed at the hub origin and tap Align. Validated on-device (Pixel 9a) — anchors land at correct hub positions, zero drift (ARCore 6DoF), and re-aligning from different viewpoints keeps them in place (confirms the pose math). - FiducialAligner: detect DICT_4X4_50 id0 in the ARCore camera image (acquireCameraImage + imageIntrinsics), solvePnP for tag-in-camera, flip OpenCV->ARCore camera axes (diag(1,-1,-1)), compose with camera.pose to get the tag pose in the ARCore world. - MainActivity: load OpenCV (initLocal); the Align action now does fiducial-first and falls back to phone-at-origin when no tag is in view. - Reuses FrameAlignment.align() unchanged (tag = hub origin, identity rot). - OpenCV via org.opencv:opencv:4.11.0, arm64 abiFilter. - Marker asset: fiducials/aruco_DICT4X4_50_id0.png (print black square 150mm). The tag<->hub orientation is assumed identity; add a constant rotation if anchors land rotated. Same FiducialAligner math is shared with the glasses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dedicated landscape/fullscreen Android app for the INMO Air3 (Track A) at the waveguide's native 1920x1080, separate from the phone app (:app). Reuses the validated INMO native head-tracking (decision 011) and shares the ArUco FiducialAligner math with the phone. Working / validated on-device (IMA301, Android 14): - INMO native fusion head tracking (GyroRotation.vQuat), tap-to-recenter. - Connects to the hub rosbridge, renders hub/phone anchors in GL, streams camera + IMU. HUD overlay; D-pad controls. In progress / not yet verified: - Fiducial alignment (ArucoAligner + OpenCV 4.11): built and detector runs, but on-device test was blocked by a lens cover (now resolved) — needs a re-test pointing at the marker. - Magnetometer yaw-drift correction: present but OFF by default (made yaw worse on the noisy desk); runtime toggle/sign-flip keys for tuning. Archive: launch_glasses_localizer.sh (dead-end cross-sensor relocalization experiment, see archive/README.md) and removed session debug artifacts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Move publish_random_anchors.py -> archive/ (unreferenced test tool; not used by any launch script). Documented in archive/README.md. - Kept index.html in place: it is the LIVE GitHub Pages homepage (https://isiazu.github.io/Pri4L/, served from main root) — not unused. - Kept decompress_image.py and pose_to_odom.py (used by launch_phone_mapper.sh). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two threads of AR-client work, sharing the same ArUco
FiducialAlignermath::app, Track B) — validated on-device. Replaces decision-009's awkward "stand at the D435 and tap Align" with automatic ArUco fiducial alignment: point the phone at a marker placed at the hub origin and tap Align. Anchors land at correct hub positions with zero drift (ARCore 6DoF), and re-aligning from different viewpoints keeps them in place.GyroRotation.vQuatfusion (resolves the decision-011 axis blocker), plus a new purpose-built:glassesmodule — a landscape/fullscreen waveguide client at native 1920×1080.Validated on-device
:glassesclient — connects to hub rosbridge, renders anchors in GL, streams camera + IMU, HUD.WIP / not yet verified
ArucoAligner+ OpenCV built and the detector runs, but the on-device test was blocked by a (since-removed) lens cover; needs a re-test pointing at the marker.ArPoseCore), future work.Key findings
air3_coreAAR outside Unity needs aUnityPlayerstub +System.loadLibrary("inmoair3")(decision 011).archive/.Notes
fiducials/aruco_DICT4X4_50_id0.png(print black square at 150 mm, place at the D435/hub origin).🤖 Generated with Claude Code