Skip to content

Add MediaPipe Hand & Gesture Detector block (JdeRobot/VisualCircuit#470)#17

Open
prawnsgupta wants to merge 2 commits into
JdeRobot:mainfrom
prawnsgupta:add-handgesture-block-v1.2.0
Open

Add MediaPipe Hand & Gesture Detector block (JdeRobot/VisualCircuit#470)#17
prawnsgupta wants to merge 2 commits into
JdeRobot:mainfrom
prawnsgupta:add-handgesture-block-v1.2.0

Conversation

@prawnsgupta

@prawnsgupta prawnsgupta commented Jul 17, 2026

Copy link
Copy Markdown

Implements the Hand & Gesture Detector block approved in JdeRobot/VisualCircuit#470 (thanks @BkPankaj for the go-ahead).

What it does: runs MediaPipe's Hand Landmarker on the input image, detects up to two hands (21 landmarks each), and estimates a simple gesture (extended-finger count). Fills the gap in VisualCircuit's CV blocks (FaceDetector/ObjectDetector exist, no hand/landmark block) and pairs with MotorDriver/Teleoperator for gesture-driven robotics.

  • Input: BGR image (Img)
  • Outputs (via OutputType): image (annotated frame with 21-point skeleton + gesture label) or landmarks (flat array of 21 (x,y) landmarks + finger count)
  • Parameters: OutputType (image/landmarks), MirroredInput (yes/no for pre-mirrored feeds)

Implementation notes:

  • MediaPipe Tasks API (mediapipe.tasks.vision.HandLandmarker) — the legacy mp.solutions.hands API was removed in 0.10.30+, so this is the durable choice.
  • Finger counting is rotation-invariant (distances from wrist, not raw x/y), verified on tilted/rotated hands.
  • Thumb uses a dedicated pinky-MCP reference (the thumb moves laterally, not radially), so a closed/tucked thumb is detected correctly.
  • Verified on real hand photos and a real thumbs-up image.

Needs a maintainer decision:

  • New runtime dependency mediapipe (declared in the block's dependencies).
  • Bundled model hand_landmarker.task (~7.6 MB) — should it live under utils/models/hand_landmarker/ alongside the Haar/YOLO models?

Demo video below.

Hand.Gesture.Demo.Video.mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant