A two-machine mobile manipulator monorepo: ROS 2 Jazzy on Ubuntu 24.04, with a Raspberry Pi 5 as the robot target and a laptop as the dev / perception machine. The research core evaluates multi-ArUco fusion strategies (A0–A3) for docking alignment, in a Gazebo Harmonic simulation with a headless experiment + statistics pipeline; the hardware side is a tuned diff-drive base (ESP32 firmware) and a low-latency network camera pipeline.
┌─ laptop (dev PC, 10.42.0.1) ────────────┐ ┌─ RPi5 (robot, 10.42.0.126) ──────┐
│ camera rx + ArUco perception (fusion) │ │ camera tx (C920 MJPEG │
│ align / skill / mission control │◄──┤ passthrough, systemd) │
│ Gazebo sim + experiment pipeline │GbE│ base driver (mm_base, systemd) │
│ rviz / teleop / tuning tools ├──►│ │ serial /dev/esp32_base │
└─────────────────────────────────────────┘ │ ▼ │
ROS_DOMAIN_ID=42, FastDDS interface │ ESP32-S3: wheel PID + encoders │
whitelist (wired only — mandatory) └──────────────────────────────────┘
- The laptop's git checkout is the single source of truth; the Pi runs a
one-way rsync copy deployed by
tools/deploy_rpi5.sh(docs/deployment.md). - DDS traffic is pinned to the wired link by a FastDDS interface whitelist — without it the camera stream collapses from 30 Hz to ~4 Hz (docs/networking.md).
| Part | Role |
|---|---|
| Raspberry Pi 5 (Ubuntu 24.04, ros-base) | Robot computer: camera tx, base serial link |
| Adafruit ESP32-S3 TFT Feather | Wheel PID + feed-forward control, encoder odometry ticks, HUD |
| Logitech C920 | Robot camera, hardware MJPEG (720p30 passthrough — no Pi-side encode) |
| Diff-drive base (2 DC motors + encoders) | Mobile base, tuned via tools/base_tune.py |
| SO-ARM101-equivalent 6-joint arm | Simulation only for now (see ROADMAP.md) |
[RPi5] usb_cam ──/camera/image_raw/compressed──► [laptop] camera_rx ──/camera/image_raw──► aruco_detector
[RPi5] base_driver_node ──/odom──► [laptop] align/mission control ──/cmd_vel──► [RPi5] base_driver_node
└─ serial C/T frames ─ ESP32
aruco_detector ──per-site pose/error/status──► align_server (/align_to_marker action)
mission_fsm ──► /align_to_marker, /execute_skill actions, /mission/start service
In simulation the same perception/control nodes run against Gazebo through
ros_gz bridges — the sim/real split lives in the launch layer, not the nodes.
| Package | Role |
|---|---|
mm_msgs |
Interfaces: marker detections, fusion status, AlignToMarker / ExecuteSkill actions |
mm_description |
Xacro robot model (real base geometry, 6-joint arm, C920-like camera), all dimensions from YAML |
mm_base |
Diff-drive base driver: serial protocol to the ESP32, kinematics, odometry |
mm_camera |
C920 network camera pipeline (usb_cam passthrough + in-repo MJPEG fallback, rx side) |
mm_perception |
ArUco detection, degradation layer, fusion strategies A0–A3 with confidence lock |
mm_control |
Align action server (split-P), skill server (envelope model), mission FSM |
mm_sim |
Gazebo world/texture/bridge generators, shared sites.yaml |
mm_experiments |
Envelope sweep, headless batch runner, statistical analysis (Wilcoxon + Holm, bootstrap CI, Cliff's delta) |
mm_bringup |
Launch layer: sim / experiment_stack (sim) vs robot / base (real), perception / align / mission (shared, sim:= switch) |
Every launch file documents its arguments: ros2 launch <pkg> <file> --show-args.
source /opt/ros/jazzy/setup.bash
cd ros2_ws && colcon build --symlink-install && source install/setup.bash
ros2 launch mm_bringup sim.launch.py # full sim (add headless:=true, rviz:=true)
ros2 run mm_experiments batch_runner --config src/mm_experiments/config/experiments.yamlDetails, experiment workflow and every tunable config file: docs/experiments.md.
One-time Pi provisioning: docs/setup_rpi5.md. Full runbook: docs/operations.md.
# Laptop → Pi deploy (rsync + colcon build + service restart)
make deploy-restart # = tools/deploy_rpi5.sh --build --restart
# Pi: base + camera come up via systemd (or manually:)
ros2 launch mm_bringup robot.launch.py
# Laptop: EVERY shell needs the network env first
source tools/network/ros_net_env.sh
ros2 launch mm_bringup perception.launch.py sim:=false
# Verify: ~30 Hz on the wire
ros2 topic hz /camera/image_raw/compressedros2_ws/ ROS 2 Jazzy workspace (packages under src/mm_*)
firmware/esp32_base/ PlatformIO firmware for the ESP32-S3 base controller
tools/ deploy script, network env + FastDDS whitelists, udev,
systemd units, teleop & tuning utilities
docs/ setup, deployment, networking, operations,
troubleshooting, experiments (+ design/, notes/)
Makefile build / test / deploy / firmware shortcuts (make help)
.github/workflows/ CI: colcon build + test (ros:jazzy), PlatformIO build
Implemented and working: the full simulation/experiment pipeline (78 unit tests in CI), the tuned diff-drive base with ESP32 firmware, the 30 Hz laptop↔Pi camera pipeline with systemd + deploy tooling.
Not yet: real arm hardware, Nav2 navigation, camera calibration / odometry accuracy work, on-robot perception. The full list with rationale lives in ROADMAP.md.
make build # colcon build --symlink-install
make test # colcon test + result summary
make help # all targets, including deploy + firmwareCI (.github/workflows/ci.yml) runs the same
colcon build + test in a ros:jazzy container and a PlatformIO build of the
firmware on every push/PR. Gazebo behavior is validated by the smoke
procedure in docs/experiments.md, not CI.
make firmware # pio run
make firmware-upload # pio run -t uploadSerial protocol (ESP32 ↔ Pi/laptop):
- Pi → ESP32:
C <left_tps> <right_tps>,G <kp> <ki> <kff> <ilimit>,S - Laptop debug:
D 1enables USB debug idle mode,D 0returns toWAIT PI - Laptop tuning only:
P <left_pwm> <right_pwm>(watchdog-protected open-loop PWM) - ESP32 → Pi:
T <ms> <lticks> <rticks> <ltps> <rtps> <lpwm> <rpwm> <wd>
USB upload instability: firmware/esp32_base/TROUBLESHOOTING.md. Direct
serial testing: firmware/esp32_base/SERIAL_DEBUG_GUIDE.md.