- Raspberry Pi Real-Time Object Detection and Tracking
- 1. Introduction
- 2. Dependency
- 3. What's in this repository
- 3.1. Camera Test
- 3.2. Motion Detection
- 3.3. Color-based Object Detection and Tracking
- 3.4. Shape-based Object Detection and Tracking
- 3.5. Feature-based Object Detection and Tracking (with ORB)
- 3.6. Face Detection and Tracking
- 3.7. Object Detection using YOLO
- 3.8. Object Detection using TensorFlow Lite
- 4. How to Run
- 5. Q&A
- License
Table of contents generated with markdown-toc
Using a Raspberry Pi and a camera module for computer vision with OpenCV, YOLO, and TensorFlow Lite. The aim of this project is to provide a starting point for using RPi & CV in your own DIY / maker projects. Computer vision based on cameras is very powerful and will bring your project to the next level. This allows you to track complicated objects that would otherwise not be possible with other types of sensors (infrared, ultrasonic, LiDAR, etc).
Note the code is based on Python and OpenCV meaning it is cross-platform. You can run this on other Linux-based platforms as well, e.g. x86/x64 PC, IPC, Jetson, Banana Pi, LattaPanda, BeagleBoard, etc.
This project is dependent on the following packages:
- Python >= 3.6.9
- OpenCV-Python
- OpenCV-Contrib-Python
- NumPy
- SciPy
- Matplotlib
- Ultralytics
- Support Raspberry Pi 1 Model B, Raspberry Pi 2, Raspberry Pi Zero, and Raspberry Pi 3/4/5 (preferable)
- Different boards will have very varied performances.
- RPi 3/4/5 are preferable as they have more powerful CPUs;
- RPi 1/2 may be struggling and produce very low FPS, in which case you can further reduce the camera resolution (160 x 120).
- Nvidia Jetson
- Jetson Nano (A01) also passed the test.
- Jetson Orin Nano should work as well.
- Jetson has its own repo now: jetson-object-detection
- Any USB camera supported by Raspberry Pi
- To see a list of all supportive cameras, visit http://elinux.org/RPi_USB_Webcams
- The official RPi camera module is supported through
Picamera2.
Currently, the following applications are implemented:
src/camera-test: Test if the camera is workingsrc/motion-detection: Detect any motion in the framesrc/object-tracking-color: Object detection & tracking based on colorsrc/object-tracking-shape: Object detection & tracking based on shapesrc/object-tracking-feature: Object detection & tracking based on features using ORBsrc/face-detection: Face detection & trackingsrc/object-detection-yolo: Object detection using YOLO (RPi 3/4/5 only)src/object-detection-tflite: Object detection using TensorFlow Lite (RPi 3/4/5 only)
Test the RPi and OpenCV environment. You are expected to see a pop-up window that has video streams from your USB camera if everything is set up correctly. If the window does not appear, you need to check both of (1) your environment; (2) camera connection.
Detect object movements in the camera frame and print a warning message when motion is sustained.
Pipeline: the current and previous frames are converted to grayscale, Gaussian-blurred to suppress sensor noise, then compared with a per-pixel cv2.absdiff. The resulting diff is thresholded, dilated, and passed through cv2.findContours; only connected regions with area ≥ MIN_MOTION_AREA count as motion, and a debounce (MOTION_CONSECUTIVE_FRAMES) rejects one-frame transients. All thresholds are constants at the top of the script, so you can trade sensitivity for false-positive rate without touching the loop.
Track an object based on its color in HSV and print its center position. You can choose your own color by clicking on the object of interest. Click multiple times on different points so a full color space is covered. You can hard code the parameters so you don't need to pick them again for the next run. The following demo shows how I track a Nintendo game controller in real-time:
Detect and track geometric shapes in the frame:
- Circles via
cv2.HoughCircles() - Triangles and rectangles via contour approximation (
cv2.findContours()+cv2.approxPolyDP())
Contours below MIN_CONTOUR_AREA pixels are filtered out to reduce noise; the epsilon used for polygon approximation and the Canny edge thresholds are configurable at the top of the script.
Detect and track an object using its feature. The algorithm I selected here is ORB (Oriented FAST and Rotated BRIEF) for its fast calculation speed to enable real-time detection. To use the example, please prepare an Arduino UNO board in hand (or replace the simple.png).
Detecting face using Harr Cascade detector.
Use YOLO (You Only Look Once) for object detection.
Note this code is based on Ultralytics YOLO. The instruction can be found at their website: Quick Start Guide: Raspberry Pi with Ultralytics YOLO11. Double check if you need to use it in a commercialised project!
Use TensorFlow Lite to recognise objects.
To use it, download the model and labels into the module directory:
cd src/object-detection-tflite
# COCO labels
wget https://raw.githubusercontent.com/google-coral/test_data/master/coco_labels.txt
# EfficientDet-Lite0 model (INT8 quantized, ~4MB)
wget https://storage.googleapis.com/mediapipemodels/object_detector/efficientdet_lite0/int8/1/efficientdet_lite0.tflite
Then install the TensorFlow Lite runtime and run:
pip install tflite-runtime
python3 src/object-detection-tflite/object_detection_tflite.py
Run the installation script. It creates a virtual environment under ./venv and installs all Python dependencies from requirements.txt:
chmod +x install.sh
./install.sh
On Raspberry Pi, the venv is created with --system-site-packages so that libcamera and picamera2 (installed in the system Python by Raspberry Pi OS) remain accessible from inside it.
Activate the environment in your shell before running any of the scripts:
source venv/bin/activate
The TensorFlow Lite example (§3.8) has an extra optional dependency — install it into the same venv only if you plan to use it:
pip install tflite-runtime
Run scripts in the /src folder by:
python3 src/$FOLDER_NAME$/$SCRIPT_NAME$.py
To stop the code, press the ESC key on your keyboard.
Changing the resolution will significantly impact the FPS. By default it is set to be 320 x 240, but you can change it to any value that your camera supports at the beginning of each source code (defined by IMAGE_WIDTH and IMAGE_HEIGHT). Typical resolutions are:
- 160 x 120
- 320 x 240
- 640 x 480 (480p)
- 1280 x 720 (720p)
- 1920 x 1080 (1080p: make sure your camera supports this high resolution.)
Q: Does this support Nvidia Jetson?
A: Yes. I have tested with my Jetson Nano 4GB. Note that Jetson has its own repo now: jetson-object-detection.
Q: Does this support the Raspberry Pi camera?
A: This is implemented in issue [#16].
Q: Does this support Raspberry Pi 5?
A: This is not officially tested (as I haven't received my Pi 5 yet) but it should work out of the box.
Q: Can we run this project on Ubuntu server 22.04/24.04?
A4: It is not officially tested but you should be able to run 99% of the things here.
Q: I am using virtual environment and get a message "no module called libcamera" issue
A: A simple solution would be (Reference):
python3 -m venv --system-site-packages env
Thanks VgerTest for issue [#20].
© This source code is licensed under the MIT License.