🦯 BEACON — Camera AI Module

The AI-powered camera vision system for the BEACON assistive wearable.

BEACON is a smart cap designed to help visually impaired individuals navigate the world safely. This repository contains the Python-based AI and camera logic — one part of the larger BEACON ecosystem. Other repositories handle the backend API and the mobile app frontend.

---

The BEACON cap with an ESP32-CAM module and battery pack

---

What This Repo Does

This is the AI brain of BEACON. It captures a live video stream from the ESP32-CAM mounted on the cap, runs real-time inference using trained YOLO models, and communicates with the backend API to deliver audio alerts through the companion mobile app.

🔍 Object Detection & Proximity Alerts

Identifies common objects (people, vehicles, obstacles) using YOLOv11 and estimates their distance. When something is dangerously close, the system triggers an immediate alert so the user can react.

🚶 Crosswalk Detection

A custom-trained model (crosswalk.pt) recognizes crosswalks in the camera feed, helping the user know when it's safe to cross the road.

💵 Currency Recognition

Multiple iterations of a trained currency model (currency_new.pt) identify Indian currency notes (e.g. ₹100) and announce the denomination to the user — making everyday transactions more independent.

🧑‍🦯 Fall Detection

An optical-flow-based fall detector (fall_detector.py) monitors the camera feed for sudden violent motion followed by stillness — the signature pattern of a fall. If a fall is confirmed, an emergency alert is pushed to the backend and spoken aloud.

🔊 Voice Alerts via Backend

All detections are sent to the BEACON backend API (/message-queues/public), which routes them to the mobile app as spoken alerts. The system also uses local TTS as a fallback.

---

Architecture

┌──────────────┐       MJPEG stream        ┌──────────────────────┐
│  ESP32-CAM   │ ────────────────────────▶  │  This Repo (Python)  │
│  (on the cap)│                            │                      │
└──────────────┘                            │  • YOLO inference    │
                                            │  • Distance est.     │
                                            │  • Fall detection    │
                                            │  • Currency ID       │
                                            │  • Crosswalk ID      │
                                            └──────────┬───────────┘
                                                       │
                                                  POST /message-queues
                                                       │
                                                       ▼
                                            ┌──────────────────────┐
                                            │  BEACON Backend API  │
                                            │  (separate repo)     │
                                            └──────────┬───────────┘
                                                       │
                                                       ▼
                                            ┌──────────────────────┐
                                            │  BEACON Mobile App   │
                                            │  (speaks alerts)     │
                                            └──────────────────────┘

---

Trained Models

Model	File	Purpose
YOLOv11 Nano	yolo11n.pt	General object detection
YOLOv11 Nano Pose	yolo11n-pose.pt	Pose estimation for fall detection
Crosswalk Detector	crosswalk.pt	Custom-trained crosswalk recognition
Currency Detector	currency_new.pt	Indian currency note identification

ONNX-exported versions (yolo11n.onnx, crosswalk.onnx) are also available for edge deployment.

---

Tech Stack

Layer	Technology
Language	Python
Inference	Ultralytics (YOLOv11), PyTorch
Vision	OpenCV
Communication	Requests (REST API to BEACON backend)
TTS (local)	Windows Speech Synthesis (PowerShell)
Hardware	ESP32-CAM streaming over Wi-Fi (MJPEG)

---

Getting Started

Prerequisites

• Python 3.10+
• An ESP32-CAM on the same network, streaming MJPEG
• The BEACON Backend running locally or on a server

Installation

# Clone the repository
git clone https://github.com/your-org/BEACON-camera.git
cd BEACON-camera

# Create and activate a virtual environment
python -m venv venv
venv\Scripts\activate        # Windows
# source venv/bin/activate   # macOS / Linux

# Install dependencies
pip install -r requirements.txt

Configuration

Update the ESP32-CAM stream URL and backend API URL in the script you want to run:

STREAM_URL = "http://<ESP32_IP>:81/stream"
API_URL    = "http://127.0.0.1:8080/message-queues/public"

Run

# Latest version with all features (object + crosswalk + currency + fall detection)
python v12.py

Press Q or Ctrl+C to stop.

---

Project Structure

BEACON-camera/
├── v12.py               # Latest: full detection pipeline + fall detection
├── latest.py            # Stable: object + currency detection
├── fall_detector.py     # Optical-flow fall detection module
├── v1.py – v11.py       # Earlier iterations and experiments
├── requirements.txt     # Python dependencies
├── public/
│   └── cap.png          # Product image of the BEACON cap
├── yolo11n.pt           # Pre-trained YOLOv11 model
├── crosswalk.pt         # Custom crosswalk detection model
├── currency_new.pt      # Custom currency recognition model
├── yolo11n-pose.pt      # Pose estimation model
├── *.onnx               # ONNX exports for edge deployment
└── experiments/         # R&D and prototyping scripts

---

Part of the BEACON Ecosystem

This repository is one component of the full BEACON product. The complete system consists of:

Repository	Role
beacon-camera (this repo)	AI vision & camera processing (Python)
beacon-backend	REST API, message queues, user management
beacon-mobile	React Native mobile app (audio alerts, UI)

Together, they form an end-to-end assistive system: the cap sees the world, the AI understands it, the backend routes the information, and the mobile app speaks it to the user.

---

License

MIT