Clarity

Clarity is a mobile app designed for visually impaired users, with two modes: near-sight text detection and far-sight text detection.

Near mode: reads close-up labels and small print, and reads it back aloud. Tap any line to hear just that line.

Far mode: reads distant signs. The text is too small for OCR to read from a full photo, so a fine-tuned YOLOv11 model locates the text regions first, then each is cropped and read.

_{Toggle, Near mode (label reading), Far mode (distant text)}

Model Training

EasyOCR's built-in detector misses small, distant text. To fix that, I fine-tuned YOLOv11 on the ICDAR scene-text dataset (~1,200 images) to detect text regions. Trained on a Kaggle GPU.

Stack

React Native (Expo), Python, FastAPI, PyTorch, Ultralytics YOLOv11, EasyOCR, OpenCV.

The phone captures images and plays audio. A FastAPI backend runs the models and returns JSON.

Far mode runs like this: YOLOv11 detects the text boxes, boxes on the same line get merged, each one is cropped, EasyOCR reads it, and the results are sorted into reading order.

Run it

# backend
cd backend
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8000

# mobile
cd mobile && npm install && npx expo start

Set BACKEND_URL in mobile/App.js to your machine's IP, then scan the QR code in Expo Go.