Skip to content

rafiul254/AirControl-Pro

BranchesTags

Repository files navigation


Python OpenCV MediaPipe scikit-learn License Platform


Control your entire PC with just your hands. No mouse. No keyboard. Just gestures, voice, and AI.



📌 Table of Contents

🧠 About

AirControl Pro is a real-time, AI-powered desktop control system that lets you interact with your computer entirely through hand gestures, voice commands, and facial recognition — no physical input device required.

Built from scratch using Python and computer vision, it goes beyond basic gesture detection by layering multiple AI systems together:

  • A face authentication gate that locks the system to the registered user
  • A multi-mode gesture engine powered by MediaPipe's 21-point hand landmark model
  • A trainable custom gesture AI using RandomForest that learns your own gestures
  • A live emotion detector that adapts the UI theme based on your expression
  • A background voice engine that accepts spoken commands alongside gestures

This project was designed and built as a real-world application of computer vision, machine learning, and human-computer interaction concepts.

✨ Features

🔐 Face Authentication

  • LBPH-based face recognizer (OpenCV)
  • Locks system to registered user only
  • Runs fully offline — no cloud, no API
  • Graceful fallback when no face is registered

😊 Emotion-Aware UI

  • Detects 5 facial expressions in real time
  • Dynamically changes HUD color theme
  • Smile + eye detection for accuracy
  • History-based smoothing to avoid flicker

🤖 Custom Gesture AI

  • Record your own gestures (50 samples each)
  • Trains a RandomForest classifier on your data
  • Model persists across sessions (.pkl file)
  • Live prediction confidence shown on HUD

🎤 Voice Command Engine

  • Background speech recognition (no hotword needed)
  • Controls modes, apps, and canvas via voice
  • Runs in a separate thread — zero lag on main loop
  • Easily extendable with new commands

🖐️ Gesture Modes

Mode Switching

Gesture Mode Activated
✌️ Peace sign (2 fingers) 🎵 Media Mode
🤙 Hang Loose 🖥️ System Mode
🖖 3 Fingers 🎨 Draw Mode
🤙 Hang Loose (inside Draw) ⬅️ Exit Draw Mode

🎵 Media Mode

Gesture Action
👌 Thumb + Index pinch Volume control (distance-based)
✋ Open Palm Play / Pause
✊ Fist Previous Track
🤙 Pinky only Next Track

🖥️ System Mode

Gesture Action
✋ Open Palm Take Screenshot
👌 Thumb + Index pinch Brightness control
✊ Fist Alt + Tab
🖖 3 Fingers Show Desktop

🎨 Draw Mode

Gesture Action
☝️ 1 Finger Draw on canvas
✊ Fist Erase
✋ Open Palm Clear entire canvas

😊 Emotion Themes

Emotion HUD Color Detection Logic
Happy 😊 Green Smile + eyes open
Neutral 😐 Grey No smile, normal eyes
Focused 🎯 Blue Both eyes open, no smile
Surprised 😮 Cyan Wide eyes detected
Tired 😴 Purple Eyes closed / drooping

🎤 Voice Commands

Say Action
"media" Switch to Media Mode
"system" Switch to System Mode
"draw" Switch to Draw Mode
"youtube" Open YouTube in browser
"screenshot" Capture screenshot
"clear" Clear drawing canvas

🏗️ System Architecture

┌─────────────────────────────────────────────────────┐
│                    Camera Input                      │
└───────────────────────┬─────────────────────────────┘
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
   ┌─────────────┐ ┌──────────┐ ┌──────────────┐
   │ Face Engine │ │ Emotion  │ │ Voice Engine │
   │  (LBPH)     │ │ Detector │ │  (Thread)    │
   └──────┬──────┘ └────┬─────┘ └──────┬───────┘
          │             │              │
          └─────────────┼──────────────┘
                        ▼
            ┌───────────────────────┐
            │     Hand Tracker      │
            │  (MediaPipe 21-pt)    │
            └───────────┬───────────┘
                        ▼
            ┌───────────────────────┐
            │   Gesture Classifier  │
            │  Rule-based + AI (RF) │
            └───────────┬───────────┘
                        ▼
            ┌───────────────────────┐
            │      Mode Engine      │
            │  Media│System│Draw    │
            └───────────┬───────────┘
                        ▼
            ┌───────────────────────┐
            │    HUD Overlay        │
            │  Emotion-themed UI    │
            └───────────────────────┘

🛠️ Tech Stack

Technology Version Purpose
Python 3.11 Core language
OpenCV 4.10 Camera, image processing, face auth, emotion
MediaPipe 0.10 21-point hand landmark detection
scikit-learn 1.3 Custom gesture RandomForest classifier
PyAutoGUI latest Keyboard & mouse automation
pycaw latest Windows audio/volume control
SpeechRecognition latest Voice command processing
NumPy 2.x Numerical computations
joblib latest Model persistence

📁 Project Structure

AirControl-Pro/
│
├── main.py                        # Entry point, main loop
├── gesture_detector.py            # MediaPipe hand tracking
├── gesture_classifier.py          # Rule-based gesture logic
├── config.py                      # All settings & constants
│
├── ai/
│   ├── __init__.py
│   ├── custom_gesture_ai.py       # Trainable RandomForest AI
│   └── emotion_detector.py        # Facial emotion detection
│
├── auth/
│   ├── __init__.py
│   └── face_auth.py               # LBPH face authentication
│
├── voice/
│   ├── __init__.py
│   └── voice_engine.py            # Threaded speech recognition
│
├── modes/
│   ├── __init__.py
│   ├── media_mode.py              # Volume, play/pause, tracks
│   ├── system_mode.py             # Screenshot, brightness, apps
│   └── draw_mode.py               # Air drawing canvas
│
├── hud/
│   ├── __init__.py
│   └── overlay.py                 # Emotion-themed HUD display
│
├── utils/
│   ├── __init__.py
│   ├── finger_counter.py          # Per-finger state detection
│   └── velocity_tracker.py        # Gesture speed tracking
│
└── data/
    ├── known_faces/             
    │   └── README.txt
    ├── gesture_samples.json       # Recorded training data (auto)
    └── gesture_model.pkl          # Trained model (auto)

🚀 Getting Started

Prerequisites

  • Windows 10 or 11
  • Python 3.11
  • Webcam (built-in or external)
  • Microphone (for voice commands)

Installation

1. Clone the repository

git clone https://github.com/rafiul254/AirControl-Pro.git
cd AirControl-Pro

2. Create and activate virtual environment

python -m venv venv
.\venv\Scripts\Activate.ps1

3. Install all dependencies

pip install numpy
pip install opencv-contrib-python --only-binary=:all:
pip install mediapipe --only-binary=:all:
pip install protobuf --only-binary=:all:
pip install scikit-learn joblib
pip install pyautogui
pip install pycaw comtypes
pip install screen-brightness-control
pip install SpeechRecognition pyaudio

4. Register your face

Add a clear, front-facing photo of yourself:

data/known_faces/YourName.jpg

5. Run

python main.py

🎮 How to Use

Keyboard Shortcuts

Key Action
Q Quit
F Toggle fullscreen
R Start recording a custom gesture
T Train the custom gesture AI

Tips for Best Results

✅  Stay 30–60 cm from the camera
✅  Keep your hand fully visible in frame
✅  Use in a well-lit environment
✅  Keep hand to the side — not in front of your face
✅  Hold gestures steady for ~1 second to trigger

🤖 Training Custom Gestures

You can teach AirControl Pro any hand sign you want:

Step 1 — Run the app: python main.py
Step 2 — Press R
Step 3 — Type a name for your gesture (e.g. "thumbs_up")
Step 4 — Hold the gesture in front of the camera
         (50 samples will be captured automatically)
Step 5 — Repeat for at least 2 different gestures
Step 6 — Press T to train the AI
Step 7 — Your model saves to data/gesture_model.pkl
         and auto-loads next time you run the app

⚙️ Configuration

Edit config.py to customize behavior:

CAMERA_ID             = 0      # Change if using external camera
FRAME_WIDTH           = 640
FRAME_HEIGHT          = 480
DETECTION_CONFIDENCE  = 0.5    # Lower = detects more, less accurate
GESTURE_HOLD_TIME     = 0.8    # Seconds to confirm a gesture
MODE_SWITCH_COOLDOWN  = 1.5    # Seconds between mode switches
MAX_HANDS             = 1

🗺️ Roadmap

  • [complete] Multi-mode gesture control (Media, System, Draw)
  • [complete] Face authentication with LBPH
  • [complete] Emotion-aware adaptive UI
  • [complete] Custom gesture AI (RandomForest)
  • [complete] Threaded voice command engine
  • [complete] Live HUD with FPS, mode, emotion display
  • [In future] Browser control mode (scroll, tab switch)
  • [In future] Virtual air keyboard
  • [In future] IoT integration via MQTT (GestureBot)
  • [In future] Multi-user face profile support
  • [In future] Gesture macro recorder

👨‍💻 Author

MD. Rafiul Islam

IoT & Robotics Engineering | University of Frontier Technology, Bangladesh


GitHub LinkedIn

📄 License

This project is licensed under the MIT License — see LICENSE for details.

If this project helped or inspired you, consider leaving a ⭐ star in this repo

About

AI-Powered Hand Gesture Controller with Face Authentication & Emotion Detection

Topics

ai-poweredaircontrol-proauthenticationcomputer-visionemotion-detectionfacial-recognitiongesture-controllerhand-gesturesmediapipemode-basednumpyopencvpyautoguipythonscikit-learnvoice-commands

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Contributors

Languages