✋ Air Gesture HMI

Real-Time Multimodal Gesture-Controlled Human–Computer Interaction System

A real-time gesture-based HCI system using MediaPipe hand tracking, trajectory analysis, and a hybrid Random Forest + Dynamic Time Warping recognition pipeline.

98.88% air-writing accuracy · 33.6 FPS · CPU-only execution · No specialised hardware required

🚀 Features

✍️ Real-time air-writing recognition (A–Z)
🖱️ Gesture-controlled virtual mouse
🎨 Transparent desktop drawing overlay
⌨️ Gesture-controlled on-screen keyboard
🎵 Virtual piano, guitar, and violin interaction
⚡ Real-time system shortcut gestures
🧠 Hybrid RF + DTW recognition pipeline
📦 CPU-only execution — no GPU required
🖨️ STL / OBJ 3D export from drawings
🔒 User handover / lock protection system

🎥 Demo

Draw mode

Write mode

Mouse mode

On Screen Keyboard mode

Gesture mode

Music mode

🧠 Project Overview

Air Gesture HMI is a touchless human-computer interaction system that translates mid-air hand gestures into real-time desktop interaction using computer vision and machine learning.

The system combines:

MediaPipe Hands for 21-point 3D hand landmark tracking
Rule-based gesture recognition for interaction control
Random Forest classification for air-writing recognition
Dynamic Time Warping (DTW) for trajectory similarity matching
Real-time desktop interaction using PyAutoGUI, Pynput, and Win32 APIs

The application runs entirely on CPU using a standard webcam.

📊 Performance

Metric	Value
Letter recognition accuracy	98.88%
Letter classes	26 (A–Z)
Random Forest estimators	200
Feature count	13
Average FPS	33.6 FPS
Average latency	29.7 ms
Runtime hardware	CPU only
GPU required	❌ No

Benchmark Environment

Windows 11 (64-bit)
Python 3.10
Standard USB webcam
CPU-only execution

🏗️ System Architecture

Webcam Feed
     ↓
MediaPipe Hand Tracking
     ↓
21 Landmark Extraction
     ↓
Gesture / Trajectory Analysis
     ↓
RF + DTW Recognition Pipeline
     ↓
Mode Router
     ↓
Desktop Interaction / Audio / 3D Export

⚡ Quick Start

1. Clone Repository

git clone https://github.com/Sidharths916/air-gesture-hmi.git
cd air-gesture-hmi

2. Create Virtual Environment

py -3.10 -m venv venv
venv\Scripts\activate

3. Install Dependencies

pip install --upgrade pip setuptools wheel
pip install -r requirements.txt

4. Extract Trained Model

Extract:

models/letter_recognizer.zip

inside the models/ directory.

Final structure:

models/
└── letter_recognizer.pkl

5. Run

python main.py

🧰 Tech Stack

Category	Technologies
Computer Vision	OpenCV, MediaPipe
Machine Learning	Scikit-learn, Random Forest, DTW
Interaction Layer	PyAutoGUI, Pynput, PyWin32
Audio	pygame, scipy
Rendering	OpenCV overlays, Win32 transparency
Language	Python 3.10

📁 Project Structure

air-gesture-hmi/
│
├── main.py
├── piano_FINALv4.py
├── guitar_FINALv4.py
├── violin_REFACTOREDv2.py
│
├── assets/
├── data/
├── models/
│   └── letter_recognizer.zip
│
├── config.json
├── requirements.txt
└── README.md

🎮 Modes

✏️ Draw Mode

Transparent desktop drawing overlay featuring:

fingertip drawing
colour switching
eraser support
STL / OBJ export
anaglyph rendering

Export Support

PNG
STL
OBJ + MTL + texture PNG
Anaglyph 3D image

Gestures

Gesture	Action
☝️ 1 finger	Draw with current colour
✌️ 2 fingers	Cycle to next colour
🤟 3 fingers	Save drawing
4 fingers	Clear canvas
🤙 Pinky	Toggle eraser
🖐️ Palm	Return to main menu

✍️ Write Mode

Real-time air-writing recognition system.

Recognition Pipeline

Trajectory capture
Stroke normalisation
Feature extraction
Random Forest classification
DTW template matching
Score fusion
Keyboard output

Recognition Features

13 handcrafted geometric features
64-point trajectory resampling
confidence threshold filtering
DTW similarity comparison

Gestures

Gesture	Action
☝️ 1 finger	Draw character stroke
Hold still	Recognise and type character
✌️ 2 fingers	Space
🤟 3 fingers	Backspace
4 fingers	Clear stroke
🖐️ Palm	Return to main menu

🖱️ Mouse Mode

Gesture-controlled virtual mouse system.

Supported Actions

cursor movement
left click
right click
drag / hold
scroll up/down

Gestures

Gesture	Action
☝️ Index only	Move cursor
💍 Ring only	Left click
🤙 Pinky only	Right click
✊ Fist	Toggle drag
✌️ 2 fingers	Scroll up
🤟 3 fingers	Scroll down
🖐️ Palm	Return to main menu

⌨️ On-Screen Keyboard

Transparent gesture-controlled virtual keyboard.

Features

fullscreen overlay
hover interaction
Z-depth press detection
static & dynamic modes
modifier key support

Gestures

Gesture	Action
Hover + push forward	Press key
Hover over MODE	Toggle input mode
🖐️ Palm	Return to main menu

🎵 Music Mode

Gesture-controlled virtual instruments.

Included Instruments

🎹 Piano
🎸 Guitar
🎻 Violin

Audio Features

synthesized tones
harmonic layering
ADSR shaping
chord detection

Music Menu Gestures

Gesture	Action
☝️ 1 finger	Piano
✌️ 2 fingers	Guitar
🤟 3 fingers	Violin
🖐️ Palm	Return to main menu

Instrument Selection

Gesture	Action
👍	Free play
👌	Song selection
🖐️ Palm	Cancel

🎮 Gesture Shortcut Mode

Maps gestures directly to system shortcuts.

Examples

Gesture	Action
👍	Screenshot
👌	Save file
✌️	Alt + Tab
🤟	Zoom in
🤙	Browser back

🔒 Handover Security System

Basic user handover detection system using torso colour verification.

Features

user verification
auto-lock on mismatch
auto-unlock on return
manual recalibration

Controls

Action	Trigger
Auto-lock	Different user detected
Auto-unlock	Original user returns
Manual recalibration	Hold ✊ Fist
Force recalibration	Press U key

🧠 Air-Writing ML Pipeline

Stage 1 — Hand Tracking

MediaPipe Hands extracts 21 3D landmarks per frame.

Stage 2 — Feature Extraction

Each trajectory is:

centroid normalised
unit scaled
resampled to 64 points

13 geometric features are extracted including:

aspect ratio
path length
curvature
spatial density
start/end coordinates

Stage 3 — Random Forest

A 200-tree Random Forest predicts class probabilities.

Stage 4 — DTW Matching

Dynamic Time Warping compares trajectory similarity against stored templates.

Stage 5 — Score Fusion

RF probability and DTW similarity scores are fused for final prediction.

Final accuracy achieved: 98.88% across all 26 letters (A–Z)

⚙️ Requirements

Software

Windows 10 / 11
Python 3.10

Hardware

Standard webcam
CPU-only execution

📦 Major Dependencies

Library	Purpose
OpenCV	Computer vision & rendering
MediaPipe	Hand landmark tracking
Scikit-learn	Random Forest classifier
pygame	Audio engine
scipy	Signal filtering
pyautogui	Mouse & keyboard control
pynput	System input simulation
pywin32	Transparent overlays

mediapipe==0.10.14 and scikit-learn==1.3.2 are intentionally pinned for compatibility.

⌨️ Runtime Keyboard Shortcuts

Key	Action
ESC	Exit application
U	Recalibrate user
R	Reset camera
H	Toggle skeleton overlay
M	Toggle mirror mode
S	Toggle sound
F	Toggle FPS display

⚠️ Known Issues

Initial startup latency during MediaPipe warm-up
Transparent overlays are Windows-dependent
First camera frame may briefly spike in latency

🔮 Future Work

Multi-hand support
Linux/macOS support
CNN-based recognition comparison
Mobile deployment
Custom gesture recording
Embedded deployment

📚 Citation

@misc{airgesturehmi2026,
  author    = {Sidharth S},
  title     = {Air Gesture HMI: Real-Time Multimodal Gesture-Controlled Human-Computer Interaction System},
  year      = {2026},
  publisher = {GitHub},
  url       = {https://github.com/Sidharths916/air-gesture-hmi}
}

📄 License

Licensed under the MIT License.

🙏 Acknowledgements

MediaPipe by Google
OpenCV
pygame
scikit-learn

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
assets		assets
data		data
models		models
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
config.json		config.json
guitar_FINALv4.py		guitar_FINALv4.py
icon.ico		icon.ico
main.py		main.py
piano_FINALv4.py		piano_FINALv4.py
requirements.txt		requirements.txt
violin_REFACTOREDv2.py		violin_REFACTOREDv2.py

Folders and files

Latest commit

History

Repository files navigation

✋ Air Gesture HMI

Real-Time Multimodal Gesture-Controlled Human–Computer Interaction System

🚀 Features

🎥 Demo

Draw mode

Write mode

Mouse mode

On Screen Keyboard mode

Gesture mode

Music mode

🧠 Project Overview

📊 Performance

Benchmark Environment

🏗️ System Architecture

⚡ Quick Start

1. Clone Repository

2. Create Virtual Environment

3. Install Dependencies

4. Extract Trained Model

5. Run

🧰 Tech Stack

📁 Project Structure

🎮 Modes

✏️ Draw Mode

Export Support

Gestures

✍️ Write Mode

Recognition Pipeline

Recognition Features

Gestures

🖱️ Mouse Mode

Supported Actions

Gestures

⌨️ On-Screen Keyboard

Features

Gestures

🎵 Music Mode

Included Instruments

Audio Features

Music Menu Gestures

Instrument Selection

🎮 Gesture Shortcut Mode

Examples

🔒 Handover Security System

Features

Controls

🧠 Air-Writing ML Pipeline

Stage 1 — Hand Tracking

Stage 2 — Feature Extraction

Stage 3 — Random Forest

Stage 4 — DTW Matching

Stage 5 — Score Fusion

⚙️ Requirements

Software

Hardware

📦 Major Dependencies

⌨️ Runtime Keyboard Shortcuts

⚠️ Known Issues

🔮 Future Work

📚 Citation

📄 License

🙏 Acknowledgements

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages