GestureOS

Gesture-Based Human Computer Interaction

A modular gesture recognition platform that transforms hand movements into real computer interactions.

Control your computer using hand gestures powered by MediaPipe and YOLO.

---

Overview

GestureOS is an open-source Human-Computer Interaction (HCI) platform that allows users to control their computer using only hand gestures.

Using a standard webcam together with MediaPipe and YOLO, the system recognizes hand movements in real time and translates them into operating system actions.

GestureOS has been designed with a strong emphasis on:

• Clean Architecture
• Object-Oriented Design
• Modular Components
• Scalability
• Performance
• Maintainability
• Extensibility

Rather than being a simple virtual mouse, GestureOS is designed as a complete interaction platform capable of evolving through independent recognition engines and interaction modes.

---

Features

Virtual Mouse

• Virtual Cursor
• Left Click
• Right Click
• Double Click
• Drag & Drop
• Smooth Cursor
• Dead Zone
• Motion Acceleration
• Vibration Filter
• Scroll

---

Virtual Keyboard

• YOLO Letter Recognition
• Stable Detection Filter
• Automatic Typing
• Dynamic Recognition Pipeline

---

Core Architecture

• Modular Design
• Interaction Modes
• Recognition Manager
• Independent Recognition Engines
• Dynamic HUD
• Layered Architecture

---

Interaction Modes

GestureOS currently provides three interaction modes.

Mode	Purpose	Recognition
Cursor	Mouse interaction	MediaPipe
Keyboard	Letter recognition	YOLO
Command	Future automation	Planned

The interaction mode determines how gestures are interpreted without changing the gesture vocabulary.

---

Recognition Pipeline

Cursor Mode

Webcam
   │
   ▼
MediaPipe
   │
   ▼
GestureRecognizer
   │
   ▼
Cursor Mode
   │
   ▼
Mouse Actions

---

Keyboard Mode

Webcam
   │
   ▼
YOLO
   │
   ▼
LetterRecognizer
   │
   ▼
Stable Detection
   │
   ▼
Keyboard Controller

---

High-Level Architecture

              Webcam
                 │
                 ▼
         Camera Module
                 │
                 ▼
      Hand Tracking (MediaPipe)
                 │
                 ▼
      Recognition Manager
                 │
      ┌──────────┴──────────┐
      ▼                     ▼
 Gesture Engine        Letter Engine
(MediaPipe)               (YOLO)
      │                     │
      └──────────┬──────────┘
                 ▼
           Interaction Modes
                 │
                 ▼
      Windows Controllers
                 │
                 ▼
         Operating System

A detailed explanation of the architecture is available in:

docs/architecture.md

---

Project Structure

GestureOS/
│
├── app/
├── assets/
├── camera/
├── docs/
├── filters/
├── gestures/
├── modes/
├── recognition/
├── tracking/
├── ui/
├── windows/
│
├── config.py
├── main.py
└── requirements.txt

Every directory encapsulates a single subsystem of the application.

---

Installation

Clone the repository.

git clone <repository-url>

Create a virtual environment.

python -m venv .venv

Activate it.

Windows

.venv\Scripts\activate

Install the dependencies.

pip install -r requirements.txt

Run GestureOS.

python main.py

---

Documentation

The project documentation is available inside the docs/ directory.

Document	Description
architecture.md	Overall system architecture
gestures.md	Gesture recognition system
modes.md	Interaction modes
recognition.md	Recognition pipeline
development.md	Development guidelines
roadmap.md	Project roadmap

---

Roadmap

v0.1.0
Virtual Mouse
✅

↓

v1.0.0
Virtual Keyboard
✅

↓

v2.0.0
Complete Virtual Keyboard
📅

↓

v3.0.0
Command Mode
📅

↓

v4.0.0
Argentine Sign Language
📅

More details:

docs/roadmap.md

---

Technologies

• Python
• OpenCV
• MediaPipe
• YOLOv11
• PyAutoGUI

---

Project Philosophy

GestureOS is built around one simple idea:

Every subsystem should do one thing, and do it well.

The project follows a modular architecture where each component has a single responsibility.

This design allows new interaction modes, recognition engines and operating system features to be added without affecting existing functionality.

---

Current Status

Version

v1.0.0

Status

Stable Core

Current capabilities include:

• Virtual Mouse
• Virtual Keyboard
• Dynamic Interaction Modes
• MediaPipe Recognition
• YOLO Letter Recognition
• Modular Architecture
• Technical Documentation

---

License

This project is released under the MIT License.

---

GestureOS

Building the future of hands-free human-computer interaction.