A webcam, two hands, and a real mix session. No mouse.
Can a mixing engineer actually mix a song using only their hands? This is a macOS app that turns a webcam into a mouse — open palm to engage, pinch to click, fist to bail out — so plugin parameters and transport in any DAW can be driven by gesture. It exists because I wanted to find out if it holds up on a real session, on camera, with no safety net. v1 is plugin params and transport only; the experiment, the failures, and the verdict are the point.
- Turns webcam input into system cursor control using MediaPipe hand tracking.
- Open palm engages tracking, pinch (thumb + index) clicks and drags, closed fist releases.
- Drives any DAW the way a mouse does: plugin parameters, transport buttons, anything clickable.
- Includes a small status overlay that floats over all apps, showing IDLE or ENGAGED state.
- macOS only. Mouse emulation, not MIDI.
- macOS 13 (Ventura) or later.
- Apple Silicon recommended. Intel Macs may run but MediaPipe wheels and frame rate are not guaranteed.
- Built-in or USB webcam.
- Python 3.10 or 3.11 if building from source. MediaPipe wheels lag behind the newest Python releases.
- Camera and Accessibility permissions granted to whichever binary you run (the
.appand a Python interpreter are treated as separate apps by macOS).
git clone https://github.com/giovicordova/hand-tracking.git
cd hand-tracking
python -m venv .venv
source .venv/bin/activate
pip install -e .
python -m hand_tracking_gcThe Python interpreter inside .venv is a separate binary from the .app, so it needs its own Camera and Accessibility grants in System Settings → Privacy & Security.
pip install py2app
python setup.py py2appOutput lands at dist/Hand Tracking GC.app. setup.py contains a macholib monkey-patch as a workaround for a current py2app bundling issue — leave it in place.
A pre-built .app may be attached to future GitHub releases once the build is signed and notarised. For now, build from source.
- Double-click
Hand Tracking GC.app, or - From terminal:
.venv/bin/python -m hand_tracking_gc, or - As a console script after
pip install -e .:hand-tracking-gc
On launch the app starts in IDLE. Show an open palm to engage. Press o to toggle the overlay. Press q to quit.
| Gesture | What it does | When |
|---|---|---|
| Open palm (held ~0.5s) | Engages tracking | From IDLE |
| Hand movement | Moves the cursor | While ENGAGED |
| Pinch (thumb + index) | Mouse down — click and drag | While ENGAGED |
| Release pinch | Mouse up | While pinched |
| Closed fist | Disengages and drops any held drag (emergency release) | Any time |
| Hand out of frame | Disengages after ~165ms | Any time |
Full v1 spec in BRIEF.md. v1 deliberately ships without:
- Channel faders. Plugin parameters and transport only.
- MIDI output. Mouse emulation only.
- Windows or Linux support.
- Multi-hand tracking. One hand at a time.
- Smoothing presets or a precision mode. Tuning is by editing constants (see Troubleshooting).
- A gesture customisation UI. Gesture vocabulary is fixed in v1.
Cursor doesn't move. Accessibility permission is missing or granted to the wrong binary. The .app and a .venv Python are separate apps to macOS — each needs its own grant in System Settings → Privacy & Security → Accessibility.
Camera is black, or the app exits with a permission error. Same idea for Camera permission. Grant it to the binary you actually launched.
Cursor jitters. Lower MIN_CUTOFF in src/hand_tracking_gc/cursor/smoothing.py. Lower is smoother but laggier.
Cursor too slow or too fast. Tune SENSITIVITY in src/hand_tracking_gc/cursor/mapping.py.
Pinch flickers, or engages by accident. Tune PINCH_ENGAGE and PINCH_RELEASE in src/hand_tracking_gc/gestures/pinch.py. Wider gap between the two = less flicker.
Engagement triggers accidentally. Raise ENGAGE_FRAMES in src/hand_tracking_gc/gestures/engagement.py so the open palm has to be held longer.
"Continuity Camera is deprecated" warning in the terminal. Cosmetic. Ignore.
"App is damaged and can't be opened" or Gatekeeper blocks the app. This build is unsigned. Right-click the app and choose Open, or go to System Settings → Privacy & Security and click Open Anyway.
The Accessibility toggle is ON but the app still says it can't move the cursor. Every rebuild of the unsigned .app produces a new binary signature, which invalidates the previous Accessibility grant — even though the toggle still appears ON, macOS is checking against a stale record. Reset the grant for the app's bundle ID and re-add it:
tccutil reset All com.giovicordova.hand-tracking-gcThen in System Settings → Privacy & Security → Accessibility: remove Hand Tracking GC if still listed, click + to re-add dist/Hand Tracking GC.app, toggle it on. This goes away once the app is code-signed (post v1).
MIT.