---

A real synthesizer, playable in real time, controlled with your hands over webcam — right in the browser. No install, no MIDI controller, no hardware.

Right hand plays the melody, left hand shapes the sound. Pitch is continuous and theremin-style — gliding the hand sweeps smoothly through every frequency in between two notes, with no audible steps — while the current scale's notes stay marked on-screen, on the grid itself, as exact targets to land on.

What makes this different

Most webcam hand-tracking demos either quantize pitch so hard that motion feels like flipping through a slideshow of notes, or leave pitch unconstrained and unplayable. Air-Synth does neither:

	No quantized jumps — gliding sweeps through every real frequency between two in-tune target notes, exactly like a theremin, instead of stepping between them.
	Horizontal lines sit exactly at the on-screen height each scale note sounds at, labeled on both edges, root picked out in orange — visible before you ever raise a hand, not something to hunt for by ear.
	Superwave, Karplus-Strong (plucked string), Wavetable, Pad, Analog — two run on hand-written AudioWorkletProcessors, not just built-in oscillators.
	Right hand: melody, pinch-to-play, pan. Left hand: filter cutoff, resonance, mute.
	Sustain, reverb, delay, chorus — four separate sliders, not one blended "space" knob standing in for all of them.
	Works on desktop and phone, with a controls panel that collapses to a thin bar when it's in the way.
	Load an audio file to play along with; it's mixed into the same audio graph the synth uses, so Record captures both together — external tab/app audio can't be captured without an intrusive screen-share permission, so this is the reliable way to record a performance over one.
	Captures video + hand skeleton overlay + audio together via MediaRecorder, downloadable as a single clip.

Note

The theremin comparison above it's literally where the idea came from. The pitch was originally hard-quantized with no glide at all; my musician friends (the same ones who caught the engine bugs below) suggested it should behave like a theremin instead, and they were right. It's a much better instrument for it.

How to play

	Vertical	Horizontal	Pinch (thumb + index)	Fist
Right hand	Pitch — continuous glide across one octave of the scale	Pan	Play the note; how tight sets its volume	—
Left hand	Filter cutoff (up = brighter)	Filter resonance	—	Mute everything

There's no snapping or stepping on pitch: moving between two notes glides through everything in between rather than jumping. The scale's notes are still exactly in tune at their marked position on the grid — hitting them precisely is up to your aim, the same way it would be on a real theremin.

Sustain, reverb, delay and chorus live as sliders in the controls panel, not gestures — easier to dial in precisely than a hand-tracked mix. An onboarding card explains all of this on first load; dismiss it and start playing, or reopen it any time with the ? button.

Want to play over a backing track? Load an audio file in the controls panel — it loops automatically, has its own volume slider, and (unlike your system or browser tab's other audio) is mixed directly into the app's own audio graph, so hitting REC captures it together with your performance.

Tip

Landing precisely on a target note is a real skill, the same way it is on an actual theremin — the grid tells you where to aim, but hitting it takes a bit of practice. Play with headphones at first; it makes the pitch a lot easier to judge than through open speakers feeding back into the mic-adjacent part of your brain.

Under the hood

• Hand tracking: MediaPipe Tasks Vision (HandLandmarker), running fully client-side. All continuous control signals go through one-pole exponential smoothing before touching any audio parameter, and freeze-then-decay to a safe default if a hand briefly leaves the frame — raw per-frame landmark noise never reaches the audio graph directly. If only one hand is in frame, it plays melody regardless of which hand MediaPipe classifies it as, so audibility never hinges on getting that mirroring-dependent label exactly right.
• Pitch: continuous rather than quantized — hand position interpolates the actual MIDI pitch between the two nearest scale degrees (not just the degree index, since scale steps aren't evenly spaced in semitones), so motion between two in-tune notes sweeps through real, in-between frequencies like a theremin instead of stepping.
• Audio engine: Tone.js for scheduling, envelopes and effects, plus two custom AudioWorkletProcessors for the more DSP-heavy oscillator engines. Five selectable engines; the first three are loosely inspired by the openly-published macro-oscillator algorithms behind Mutable Instruments' Plaits (not a clone of any proprietary firmware or preset bank):
  • Superwave — a detuned stack of oscillators (Tone.FatOscillator), the "wide digital" default sound.
  • Karplus-Strong — a plucked-string physical model (noise burst through a damped delay loop). Thematically the most literal fit: you're plucking notes out of the air.
  • Wavetable — procedurally generated single-cycle waveform frames, interpolated by a slowly drifting internal position for a bit of movement on a held note.
  • Pad — a slow-swelling detuned triangle stack with a light built-in vibrato, for a lush, evolving sustained texture.
  • Analog — a tight sawtooth unison over a square sub-oscillator an octave down: the thin-but-wide unison plus fat low end behind a lot of vintage synth leads/basses (not a clone of any specific patch).
• Effects: reverb, delay (ping-pong) and chorus run as three independent parallel sends off the filtered signal, each with its own slider, rather than one combined wet/dry mix. Envelope sustain level is a fourth live slider, shared across all five engines.
• Scales: Minor Pentatonic (default — hardest to play "wrong"), Major Pentatonic, Major (Natural), Minor (Natural), Dorian, and Hijaz (Phrygian Dominant), selectable with a root note and octave.
• HUD: a canvas overlay drawing the hand skeleton, the pitch guide grid, and a brutalist, viewfinder-style readout — kept on a separate, unmirrored layer from the selfie-mirrored video so the text stays legible instead of reading backwards.
• Backing track: a Tone.Player loaded from a user-supplied file, connected to its own gain node that sums into an outputBus sitting after the fist-mute gain — so muting your own playing doesn't cut the backing track, and the same bus is what MediaRecorder taps, so both are captured together automatically.

Important

Two of the five engines shipped silently broken, and I wouldn't have caught it myself — some musician friends tried the app, noticed Karplus-Strong and Wavetable weren't making a sound, and said so.

What was actually wrong, found by measuring the real audio output instead of watching for errors

Both engines passed every "does it throw?" check — no console errors, no rejected promises — while producing next to no sound. That meant the bug hunt had to start from "my friends say it's silent" and work backwards to a cause, rather than from a stack trace:

• Tone.Context.addAudioWorkletModule() caches a single shared promise per context regardless of which URL is requested — loading a second, different worklet module after the first silently no-ops instead of fetching it. Whichever of Karplus-Strong/Wavetable got selected second would consistently fail with no error. Fixed by calling the native audioWorklet.addModule() directly, bypassing Tone's broken cache.
• noteOn() had no guard against firing while an engine switch was still in flight. Loading a worklet engine for the first time (fetch + compile + build 4 voices) can take over a second, and a pinch landing in that window hit an unready voice pool and produced a near-silent note with no indication why.

Both were only confirmed by tapping the real audio output with an AnalyserNode and comparing peak/RMS levels against a known-good baseline — the DSP itself measured perfectly healthy in isolation in both cases; the bugs were entirely in the loading and sequencing around it, not the sound-generating code.

Stack

Vite + TypeScript, vanilla (no UI framework — this is a real-time instrument, and it keeps the bundle light and the latency low).

Running locally

npm install
npm run dev

Note

Requires a browser with camera access, AudioWorklet support, and WebGL — any recent Chrome, Edge, or Safari. Works on mobile browsers too, though hand-tracking two hands at once is naturally easier with the phone propped up than hand-held.

Deployment

Pushes to main build and deploy automatically to GitHub Pages via .github/workflows/deploy.yml.

Changelog

v1.9.0 — Backing tracks

• Load an audio file to loop and play along with; mixed into the same graph the synth uses (after the mute gain, so a fist pause doesn't cut it) so Record captures both together — the only reliable way to record over one, since external tab/app audio can't be captured without an intrusive screen-share permission.

v1.8.0 — README overhaul

• Animated SVG banner, badges, restructured around a "what makes this different" features table instead of leading straight into usage instructions.

v1.7.1 — Karplus-Strong and Wavetable actually fixed

• Reported by friends who actually tried the app and noticed two engines were silent — thank you. Root-caused by measuring real audio output (an AnalyserNode, peak/RMS) rather than trusting "no thrown errors": Tone.Context.addAudioWorkletModule() caches a single shared promise regardless of URL, silently skipping the second distinct worklet module, and noteOn() had no guard against firing before an in-flight engine switch finished loading.
• On-screen credit enlarged onto a solid background chip — was 10px at 40% opacity, illegible on phones and hard to read on desktop.

v1.7.0 — Continuous, theremin-style pitch

• Idea suggested by friends testing the app — thank you. Replaced hard scale quantization with continuous MIDI interpolation between neighboring scale degrees — gliding sweeps through real in-between frequencies instead of stepping between notes.
• Added an MIT license.

v1.6.1 — Collapsible controls panel

• A CONTROLS toggle bar collapses the full panel to a thin strip, freeing the frame on small phone screens.

v1.6.0 — Sustain control, Pad and Analog engines

• Sustain became a live slider (previously hardcoded) shared across every voice.
• Added Pad (slow-swelling detuned triangle stack) and Analog (sawtooth unison over a square sub-oscillator) engines.

v1.5.0 — Bigger pitch guide, two more scales, independent effects

• Pitch guide labels enlarged onto background chips for contrast, and the usable range narrowed from 2 octaves to 1 for legible spacing between lines.
• Added Major (Natural) and Minor (Natural) scales.
• Reverb, delay and chorus split into three independent sliders, replacing one combined "space" control.

v1.4.1 — The grid itself became the pitch guide

• Horizontal grid lines moved to sit exactly at each scale degree's height, spanning the full frame width and labeled on both edges, replacing a narrow ruler along one edge.

v1.4.0 — Manual reverb/delay, first pitch guide

• Reverb/delay moved from a left-hand pinch gesture to a manual slider.
• Added the first version of an on-screen pitch reference guide.

v1.3.0 — Karplus-Strong and Wavetable engines (first fix attempt)

• Fixed AudioWorkletNode construction failing because Tone's context isn't a native BaseAudioContext.

v1.2.0 — Mobile audio fix, clearer onboarding

• Fixed audio staying silently locked on mobile Safari — the AudioContext now starts before awaiting the camera permission prompt, not after.
• Rewrote onboarding as numbered per-hand steps with a reopenable help button.

v1.1.0 — Audibility fixes

• The filter's default cutoff before the left hand was ever used was fully closed (150Hz), cutting off most playable notes' fundamentals — raised to a bright, pleasant default.
• Added a single-hand melody fallback, so audibility doesn't depend on MediaPipe's mirroring-dependent handedness label.

v1.0.0 — Initial release

• MediaPipe hand tracking, Tone.js audio engine (Superwave, Karplus-Strong, Wavetable), scale-quantized pitch, brutalist HUD, GitHub Pages deploy.

License

MIT