Gesture Synth is a free browser-based instrument that turns a webcam into a chord pad and a theremin, letting you play music with hand gestures. Open the webpage and it is ready to use, with no software installation or account registration required, and hand tracking is done entirely on the device.
Uses
- Use the camera to track hand movements and generate music in real time.
- Suitable for users who want to improvise with body movements and experience gesture-based musical interaction.
Main capabilities
- Two-hand division of labor: the left hand is responsible for selecting chords, and the right hand is responsible for shaping volume, timbre, and chord style.
- Left-hand chords: choose from I, vi, IV, V, and ii, plus VI (index finger plus pinky) and VII (index finger, pinky, plus thumb). Wrist tilt can switch between major and minor.
- Right-hand controls: hand height determines volume (low to high); raising 1 to 4 fingers switches between triads, inversions, or seventh chords; wrist tilt controls timbre (filter sweep); extending the thumb lowers by one octave; making a fist mutes.
- Volume changes in real time according to the Y-axis position of the right wrist, not the distance from the camera. To keep the volume stable, you can rest your elbow on the desk and keep your hand at a fixed height.
- Two instrument modes: Gesture mode provides a continuous chord pad, and Theremin mode can play free pitch. Both can be switched within the same tool, using the same hands and camera.
- Three timbre scenes: Pop Pulse, Soft Ballad, and Retro Night. The same gestures sound different in different scenes, and you can switch at any time without reloading.
- In Theremin mode, you can play any pitch roughly between 65 Hz and 1200 Hz.
How to use
- Open Gesture Synth in a browser and allow camera access.
- Follow the on-screen prompts to make the corresponding gestures: the left hand selects chords such as I, vi, IV, V, and ii, and wrist tilt switches between major and minor.
- Use the right hand to control volume, timbre, chord type, and octave: raising the hand increases volume, lowering it decreases volume; raising 1 to 4 fingers switches between triads, inversions, or seventh chords; wrist tilt performs a filter sweep; extending the thumb lowers by one octave; making a fist mutes.
- Switch between Gesture mode and Theremin mode to choose the desired chord pad or free-pitch playing style.
- In Play, select the Pop Pulse, Soft Ballad, or Retro Night timbre scene.
Privacy and how it runs
- Hand tracking uses MediaPipe Hand La- ndmarker, this is Google's open-source machine learning model, running in the browser as WebAssembly.
- Video frames are processed locally and immediately discarded, not uploaded, with no telemetry, and no reliance on the cloud.
- No installation, no account needed.
Use cases
- Quickly experience gesture performance, play the first chord within a minute of opening the webpage.
- Practice or improvise the I, vi, IV, V, and ii chord progressions common in pop music.
- Explore free-pitch melodies in Theremin mode.
Limitations
- Currently optimized for desktop browsers, with the officially listed supported browsers being Chrome and Edge.
- Mobile support is on the roadmap and is not yet available.
- The camera is a required input, and hand tracking requires visible hand movement.
Pricing
- The official statement is that Gesture Synth is permanently free, with no subscription, no in-app purchases, and no upsells.
Other notes
- Compatibility with other browsers or platforms, specific details of the sound engine implementation, the number of playable pieces, and so on are not yet available.
