Gesture Synth

Gesture Synth is a free online gesture synthesizer that lets users play music with both hands. It tracks hand movements through a camera and turns gestures into musical controls in real time, with no account, download, or special hardware required, and it can be used directly in the browser. It has been officially tested in Google Chrome.

Purpose

Gesture Synth is positioned for the stage "before a musical idea takes shape": explore chord progressions, shape timbre, and then record a short performance while inspiration is still fresh. It can be used to try familiar I-IV-V progressions, test chord voicing configurations, or find a texture before opening a large music project.

Main Capabilities

• Two-hand control: The left hand shapes harmony, selecting the chord degree and major/minor color; the right hand controls voicing, octave, volume, and filter. On-screen chord and quality readouts help users connect hand shapes with the sounds they hear.
• Playable sound engine: Choose from 12 keys and switch between three synth timbres: warm, bright, and retro. The browser-native sound engine responds instantly to changes in hand pose.
• Local MP4 performance recording: Capture up to 5 minutes of real-time synthesizer visuals with generated audio and download it as MP4. When recording is selected, the browser requests microphone permission to combine microphone audio with the performance canvas. The download contains only the performance stage, not page controls, other tabs, application windows, or the desktop screen.
• Privacy by design: Camera footage, microphone audio, synth audio, hand keypoints, and recordings all remain in the local instrument; when enabled, anonymous page usage telemetry may be sent to Microsoft Clarity.

How to Use

1. Keep both hands in frame, with good lighting and enough room to move.
2. Once the preview is ready, select the central circular control to start making sound.
3. To make a local MP4 recording, use the red recording control.

Basic performance flow: allow camera access (requested when the homepage opens), select the central play button, form a I chord with the left hand, then add a root-position voicing with the right hand. Extending one non-thumb finger on the left hand represents the I degree; tilting inward is major (a neutral tilt is also major), and hold the gesture until the chord label stabilizes; try a small outward tilt to hear the minor color. Extending one non-thumb finger on the right hand represents root position; raise the hand to increase volume and lower it to decrease volume, and add another non-thumb finger to hear the first inversion. It is recommended to change only one control at a time and let the visual readout stabilize before switching to the next hand.gesture.

Use cases

• Use the left hand's movement to switch between chord degrees, use tilt to shape major/minor color, and find progressions.
• Use the right hand to change voices, octaves, volume, and filter one at a time, giving the sound character (thumb position changes octave, right-hand tilt changes filter response).
• When a short idea appears, record it locally as MP4 in a supported browser, and decide for yourself where to share it; the downloaded file stays on the device, and Gesture Synth has no publishing workflow.

How it works

MediaPipe estimates 21 keypoints for each hand. A lightweight gesture mapper turns these positions into stable musical states, and then Web Audio responds, with no server round trip. The tracker prefers GPU processing and falls back to CPU when necessary, providing a second path to start the instrument on compatible devices.

Limitations and notes

• The instrument is free to use and does not require an account, subscription, or payment method.
• Hand detection, microphone mixing, and recording run in the browser; the camera image, keypoints, microphone audio, synthesized audio, and recorded content are not sent to a server. Camera access can be stopped by closing the page, revoking browser permissions, or disabling the camera on the device.
• Tested in Google Chrome; camera access, hand tracking, sound, canvas capture, and MP4 recording depend on the browser and device used, and other browsers are not guaranteed. If a feature is unavailable, please first read the specific prompt shown in the instrument before retrying.
• Recording automatically stops after 5 minutes.
• Modern browsers only allow camera access on secure pages: localhost works during development, and public deployments must use HTTPS.
• Other details (such as specific pricing, more platform support, etc.) are not yet provided.