Theremin Simulator

Browser-based theremin simulator: detects your hands through the webcam with MediaPipe and synthesises sound in real time with the Web Audio API.

Live demo: https://thereminsimulator.aaranda.es/

• Right hand (on screen) → pitch. Higher = more treble.
• Left hand (on screen) → volume. Higher = louder.
• Optional hand swap (pitch ↔ volume).
• Optional scale quantization (chromatic, major, minor, pentatonic, blues, free).
• 4 waveforms with a visual picker: sine, triangle, sawtooth, square.
• Real-time oscilloscope powered by AnalyserNode.
• Presets stored in localStorage, with JSON import/export.
• Recording with MediaRecorder: clip list, inline playback, downloads.
• Multi-language UI (EN/ES) via react-i18next.
• Immersive / fullscreen mode.
• Retro-futuristic look (neon cyan/magenta palette, LCD readouts, CRT scanlines).

Stack

• Vite + React 18
• @mediapipe/tasks-vision — HandLandmarker (VIDEO mode, GPU delegate)
• Web Audio API — OscillatorNode → GainNode → BiquadFilterNode → [ctx.destination, AnalyserNode, MediaStreamAudioDestinationNode]
• i18next + react-i18next — translations in src/i18n/locales/{en,es}.json

Running it

Requires Node.js 18+.

npm install
npm run dev

Open http://localhost:5173, click Start Theremin and grant camera access. The AudioContext is initialised by that same user gesture.

For production:

npm run build
npm run preview

Usage

1. Start Theremin — grant camera access.
2. Move your right hand up/down to change pitch, and your left hand up/down for volume. The two dashed lines on the canvas mark the active zone — keep your hands within them to use the full range.
3. Use the side panel (icon-only tabs):
   • Live — live LCD readouts (frequency, note, volume, hands).
   • Wave — CRT oscilloscope of the generated audio.
   • Controls — sliders (pitch range, active zone, smoothing, volume curve), visual waveform picker, scale, hand swap, and visualization toggles (skeleton, landmarks, key point, bounding box, active band).
   • Presets — save / apply / rename / overwrite / delete and JSON import/export.
   • Recording — record button, clip list with play / download / delete.
4. Language switch (EN/ES) and immersive mode live in the header, to the left of the start button.

Technical notes

• getUserMedia requires https:// or localhost. Vite covers localhost out of the box; serve over HTTPS when deploying.
• The hand_landmarker.task model is fetched from Google's CDN on first run. For offline use, copy it into public/ and adjust MODEL_URL in src/vision/handTracker.js.
• The video is mirrored (scaleX(-1)); the code swaps MediaPipe's Handedness so "right" matches your right hand as it appears on screen.
• The stage adopts the actual video aspect-ratio dynamically once the metadata loads, avoiding the object-fit: cover crop that was misaligning the landmarks.
• The HUD volume bar is the RMS of the AnalyserNode, written to the DOM via requestAnimationFrame and a ref. It reflects the actual audible output (post-smoothing), not the instantaneous target value.
• Persistence keys: theremin:settings, theremin:viz, theremin:presets, theremin:lang (all in localStorage).

Adding a language

1. Create src/i18n/locales/<code>.json using en.json as a template.
2. Register it in src/i18n/index.js (resources).
3. Add an entry in LANGS in src/components/LanguageSwitcher.jsx.

Project structure

src/
├── App.jsx                       # composition, tabs, immersive mode
├── main.jsx
├── styles.css                    # retro-futuristic theme (Orbitron + Share Tech Mono)
├── i18n/
│   ├── index.js                  # i18next init + language persistence
│   └── locales/
│       ├── en.json
│       └── es.json
├── audio/
│   └── thereminSynth.js          # Web Audio chain + analyser + recorder stream
├── vision/
│   └── handTracker.js            # MediaPipe init + per-frame inference
├── hooks/
│   ├── useWebcam.js
│   ├── useHandTracking.js
│   ├── useAudioEngine.js
│   └── usePersistentState.js     # useState ↔ localStorage
├── components/
│   ├── WebcamView.jsx            # video + canvas overlay (skeleton/landmarks/bbox/band)
│   ├── HUD.jsx                   # LCD readouts + RMS meter
│   ├── ControlsPanel.jsx         # sliders, waveform picker, swap, viz toggles
│   ├── PresetsPanel.jsx          # CRUD + import/export
│   ├── Recorder.jsx              # MediaRecorder + clip list
│   ├── Oscilloscope.jsx          # CRT view of the AnalyserNode
│   ├── LanguageSwitcher.jsx      # globe + EN/ES
│   └── Icons.jsx                 # inline SVG set + WaveformIcon
└── utils/
    ├── mapping.js                # remapWithMargin + position→freq/gain + bbox
    └── scales.js                 # quantization + note naming

TODO

Done

• Persist settings in localStorage.
• Presets / favourites section.
• Manage presets (create, rename, overwrite, delete).
• Import/export presets (JSON).
• Hand visualization options (skeleton, landmarks, key point, bounding box, active band).
• Active zone so the control range reaches 100% without touching the edges of the frame.
• HUD with no overflow and no global app scroll.
• Recording list with per-item and bulk delete.
• Camera/canvas alignment (dynamic aspect-ratio from videoWidth/Height).
• Hand swap for pitch/volume.
• SVG icons for every action.
• Retro-futuristic look (Orbitron, neon, scanlines, LCD readouts).
• Visual waveform picker (4 waveforms with SVG thumbnails).
• Real-time oscilloscope (AnalyserNode).
• EN/ES language switch with react-i18next.
• Universal icons on the tabs (Live / Wave / Controls / Presets / Recording).
• Immersive / fullscreen mode.
• Stable volume meter (RMS reading from the analyser, no lag).

Ideas

• "Freeze" mode to hold a fixed pitch while modulating only volume.
• Configurable vibrato / portamento.
• Optional LFO over frequency or filter.
• Polyphony with multiple detuned oscillators.
• Export recordings as .wav (offline render).
• PWA / installable, with the MediaPipe model served locally.