How meloform works: WASM pitch detection plus differentiable circuits learning 1,022 sine coefficients

henloitsjoyce · x · 2026-10-16

A technical breakdown of the meloform voice-reproduction pipeline:

• Front end: Rust compiled to WASM detects the sung pitch and analyzes harmonics; 16 snippets of 4 vocal cycles are aligned and resampled to 1,024 samples each.
• Differentiable circuits (handwired): each snippet gets a circuit that learns 1,022 signed sine/cosine coefficients from zero, capturing both amplitude and phase — enough to encode vowel shape and some raspy noise texture.
• Training: FFT provides target coefficients; squared-error loss is backpropagated through the circuit, updated by gradient descent.
• Elegant property: by Parseval's identity, matching these coefficients is proportional to waveform MSE (excluding DC and Nyquist components), so optimizing coefficients optimizes the waveform itself.

Original post →

More from Multimodal

• vLLM officially supports Muse Spark Open: multimodal, 1M context, agentic — vllm_project · 2026-10-16
• InfoVis: 9B unified model tops open-source slide generation and editing, trained on 13M records — Jiahao Wang · 2026-10-16
• RepForcing: feature-space supervision improves few-step autoregressive video distillation — zju · 2026-10-16
• SCOPE: semantic span-aware GRPO objective improves T2I prompt rewriting — university-of-rochester · 2026-10-16
• Orbia benchmark asks: do generated videos form a consistent 3D world? — tencent · 2026-10-16
• 3DCodeVerse: USC's ecosystem generates verifiable 3D worlds as code, with a 2M-sample dataset — UniversityofSouthernCalifornia · 2026-10-16

Multimodal channel →