Adobe trains an SFX instruction editor on 464k synthetic triplets, hitting SOTA edit adherence

FoleyForge: Free-form, Controllable, Rectified-flow Editing of Sound Effects

Giovana Morais, Hugo Flores García, Prem Seetharaman, Johannes Imort, Justin Salamon, Magdalena Fuentes, Oriol Nieto

cs.SD, cs.AI

2026-11-06

Adobe uses an audio LLM to author 464k edit triplets and fine-tunes a rectified-flow model into FoleyForge, an instruction-driven SFX editor that leads all baselines on adherence.

What problem this solves

Text-to-audio models are good enough now, but that is not how sound designers work. Film and game audio is built by layering single sources and refining each one against picture and direction, take after take: “muffle the bark”, “make the punch heavier”. The only workflow current models support is rewriting the prompt and regenerating from scratch, which discards the timing and timbre already dialed in, along with sync to picture.

Images solved this years ago with InstructPix2Pix-style editors that change one thing and leave the rest alone. Audio has almost nothing comparable. The strongest instruction-based system, SAO-Instruct, edits timidly; the training-free SDEdit regenerates with little tie to the input; EchoEdit+ is extremely faithful but barely edits at all. Underneath sits a data problem: supervised editing needs input/output/instruction triplets, and no public SFX editing dataset exists. FoleyForge, from Adobe and NYU, attacks both layers.

Method

Data. An audio language model (a Qwen3-Omni captioner variant) authors 464k triplets in three families. DSP edits (173k) are confined to operations like reverb, low-pass and pitch shift: the model listens to the input and writes an instruction achievable only with those operations, and the output is rendered with librosa/torchaudio, so instruction and audio are verifiably consistent. Generative edits (118k) cover semantic changes DSP cannot do, like metal door to wooden door; transformations are restricted to same-subject actions, small subject changes, location or material changes, and both clips come from the base generator Sketch2Sound, sharing an energy curve or a noise latent so the pair stays close. Auxiliary edits (173k) add compound instructions, A→B/B→A inverse pairs, and no-op samples whose instruction is “do nothing”.

Model. The backbone is a rectified-flow version of Sketch2Sound, with only the transformer fine-tuned and both encoders frozen. The source clip is encoded to clean latents and prepended as a variable-length token block in front of the noisy output sequence; self-attention lets the two sides see each other, and the prefix is sliced off before the loss. The output is therefore free to differ from the source in length and time grid, so a longer reverb tail needs no resampling or padding. SAO-Instruct instead concatenates the source into the input channels, locking input and output onto a shared latent grid and capping edit magnitude. Inference starts SDEdit-style from an intermediate noise level, which doubles as the edit-freedom dial, and the prepended source stays always-on under CFG as an identity anchor.

Controls. Each control axis is trained with dropout to learn its own null direction, so CFG can amplify it independently at inference: loudness (the input-output LUFS difference), edit strength (a loudness-normalized multi-scale log-mel L1 distance), an energy envelope (median-filtered RMS curve, from the target in training and the input at test time), and SILA phrases that quantize edit strength into “a subtle / moderate / pronounced / drastic edit” appended to the instruction.

Results

Subtle Set has 100 human-reviewed samples with reference edits; Creative Set has 90 with no ground truth and out-of-distribution instructions for every model. Two audio-language-model judges, Qwen (primary) and Gemini 3.7 Flash, rate adherence 1-5.

MethodQwen adherence SubtleQwen adherence CreativeFaithfulness CreativePQ Subtle
Reference edits3.28––6.77
SAO-Instruct2.942.690.8316.71
SDEdit3.283.340.4436.58
EchoEdit+2.522.310.8956.85
FoleyForge3.343.440.6026.47
FoleyForge+2.362.470.8896.96

On Subtle Set, FoleyForge scores 3.34 against 3.28 for the reference edits, statistically indistinguishable (Wilcoxon, p = 0.86); SAO-Instruct (p = 0.04) and EchoEdit+ (p < 0.001) sit significantly lower. On Creative Set, SAO-Instruct and FoleyForge post nearly identical direction scores (0.093 vs 0.100) yet the judges separate them widely (Qwen 2.69 vs 3.44, Gemini 3.65 vs 4.42): signal-level metrics cannot tell whether an instruction was actually followed. A 25-participant listening study ranks FoleyForge first on adherence (3.22 ± 1.29) over SDEdit (2.92, p = 0.011) and SAO-Instruct (2.69, p < 0.001), with quality at 3.43 below SAO-Instruct’s 3.73, a priced-in trade. Every control verifies as monotone; enabling the envelope raises input-output contour correlation from 0.11 to 0.37. FoleyForge+ reuses the same checkpoint with EchoEdit+’s training-free coupling to reach the faithful end (0.856 / 0.889), the best quality (PQ 6.96 / 6.94) and the worst adherence (2.36 / 2.47).

The baselines each occupy one corner of the faithfulness-adherence trade-off. The FoleyForge family walks the whole curve.

Why it matters

One model now covers the three edit families designers actually mix, DSP-like, semantic and stylistic, which previously lived in separate tools (plugins versus regeneration). For tool builders the data recipe may matter more than the model: an audio LLM authoring verifiable DSP instructions plus constrained semantic transformations produced 464k supervised triplets in a field with zero public data, and the recipe transfers to other editing tasks. To be clear about the contribution: rectified flow, SDEdit, CFG and control signals are all known ingredients; what is new is the pipeline, the length-flexible source conditioning, and turning a fixed trade-off into user-tunable knobs.

Limitations

The paper concedes three points: Subtle Set comes from the same pipeline as the training data, an in-distribution test; the primary judge, Qwen, also authored the training instructions, and the Gemini check ranks SAO-Instruct first on Subtle Set (4.58 vs 4.27), with no listening study possible there due to licensing, so the divergence is unresolved; the base model has the worst audio quality of all systems (PQ 6.47 vs 6.77 for reference edits, FAD 0.406 vs 0.340).

Further reservations: Creative Set inputs are generated by Sketch2Sound, in-distribution audio for their own model, so the wide margin carries home-field advantage; the training corpus is internally licensed, with no stated plan to release code or weights; the two ends of the trade-off remain two configurations rather than one; and editing is single-source only, while real sound design is layered.

Terms

Source

What people are saying

Related papers

All paper explainers