Abstract

We present ANYEDIT, a unified framework for speech and singing voice editing in real-world acoustic environments. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. A straightforward solution is to directly model noisy edited audio end to end, yet this entangles semantic editing with environmental acoustics and can weaken the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage.

Methodology

Architecture
Figure 1: Overall architecture of the ANYEDIT framework.

The AR model operates in a clean semantic space to predict prosody and content tokens, supported by a noise-robust teacher-student tokenizer. A flow-matching acoustic model then renders mel-spectrograms conditioned on source acoustic context.

ANYEDITBENCH

ANYEDITBENCH covers 12 subsets (3 splits × 2 modalities × 2 languages), totaling 600 evaluation items:

  • Clean: single-span and multi-span edits on clean recordings.
  • Simulated: diverse noise and reverberation settings.
  • Real-world: in-the-wild speech and singing recordings.

🎧 Real-World Speech Editing

Orange highlight indicates edited words. Hover baseline cards to view listening notes.

🎧 Real-World Singing Voice Editing

Orange highlight indicates edited words. Red tags indicate acoustic conditions.

🎧 More Capabilities

Extra abilities beyond standard speech/singing editing. Orange highlight marks edited spans. Hover baseline cards for listening notes. For VC/SVC cards, the first reference is prosody/content and the second is target timbre.

Varying Length Editing

Original and target differ substantially in text length and/or audio duration. Hover baselines for failure modes.

Speech to Sing

Transfer a speech speaker’s timbre onto singing content/prosody.

Voice Conversion (VC)

Keep linguistic content/prosody while converting to a target speaker timbre.

Singing Voice Conversion (SVC)

Keep singing content/prosody while converting to a target singer timbre.