SeamlessEdit: Background Noise Aware Zero-Shot Speech Editing with in-Context Enhancement

Abstract

Accepted to EUSIPCO 2026 · arXiv:2505.14066

With the fast development of zero-shot text-to-speech technologies, it is possible to generate high-quality speech signals that are indistinguishable from real ones. Speech editing, including speech insertion and replacement, appeals to researchers due to its potential applications. However, existing studies only considered clean speech scenarios. In real-world applications, the existence of environmental noise could significantly degrade the quality of generation. In this study, we propose a noise-resilient speech editing framework, SeamlessEdit, for noisy speech editing. SeamlessEdit adopts a frequency-band-aware noise suppression module and an in-context refinement strategy. It can well address the scenario where the frequency bands of voice and background noise are not separated. The proposed SeamlessEdit framework outperforms state-of-the-art approaches in multiple quantitative and qualitative evaluations.

Framework

Separation → frequency-band-aware suppression → in-context refinement.

Figure 1. SeamlessEdit separates human voice X_s and suppresses residual noise to derive an edited speech X_le. In-context refinement enhances the editing of X_s using X_le for indistinguishable noisy editing results. The SBL filter is adopted to improve robustness to noise.

1

Speech Separation

StoRM, a diffusion-based stochastic regeneration model, separates the input into voice X_s and background noise X_n in a few diffusion steps.

2

Frequency-Band Aware Suppression

Sparse Bayesian Learning (SBL) estimates the sparse voice frequency bands; a Butterworth SBL filter suppresses uniformly distributed residual noise to yield X_l.

3

Neural Codec Editing

A VoiceCraft-based editing model performs insertion, deletion, and substitution over the masked target region, producing X_le.

4

In-Context Refinement

A multi-head attention module uses low-frequency embedding X_le (K, V) to enhance X_s (Q); the separated noise X_n is added back to form output Y.

Results

Insertion and replacement on the EARS-WHAM dataset under noisy conditions.

Model	Insertion				Short Replacement				Long Replacement
Model	WER↓	PES↓	NMOS↑	SMOS↑	WER↓	PES↓	NMOS↑	SMOS↑	WER↓	PES↓	NMOS↑	SMOS↑
Ground Truth	–	–	–	–	0.12	0.51	3.97	3.82	0.12	0.51	3.97	3.82
VoiceCraft*	0.18	0.56	4.23	4.11	0.10	0.52	4.25	4.19	0.10	0.54	4.32	4.23
FluentSpeech	–	–	–	–	0.23	1.82	1.67	1.66	0.22	1.84	1.50	1.49
VoiceCraft	0.33	1.28	2.93	2.76	0.24	1.23	2.90	2.99	0.23	1.16	2.93	2.60
SeamlessEdit w/o ICL	0.28	0.89	3.75	3.48	0.23	0.80	3.38	3.12	0.24	0.86	3.42	3.13
SeamlessEdit (ours)	0.28	0.77	3.78	3.43	0.22	0.72	3.56	3.13	0.22	0.75	3.65	3.11

Table I. Lower WER/PES and higher NMOS/SMOS are better. “*” denotes the clean condition (upper bound). SeamlessEdit achieves the best noisy-condition NMOS and lowest PES, with a +29.01% NMOS and +24.28% SMOS gain over state-of-the-art systems.

Audio Demos

Listen to insertion and replacement results under noisy conditions. In each transcript, green marks inserted words and red marks replaced words.

Citation

If you find this work useful, please cite:

@inproceedings{chen2026seamlessedit,
  title     = {SeamlessEdit: Background Noise Aware Zero-Shot Speech Editing with in-Context Enhancement},
  author    = {Chen, Kuan-Yu and Li, Jeng-Lin and Lu, De-Yan and Ding, Jian-Jiun},
  booktitle = {European Signal Processing Conference (EUSIPCO)},
  year      = {2026},
  note      = {arXiv:2505.14066}
}