VoxCanvas-41M: A Large-Scale Bilingual Dataset for Instruction-Based Speech Editing

Abstract

Recent advances in text-to-speech generation have spurred the development of speech editing techniques. However, achieving high-quality speech editing remains challenging, primarily due to two key limitations. First, a major hurdle is the lack of open-source, large-scale, high-quality speech editing datasets derived from real-world speech. Constructing such datasets is time-consuming and costly, while publicly available, scalable data construction pipelines remain limited. Second, the learning dynamics of speech editing models remain poorly understood. In particular, how models progressively acquire editing capabilities and whether learning simpler editing tasks facilitates more complex editing tasks remain unclear. To address these challenges, we introduce VoxCanvas-41M, a large-scale bilingual dataset comprising approximately 41 million speech editing pairs, constructed through a scalable data synthesis pipeline. VoxCanvas-41M covers two major categories: content editing, including single-span, multi-span, and location-reference editing, and paralinguistic editing, covering the insertion and deletion of 39 classes of nonverbal events such as laughter, sighs, and coughs. Building on VoxCanvas-41M, we train VoxCanvas-Edit, an instruction-based speech editing model, demonstrating the effectiveness of our dataset in improving editing performance. Furthermore, we systematically investigate the learning dynamics of speech editing, examining how models learn different editing operations and how data scale, task interactions, and training strategies affect editing performance.

Data Construction Pipeline

Editing Taxonomy

Every sample is a triplet ⟨source speech, instruction, target speech⟩. Single-span, multi-span and paralinguistic pairs are reversed: the resynthesized speech serves as the source and the real recording as the target, so models always learn to produce real speech. Location-reference pairs keep the forward direction.

Task	English instruction	Chinese instruction	#Pairs
Single-span editing
Insertion	Add ‘all’ after ‘you’.	在‘你们’后面加上‘都’。	10M
Deletion	Delete ‘very’ before ‘good’.	删除‘好’前面的‘很’。	10M
Substitution	Replace ‘red’ with ‘blue’.	把‘红色’改成‘蓝色’。	10M
Multi-span editing (2–6 edits per instruction)
Two edits	Add ‘all’ after ‘you’; delete ‘very’ before ‘good’.	在‘你们’后面加上‘都’；删除‘好’前面的‘很’。	10M
Location-reference editing
Ordinal	Delete the second ‘the’.	删除第二个‘的’。	100K
All occurrences	Replace every ‘red’ with ‘blue’.	把所有的‘红色’改成‘蓝色’。	100K
Sentence boundary	Delete the sentence-initial ‘so’.	删除句首的‘所以’。	100K
Time	Delete the speech from 3 s to 5 s.	删除第3秒到第5秒的语音。	100K
Paralinguistic editing
Insertion	Add ‘[surprise-oh]’ before ‘am allowed’.	在‘像蚊子哼哼’前面加上‘[laugh]’。	210K
Deletion	Delete ‘[surprise-oh]’ before ‘am allowed’.	删除‘像蚊子哼哼’前面的‘[laugh]’。	210K
Total			~41M

Dataset statistics of VoxCanvas-41M — **Figure 2.** Dataset statistics. From left to right: source utterance duration, multi-span complexity, edit-span length, and nonverbal event category distributions.

Audio Samples

All samples below are taken directly from VoxCanvas-41M, in the released source → target direction. Each waveform is shown with its mel spectrogram, and highlighted regions mark the edited spans; red text is removed from the source and green text is new in the target.

VoxCanvas-Edit Results

Fine-tuning AuK with LoRA on VoxCanvas-41M (about 1.3% of the backbone parameters updated) consistently improves editing on SpeechEditBench and Ming-Freeform-Audio-Edit, and gives better content editing than fine-tuning on GigaEdit-L under the same setup. All evaluations are run with three random seeds and the results are averaged. Avg. is the mean of Content-ZH and Content-EN. GigaEdit-L contains only English training data, so its Chinese results and bilingual averages are omitted (–). Best results are in bold.

SpeechEditBench · ACC↑
Model	Content-ZH	Content-EN	Avg.	Paralinguistic
AuK	88.32	93.78	91.05	53.25
AuK + GigaEdit-L	–	94.22	–	–
VoxCanvas-Edit	93.66	96.78	95.22	54.50

Ming-Freeform-Audio-Edit · semantic editing
Metric	Model	Basic-ZH	Full-ZH	Basic-EN	Full-EN
WER↓	AuK	4.84	3.09	4.05	4.11
	AuK + GigaEdit-L	–	–	5.80	5.95
	VoxCanvas-Edit	4.40	2.93	3.68	3.77
ACC↑	AuK	92.03	91.05	85.93	85.69
	AuK + GigaEdit-L	–	–	83.94	83.50
	VoxCanvas-Edit	92.62	91.58	87.51	86.84
SIM↑	AuK	0.87	0.88	0.88	0.88
	AuK + GigaEdit-L	–	–	0.94	0.94
	VoxCanvas-Edit	0.89	0.90	0.89	0.89
no-edit WER↓	AuK	5.00	3.17	15.52	15.58
	AuK + GigaEdit-L	–	–	16.92	17.07
	VoxCanvas-Edit	4.52	3.05	15.46	15.57

Learning Dynamics of Speech Editing

With a 278M-parameter AuK-style model trained from scratch on English data unless noted, we study how editing abilities develop and interact. Every checkpoint is averaged over three seeds.

Q1 · Data scale
How does the scale of training data affect the learning dynamics of speech editing?

Finding 1. Data scaling primarily benefits insertion and substitution, while deletion achieves strong performance with limited data.

Insertion and substitution improve substantially from 50K to 500K pairs, with diminishing returns at 5M. Deletion is less sensitive to data scale, and smaller datasets degrade in later training, likely from overfitting.

Figure 3. Effect of training data scale on the convergence of insertion, deletion, and substitution.
Q2 · Task-specific convergence
How do insertion, deletion, and substitution differ in their learning dynamics?

Finding 1. Deletion converges fastest, substitution next, and insertion slowest.

Deletion stabilizes within about 50K steps. Substitution and insertion need more steps to learn to generate new content, and insertion is the hardest. Chinese and English show a similar trend.

Figure 4. Convergence curves of individual editing tasks under single-operation training.
Q3 · Task interactions
How do insertion, deletion, and substitution interact during joint training?

Finding 2. Task interactions depend on task composition. Adding substitution largely alleviates the insertion slowdown observed in insertion–deletion joint training.

Pairing insertion with deletion substantially slows insertion while deletion is largely unaffected; pairing substitution with deletion likewise slows substitution; pairing insertion with substitution speeds up early insertion but slows substitution. Adding substitution largely alleviates the insertion slowdown: under joint training of all three, insertion matches or beats single-task training, deletion stays stable, and substitution declines only modestly.

Figure 5. Task interactions under pairwise and joint training. (a)–(c) compare pairwise training with single-operation baselines, while (d) shows joint training of all three operations. Solid and dashed lines represent mixed and single-operation training, respectively.
Q4 · Data mixture ratios
How does the data balance between generation-intensive operations and deletion affect editing performance?

Finding 3. Balanced training across insertion, deletion, and substitution provides a favorable trade-off between convergence and final performance among the tested mixtures.

Insertion : substitution : deletion = 1:1:1 converges faster with competitive final performance. More deletion data hurts insertion and substitution; 5:5:1 gives them modest gains at the cost of deletion and slower early convergence.

Figure 6. Effect of data mixture ratios on speech editing convergence. The left and right panels show Exact Match (EM) and Edit Success (ES), respectively. The ratios represent generation-intensive operations to deletion, with the corresponding insertion:substitution:deletion ratios shown in parentheses.
Q5 · Editing vs. generation
How do speech editing and generation influence each other during training?

Finding 4. Editing and generation provide complementary supervision. Joint training accelerates editing convergence and improves generation learning efficiency while maintaining comparable performance.

Finding 5. Generation initialization can accelerate editing convergence, particularly for insertion and substitution.

Joint training with TTS accelerates editing, especially insertion, but too much generation data (1:10) hurts editing. On Seed-TTS test-en, compared with generation-only training, joint training converges faster at equal steps with comparable SIM and UTMOS, and improves faster on WER, SIM, and UTMOS at an equal generation data budget; it also converges faster than using the same pairs as plain TTS data. TTS initialization accelerates editing convergence, especially for insertion and substitution, though training from scratch eventually catches up.

Figure 7. Effect of joint editing and generation training on editing convergence. The left and right panels show Exact Match (EM) and Edit Success (ES), respectively.

Figure 8. Effect of editing supervision on generation performance on Seed-TTS test-en, measured against training steps (left) and cumulative generation-task audio exposure (right).

Figure 9. Effect of TTS initialization on speech editing convergence. Overall Exact Match (EM) and Edit Success (ES) are shown on the left and right, respectively.
Q6 · Curriculum learning
Does learning single-span editing before multi-span editing improve editing performance?

Finding 6. A simple-to-complex curriculum facilitates multi-span editing. Learning basic single-span operations accelerates convergence and improves overall editing performance.

Single-to-multi curriculum substantially accelerates convergence and performs best on multi-span editing, while joint single/multi training converges more slowly than multi-span training alone. Success drops as the number of edits grows.

Figure 10. Effect of curriculum learning on multi-span editing convergence. Exact Match (EM) and Edit Success (ES) are shown on the left and right, respectively.
Q7 · Data reversal
How does data reversal affect speech quality in speech editing?

Finding 7. Reversing editing pairs mitigates acoustic quality degradation caused by synthetic training targets, helping preserve speaker similarity and speech naturalness.

With all models initialized from the same TTS checkpoint, reverse pairs (synthetic → real) give consistently higher SIM and UTMOS than forward (real → synthetic) or mixed training, suggesting that synthetic targets may pass their artifacts on to the model.

Figure 11. Effect of data reversal on speech quality. Speaker similarity (SIM) and UTMOS are shown on the left and right, respectively.
Q8 · Content vs. paralinguistic
How do content and paralinguistic editing capabilities differ in their learning dynamics?

Finding 8. Paralinguistic insertion develops faster than content insertion, while both deletion capabilities exhibit similar learning trajectories once the generated speech becomes intelligible.

With Chinese-trained models, paralinguistic insertion improves shortly after the model produces intelligible speech, whereas content insertion needs substantially more training because the model first tends to reproduce the input. For deletion, both follow similar learning trajectories once the speech becomes intelligible.

Figure 12. Learning dynamics of content and paralinguistic editing for insertion (left) and deletion (right).

Citation

@misc{niu2026voxcanvas,
  title         = {VoxCanvas-41M: A Large-Scale Bilingual Dataset for Instruction-Based Speech Editing and Its Learning Dynamics},
  author        = {Niu, Zhikang and Tu, Wenming and Li, Yalin and Huo, Yanru and Liu, Junxi and Wang, Tianrui and Yan, Ruiqi and Liang, Junxuan and Ma, Ziyang and Zheng, Zilong and Chen, Xie},
  year          = {2026},
  eprint        = {2611.08251},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2611.08251}
}