Semantic-VAE

Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis

[Paper] [Code] [

Zhikang Niu^1,2, Shujie Hu³, Jeongsoo Choi⁴, Yushen Chen^1,2, Peining Chen¹, Pengcheng Zhu⁵, Yunting Yang⁵,
Bowen Zhang⁵, Jian Zhao⁵, Chunhui Wang⁵, Xie Chen^1,2

¹MoE Key Lab of Artificial Intelligence, X-LANCE Lab, School of Computer Science,
Shanghai Jiao Tong University, China
²Shanghai Innovation Institute, China
³The Chinese University of Hong Kong, China
⁴Korea Advanced Institute of Science and Technology, South Korea
⁵Geely, China

Abstract. While mel-spectrograms have been widely utilized as intermediate representations in zero-shot text-to-speech (TTS), their inherent redundancy leads to inefficiency in learning text-speech alignment. Compact VAE-based latent representations have recently emerged as a stronger alternative, but they also face a fundamental optimization dilemma: higher-dimensional latent spaces improve reconstruction quality and speaker similarity, but degrade intelligibility, while lower-dimensional spaces improve intelligibility at the expense of reconstruction fidelity. To overcome this dilemma, we propose Semantic-VAE, a novel VAE framework that utilizes semantic alignment regularization in the latent space. This design alleviates the reconstruction-generation trade-off by capturing semantic structure in high-dimensional latent representations. Extensive experiments demonstrate that Semantic-VAE significantly improves synthesis quality and training efficiency. When integrated into F5-TTS, our method achieves 2.10% WER and 0.64 speaker similarity on LibriSpeech-PC, outperforming mel-based systems (2.23%, 0.60) and vanilla acoustic VAE baselines (2.65%, 0.59). We also release the code and models to facilitate further research.

Contents

Model Overview
Performance over Training Steps
Model Comparison
Citation

Model Overview

Figure 1. Overall training framework. The encoder maps the input into latent representations, regularized by a KL loss. The decoder reconstructs the waveform guided by a discriminator loss, while a semantic regularization loss aligns the latent space with features from a frozen pre-trained SSL model.

Performance over Training Steps

Figure 2. Comparison of WER and SIM on the LibriSpeech-PC test-clean dataset between the F5-TTS baseline, Vallina VAE, and Semantic-VAE across different training steps.

Model Comparison

* Each text below each audio is ASR transcribed and normalized.

Text: Harangue The tiresome product of a tireless tongue.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Harangue The tiresome product of a tireless tongue.	hair range[missing: Harangue] the tires and [missing: tiresome] product too of the [missing: a] tireless tongue	Herring, [missing: Harangue] the tiresome product of a tireless tongue.	harangue the tiresome product of a tireless tongue	harangue the tiresome product of a tireless tongue	harangue the tiresome product of a tireless tongue

Text: So I return rebuked to my content, And gain by ill thrice more than I have spent.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	So I return rebuked to my content, And gain by ill thrice more than I have spent.	syretan rebuke [missing: So I return rebuked] to my contef [missing: content,] and deign [missing: gain] by ill thrice more than i have spent	so i return rebuke [missing: rebuked] to my content and gain thy an ale [missing: by ill] thrice more than i have spent	So I return rebuke [missing: rebuked] to my content, [missing: And] Gain by ill thrice more than I have spent.	So I return rebuked to my content, And gain by ill thrice more than I have spent.	So I return rebuked to my content, And gain by ill thrice more than I have spent.

Text: Then of a sudden, at midday, the sun shot out, hot and still; no breath of air stirred; the sky was like blue steel; the earth steamed.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Then of a sudden, at midday, the sun shot out, hot and still; no breath of air stirred; the sky was like blue steel; the earth steamed.	Then of a sudden, at midday, the sun shot out hot and still. No breath of air stirred. The sky was like blue steel. The earth steamed.	then of a sudden at midday the sun shot out hot and still no breath of air stirred the sky was like blue steel the earth steamed	then of a sudden at mid-day the sun shot out hot and still no breath of air stirred the sky was like blue steel the earth steamed	Then of a sudden, at midday, the sun shot out, hot and still. No breath of air stirred. The sky was like blue steel. The earth steamed.	Then of a sudden, at midday, the sun shot out, hot and still. No breath of air stirred. The sky was like blue steel. The earth steamed.

Text: Several clothes horses, a pillion, a spinning wheel, and an old box wide open and stuffed full of coloured rags.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Several clothes horses, a pillion, a spinning wheel, and an old box wide open and stuffed full of coloured rags.	several clotheless [missing: clothes] horses, a pillion, a spinning wheel, and an old box wide open and stuffed full of colored rags.	several clothes-horses, a pillion, a spinning-wheel, and an old box wide open and stuffed full of colored rags.	Several clothes-horses, a pillion, a spinning-wheel, and an old box wide open and stuffed full of colored rags.	several clothes-horses, a pillion, a spinning-wheel, and an old box [missing: wide] open and stuffed full of colored rags.	several clothes-horses, a pillion, a spinning-wheel, and an old box wide open and stuffed full of colored rags.

Text: It is such a noble ambition, that it is a pity it has usually such a shallow foundation.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	It is such a noble ambition, that it is a pity it has usually such a shallow foundation.	it is such a noble ambition that [missing: it] is a pity [missing: it] has usually such a shallow foundation	it is such a noble ambition that it is a pity it has usually such a shallow foundation you	It is such a noble ambition, that it is a pity it has usually such a shallow foundation.	It is such a noble ambition, that it is a pity it has usually such a shallow foundation.	It is such a noble ambition, that it is a pity it has usually such a shallow foundation.

Text: Why, it's in Missouri somewhere, on the frontier I think. We'll get a map".

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Why, it's in Missouri somewhere, on the frontier I think. We'll get a map".	Why, it's in Missouri somewhere, on the frontier, I think. We'll get a map.	Why, it's in Missouri somewhere, on the frontier, I think. We'll get a map.	Why, it's in Missouri somewhere. On the frontier, I think. We'll get a map.	Why, it's in Missouri somewhere. On the frontier, I think. We'll get a map.	Why, it's in Missouri somewhere. On the frontier, I think. We'll get a map.

Text: Servadac took it for granted that the Dobryna was endeavoring to put in.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Servadac took it for granted that the Dobryna was endeavoring to put in.	Servadac took it for granted that the Dorbina [missing: Dobryna] was endeavoring to put in.	Cervidac [missing: Servadac] took it for granted that the Dabrina [missing: Dobryna] was endeavoring to put in.	Cervidac [missing: Servadac] took it for granted that the Dobrena [missing: Dobryna] was endeavoring to put in.	Servadac took it for granted that the Dobrina [missing: Dobryna] was endeavoring to put in.	Servadac took it for granted that the Dobrina [missing: Dobryna] was endeavoring to put in.

Text: Would not the loftiest eminences of the city at least be visible?

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Would not the loftiest eminences of the city at least be visible?	Would not the loftiest eminences of the city at least be visible?	Would not the loftiest eminences of the city at least be visible?	Would not the loftiest eminences of the city at least be visible?	Would not the loftiest eminences of the city at least be visible?	Would not the loftiest eminences of the city at least be visible?

Text: Then I will get me a farm and will winter in that land. Now who will follow me?

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Then I will get me a farm and will winter in that land. Now who will follow me?	then i will get me a farm and will winter in that land now who will follow me	then I will get me a farm and will winter in that land, now who will follow me.	Then I will get me a farm and will winner [missing: winter] in that land now who will follow me	then i will get me a farm and will winter in that land now who will follow me	then i will get me a farm and will winter in that land now who will follow me

Text: I like you still, Rachel; I'm sure I'll always like you.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	I like you still, Rachel; I'm sure I'll always like you.	\"'I like you still, Rachel. I'm sure I'll always like you.'	\"'I like you still, Rachel. I'm sure I'll always like you.'	I like you still, Rachel. I'm sure I'll always like you.	I like you still, Rachel. I'm sure I'll always like you.	I like you still, Rachel. I'm sure I'll always like you.

Text: Friends," said Montfichet, faintly, to the wrestlers, "bear us escort so far as the Sheriff's house.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Friends," said Montfichet, faintly, to the wrestlers, "bear us escort so far as the Sheriff's house.	\"'Friends,' said Mount Fitchet, [missing: Montfichet,] faintly, \"'to the wrestlers. \"'Bear us escort so far as the sheriff's house.'	Friends, said Montfichet faintly to the wrestlers, bear us escorts [missing: escort] so far as the sheriff's house.	\"'Friends,' said Montvichet, [missing: Montfichet,] faintly to the wrestlers, \"'Barros Escofar is [missing: "bear us escort so far as] the sheriff's house.'	Friends, said Monfichet [missing: Montfichet,] faintly to the wrestlers, bear us escort so far as the sheriff's house	Friends, said Monfichet [missing: Montfichet,] faintly to the wrestlers, bear us escort so far as the sheriff's house

Text: His conduct and presence of mind in this emergence appeared conspicuous.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	His conduct and presence of mind in this emergence appeared conspicuous.	His conduct and presence of mind in this emergence appeared conspicuous.	His conduct and presence of mind appeared conspemergence, [missing: in this emergence] appeared conspicuous.	his conduct and presence of mind in this emergence appeared conspicuous.	His conduct and presence of mind in this emergence appeared conspicuous.	His conduct and presence of mind in this emergence appeared conspicuous.

Text: Lieutenant Crosstrees is a very gallant officer.

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Lieutenant Crosstrees is a very gallant officer.	I'm no he. Lieutenant Crustrees [missing: Crosstrees] is a very gallant officer.	Lieutenant Crossstrees [missing: Crosstrees] is a very gallant officer.	Lieutenant Crosstrees is a very gallant officer.	Lieutenant Crosstrees is a very gallant officer.	Lieutenant Crosstrees is a very gallant officer.

Text: Hold him fast, my men, and as soon as I've had my coffee and oatmeal I'll take him to the Room of the Great Knife and patch him".

Prompt Speaker	Ground Truth	E2-TTS	F5-TTS	F5-TTS vanilla VAE	E2-TTS Semantic VAE	F5-TTS Semantic VAE
	Hold him fast, my men, and as soon as I've had my coffee and oatmeal I'll take him to the Room of the Great Knife and patch him".	hold him fast my men and as soon as i've had my coffee and oatmeal i'll take him to the room of the great knife and patch him	hold him fast my men and as soon as i've had my coffee and oatmeal i'll take him to the room of the great knife and patch him	hold him fast my men and as soon as i've had [missing: my] coffee and oatmeal i'll take him to the room of the great knife and patch him	hold him fast my men and as soon as i've had my coffee and oatmeal i'll take him to the room of the great knife and patch him	hold him fast my men and as soon as i've had my coffee and oatmeal i'll take him to the room of the great knife and patch him

Citation

If you use this work, please cite:

@article{niu2025semantic,
  title={Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis},
  author={Niu, Zhikang and Hu, Shujie and Choi, Jeongsoo and Chen, Yushen and Chen, Peining and Zhu, Pengcheng and Yang, Yunting and Zhang, Bowen and Zhao, Jian and Wang, Chunhui and others},
  journal={arXiv preprint arXiv:2509.22167},
  year={2025}
}