Zhikang Niu (牛志康)

Ph.D. Student. Cross Media (X-)Language Intelligence Lab
Department of Computer Science and Engineering, Shanghai Jiao Tong University.
Work at Shanghai Innovation Institute.

Shanghai, China

Research Interests Education Research Experience Publications Research Projects Honors and Awards Activities

if you have any questions, please feel free to contact me with zhikangniu@sjtu.edu.cn

[GitHub] [Google Scholar] [WeChat] [Resume]

Research Interest

I work in the field of Audio Singal Processing, Audio Codec Model,Multimodal Large Language Model, Machine learning, and Deep learning supervised by Prof. Xie Chen, I will try my best in the next five exciting years! 💪. Currently, I focus on the following research topics:

Audio Tokenizer (Discrete and Continuous)
Speech Synthesis (Text to Speech)
Multimodal Large Language Model

Education

2024.09–2029.06 Ph.D. Student, Department of Computer Science and Engineering, Shanghai Jiao Tong University.
2020.09–2024.06 Bachelor Degree of Engineering, School of Artificial Intelligence, Xidian University.

Research Experience

2025.07–current Research Intern, Minimax Speech Team.
2025.01–2025.06 Research Intern, Shanghai Artificial Intelligence Laboratory.
2023.08–2024.09 Research Intern, Natural Language Computing Group (NLC), Microsoft Research Asia (MSRA). Led by Furu Wei, supervised by Shujie Liu and Long Zhou. Focus on Audio Codec and Speech Synthesis.

Publications

Speech Synthesis/Omni System

Zhikang Niu, Sanyuan Chen, Long Zhou, Ziyang Ma, Xie Chen, Shujie Liu.
NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization.
SLT 2024, [Link] [PDF] [Code] [BibTeX]

Zhikang Niu, Shujie Hu, Jeongsoo Choi, Yushen Chen, Peining Chen, Pengcheng Zhu, Yunting Yang, Bowen Zhang, Jian Zhao, Chunhui Wang, Xie Chen.
Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis.
[Link] [PDF] [Code] [BibTeX]

Jeongsoo Choi^*, Zhikang Niu^*, Ji-Hoon Kim, Chunhui Wang, Joon Son Chung, Xie Chen.
Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment.
InterSpeech 2025 Oral, [Link] [PDF] [Code] [BibTeX]

Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen.
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching.
ACL 2025 Main, [Link] [PDF] [Code] [BibTeX] [Talk]
F5-TTS has collected 13,000+ stars on GitHub.

Wenhao Guan, Zhikang Niu, Ziyue Jiang, Kaidi Wang, Peijie Chen, Qingyang Hong, Lin Li, Xie Chen.
UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models.
[Link] [PDF] [Code] [BibTeX]

Qixi zheng, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiaofei Wang, Kai Yu, Xie Chen.
Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling.
InterSpeech 2025, [Link] [PDF] [Code]

Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, et al.
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training.
ACL 2025 Findings, [Link] [PDF] [Code] [BibTeX]

Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen.
VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech.
ICASSP 2025, [Link] [PDF] [BibTeX]

Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, Xie Chen.
Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning.
ASRU 2023, [Link] [PDF] [Code] [BibTeX]

Yuzhe Liang, Wenzhe Liu, Chunyu Qiang, Zhikang Niu, Yushen Chen, Ziyang Ma, et al.
Towards Flow-Matching-based TTS without Classifier-Free Guidance.
[Link] [PDF]

Xiquan Li, Junxi Liu, Yuzhe Liang, Zhikang Niu, Wenxi Chen, Xie Chen.
MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows.
[Link] [PDF] [Code] [BibTeX]

Yuxiang Zhao, Yunchong Xiao, Yushen Chen, Zhikang Niu, Shuai Wang, Kai Yu, Xie Chen.
Traceable TTS: Toward Watermark-Free TTS with Strong Traceability.
[Link] [PDF]

Benchmark

MMAR Team.
MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix.
NeurIPS Dataset & Benchmark 2025, [Link] [PDF] [Code]

Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen.
URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models.
EMNLP 2025 Findings, [Link] [PDF] [Code] [BibTeX]

Zihan Liu*, Zhikang Niu*, Qiuyang Xiao, Zhisheng Zheng, et al.
STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence.
[Link] [PDF] [Code] [BibTeX]

Research Projects

Open-Source Projects:

thorough-pytorch: A Chinese PyTorch tutorial and it has already collected 2,300 more stars and 333 forks on GitHub.
CSBasicKnowledge: This repo will record some knowledge about computer science, artificial intelligence and EE. It has already collected 560 more stars on GitHub.
More open-source contents can be found on my GitHub.

Research Projects

encodec-pytorch: An unofficial PyTorch implementation of the High Fidelity Neural Audio Compression and it has already collected 163 stars on GitHub. [checkpoint]

Honors and Awards

2024, Third Prize, 21/1600, Wenxin Cup Entrepreneurship Competition, Baidu.
2024, Stars of Tomorrow, Microsoft Research Asia.
2022, National Scholarship, Ministry of Education in China.
2021, Meritorious Winner, Interdisciplinary Contest In Modeling.
2021, 2023, The First Prize Scholarship, Xidian University.

Activities

ICME Reviewer, 2025, 2026
2023.09-2024.9, CS-BAOYAN owner (an open-source CS-BAOYAN organization).
2021.11-Now, Datawhale member (an open-source AI organization), helped data science fans get involved in the AI community.
2021.11-Now, Xmart forum maintainer (an open-source student forum from SJTU X-LANCE Lab), for helping students get involved in the speech AI community.

Updating time: 2025.8.20