zhikangniu@sjtu.edu.cn

Zhikang Niu牛志康

Ph.D. student working on speech generation and editing.

Cross Media (X-)Language Intelligence Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University. Jointly trained with Shanghai Innovation Institute.

I work on audio signal processing, speech representation and multimodal large language models, supervised by Assoc. Prof. Xie Chen. I am also interested in building scalable systems. I will try my best in the next five exciting years! 💪

Current focus

Speech-Centric Editing
Audio Tokenizer (Discrete & Continuous)
Speech Synthesis (TTS)
Multimodal LLM

Google Scholar GitHub WeChat

Research Experience

2026.04 – Present
Tencent Hunyuan
Research Intern · Foundation Model Department, Speech Team
2025.07 – 2026.02
MiniMax
Research Intern · Speech Team
2025.01 – 2025.06
Shanghai Artificial Intelligence Laboratory
Research Intern
2023.08 – 2024.09
Microsoft Research Asia (MSRA)
Research Intern, Natural Language Computing Group · Led by Furu Wei, supervised by Shujie Liu and Long Zhou. Audio codec and speech synthesis.

Education

2024.09 – 2029.06
Shanghai Jiao Tong University
Ph.D., Computer Science and Engineering
2020.09 – 2024.06
Xidian University
B.Eng., School of Artificial Intelligence

Publications

* equal contribution · † project leader · ‡ core contributor

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

Ziyang Ma^†‡, Zhikang Niu^‡, Wenming Tu^‡, Tianrui Wang^‡, Ruiqi Yan^‡, Junxi Liu^‡, Yanru Huo^‡, et al.

Technical Report 2026 Paper PDF Code Project Demo

NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization

Zhikang Niu, Sanyuan Chen, Long Zhou, Ziyang Ma, Xie Chen, Shujie Liu.

SLT 2024 Paper PDF Code BibTeX

Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis

Zhikang Niu, Shujie Hu, Jeongsoo Choi, Yushen Chen, Peining Chen, Pengcheng Zhu, Yunting Yang, et al.

Interspeech 2026 · Oral Paper PDF Code BibTeX

Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

Jeongsoo Choi^*, Zhikang Niu^*, Ji-Hoon Kim, Chunhui Wang, Joon Son Chung, Xie Chen.

Interspeech 2025 · Oral Paper PDF Code BibTeX

STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

Zihan Liu^*, Zhikang Niu^*, Qiuyang Xiao, Zhisheng Zheng, et al.

ICLR 2026 Paper PDF Code BibTeX

MMAE: A Massive Multitask Audio Editing Benchmark

MMAE Team, core contributor.

Preprint Paper PDF Code

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen.

ACL 2025 Main Paper PDF Code BibTeX Talk

F5-TTS · 15,000+ stars on GitHub.

UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

Wenhao Guan, Zhikang Niu, Ziyue Jiang, Kaidi Wang, Peijie Chen, Qingyang Hong, Lin Li, Xie Chen.

Interspeech 2026 Paper PDF Code BibTeX

Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling

Qixi Zheng, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiaofei Wang, Kai Yu, Xie Chen.

Interspeech 2025 Paper PDF Code

Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis

Qingyu Liu, Yushen Chen, Zhikang Niu, Chunhui Wang, Yunting Yang, Bowen Zhang, Jian Zhao, Pengcheng Zhu, Kai Yu, Xie Chen.

ICASSP 2026 Paper PDF

A Progressive Generation Framework with Speech Pre-trained Model for Expressive Voice Conversion

Tianrui Wang, Meng Ge, Zhikang Niu, Cheng Gong, Chunyu Qiang, Haoyu Wang, Zhongqian Huang, Ziyang Ma, Xiaofei Wang, et al.

ICME 2025

SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization

Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, et al.

ACL 2026 Main Paper PDF Code Demo BibTeX

Towards Flow-Matching-based TTS without Classifier-Free Guidance

Yuzhe Liang, Wenzhe Liu, Chunyu Qiang, Zhikang Niu, Yushen Chen, Ziyang Ma, et al.

Preprint Paper PDF

MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows

Xiquan Li, Junxi Liu, Yuzhe Liang, Zhikang Niu, Wenxi Chen, Xie Chen.

ACL 2026 Paper PDF Code BibTeX

DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation

Yakun Song, Xiaobin Zhuang, Jiawei Chen, Zhikang Niu, Guanrou Yang, Chenpeng Du, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen.

Preprint Paper PDF

Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis

Pengchao Feng, Yao Xiao, Ziyang Ma, Zhikang Niu, Shuai Fan, Yao Li, Sheng Wang, Xie Chen.

ICASSP 2026 Paper PDF

Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis

Yushen Chen, Junzhe Liu, Yujie Tu, Zhikang Niu, Yuzhe Liang, Chunyu Qiang, Chen Zhang, Kai Yu, Xie Chen.

Preprint Paper PDF Code

WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling

Wenxi Chen, Dongya Jia, Yushen Chen, Zhikang Niu, Yuzhe Liang, Xiquan Li, Ruiqi Yan, Ziyang Ma, Guanrou Yang, et al.

Preprint Paper PDF Code

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

Guanrou Yang, Tian Tan, Qian Chen, Zhikang Niu, Yakun Song, Ziyang Ma, Yushen Chen, Zeyu Xie, Tianrui Wang, et al.

Preprint Paper PDF Code

Evaluating the Expressive Appropriateness of Speech in Rich Contexts

Tianrui Wang, Ziyang Ma, Yizhou Peng, Haoyu Wang, Zhikang Niu, et al.

ACL 2026 Main Paper PDF Code BibTeX

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning

Rixi Xu, Qingyu Liu, Haitao Li, Yushen Chen, Zhikang Niu, et al.

arXiv 2026 Paper PDF Code Demo

VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech

Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen.

ICASSP 2025 Paper PDF BibTeX

Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning

Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, Xie Chen.

ASRU 2023 Paper PDF Code BibTeX

SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training

Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, et al.

ACL 2025 Findings Paper PDF Code BibTeX

URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models

Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen.

EMNLP 2025 Findings Paper PDF Code BibTeX

MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix

MMAR Team.

NeurIPS 2025 Datasets & Benchmarks Paper PDF Code

Projects

Open source

More on GitHub · encodec checkpoints on Hugging Face · EnCodec paper

Honors and Awards

2024
Third Prize (21/1600), Wenxin Cup Entrepreneurship Competition
Baidu
2024
Stars of Tomorrow
Microsoft Research Asia
2022
National Scholarship
Ministry of Education of China
2021
Meritorious Winner, Interdisciplinary Contest in Modeling
COMAP
2021, 2023
First Prize Scholarship
Xidian University

Activities

2026 – 2027
ICASSP 2027 Audio Editing Challenge Lead Organizer
Lead organizer and primary maintainer. General-purpose audio editing with natural-language instructions, with a Single Model Track and an Agent Track.
2025, 2026
Reviewer, ICME
IEEE International Conference on Multimedia and Expo
2023.09 – 2024.09
CS-BAOYAN
Owner of an open-source community for CS graduate admission.
2021.11 – Present
Datawhale
Member of an open-source AI organization, helping data science fans get involved in the AI community.
2021.11 – Present
Xmart Forum
Maintainer of an open-source student forum from SJTU X-LANCE Lab for the speech AI community.

Zhikang Niu牛志康

Research Experience

Education

Publications

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization

Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis

Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

MMAE: A Massive Multitask Audio Editing Benchmark

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling

Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis

A Progressive Generation Framework with Speech Pre-trained Model for Expressive Voice Conversion

SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization

Towards Flow-Matching-based TTS without Classifier-Free Guidance

MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows

DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation

Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis

Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis

WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

Evaluating the Expressive Appropriateness of Speech in Rich Contexts

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning

VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech

Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning

SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training

URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models

MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix

Projects

thorough-pytorch

CSBasicKnowledge

encodec-pytorch

Honors and Awards

Activities