Zhikang Niu (牛志康)

Ph.D. Student. Cross Media (X-)Language Intelligence Lab
Department of Computer Science and Engineering, Shanghai Jiao Tong University.
Jointly trained with Shanghai Innovation Institute.
Current Topic Speech-Centric Editing
Shanghai, China
Research Interests Education Research Experience Publications Research Projects Honors and Awards Activities
if you have any questions, please feel free to contact me with zhikangniu@sjtu.edu.cn
[GitHub] [Google Scholar] [WeChat] [Resume]

Research Interest

 I work in the field of Audio Singal Processing, Speech Representation, and Multimodal Large Language Model supervised by Assoc. Prof. Xie Chen, I will try my best in the next five exciting years! 💪. I am also interested in building scalable systems. Currently, I focus on the following research topics:

Education

Research Experience

Publications

Speech Synthesis/Omni System
  • Zhikang Niu, Sanyuan Chen, Long Zhou, Ziyang Ma, Xie Chen, Shujie Liu.
    NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization.
    SLT 2024, [Link] [PDF] [Code] [BibTeX]

  • Zhikang Niu, Shujie Hu, Jeongsoo Choi, Yushen Chen, Peining Chen, Pengcheng Zhu, Yunting Yang, et al.
    Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis.
    InterSpeech 2026, [Link] [PDF] [Code] [BibTeX]

  • Tianrui Wang, Ziyang Ma, Yizhou Peng, Haoyu Wang, Zhikang Niu, et al.
    Evaluating the Expressive Appropriateness of Speech in Rich Contexts.
    ACL 2026 Main, [Link] [PDF] [Code] [BibTeX]
  • Rixi Xu, Qingyu Liu, Haitao Li, Yushen Chen, Zhikang Niu, et al.
    X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning.
    arXiv 2026, [Link] [PDF] [Code] [Demo]
  • Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, et al.
    SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization.
    ACL 2026 Main, [Link] [PDF] [Code] [Demo] [BibTeX]
  • Jeongsoo Choi*, Zhikang Niu*, Ji-Hoon Kim, Chunhui Wang, Joon Son Chung, Xie Chen.
    Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment.
    InterSpeech 2025 Oral, [Link] [PDF] [Code] [BibTeX]

  • Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen.
    F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching.
    ACL 2025 Main, [Link] [PDF] [Code] [BibTeX] [Talk]
    F5-TTS has collected 13,000+ stars on GitHub.
  • Wenhao Guan, Zhikang Niu, Ziyue Jiang, Kaidi Wang, Peijie Chen, Qingyang Hong, Lin Li, Xie Chen.
    UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models.
    InterSpeech 2026, [Link] [PDF] [Code] [BibTeX]

  • Qixi zheng, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiaofei Wang, Kai Yu, Xie Chen.
    Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling.
    InterSpeech 2025, [Link] [PDF] [Code]

  • Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, et al.
    SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training.
    ACL 2025 Findings, [Link] [PDF] [Code] [BibTeX]

  • Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen.
    VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech.
    ICASSP 2025, [Link] [PDF] [BibTeX]

  • Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, Xie Chen.
    Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning.
    ASRU 2023, [Link] [PDF] [Code] [BibTeX]

  • Yuzhe Liang, Wenzhe Liu, Chunyu Qiang, Zhikang Niu, Yushen Chen, Ziyang Ma, et al.
    Towards Flow-Matching-based TTS without Classifier-Free Guidance.
    [Link] [PDF]

  • Xiquan Li, Junxi Liu, Yuzhe Liang, Zhikang Niu, Wenxi Chen, Xie Chen.
    MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows. ACL 2026,
    [Link] [PDF] [Code] [BibTeX]

  • Yuxiang Zhao, Yunchong Xiao, Yushen Chen, Zhikang Niu, Shuai Wang, Kai Yu, Xie Chen.
    Traceable TTS: Toward Watermark-Free TTS with Strong Traceability.
    [Link] [PDF]

  • Qingyu Liu, Yushen Chen, Zhikang Niu, Chunhui Wang, Yunting Yang, Bowen Zhang, Jian Zhao, Pengcheng Zhu, Kai Yu, Xie Chen.
    Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis.
    ICASSP 2026, [Link] [PDF]

  • Tianrui Wang, Meng Ge, Zhikang Niu, Cheng Gong, Chunyu Qiang, Haoyu Wang, Zhongqian Huang, Ziyang Ma, Xiaofei Wang, et al.
    A Progressive Generation Framework with Speech Pre-trained Model for Expressive Voice Conversion.
    ICME 2025.

  • Yakun Song, Xiaobin Zhuang, Jiawei Chen, Zhikang Niu, Guanrou Yang, Chenpeng Du, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen.
    DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation.
    [Link] [PDF]

  • Pengchao Feng, Yao Xiao, Ziyang Ma, Zhikang Niu, Shuai Fan, Yao Li, Sheng Wang, Xie Chen.
    Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis.
    ICASSP 2026, [Link] [PDF]

  • Yushen Chen, Junzhe Liu, Yujie Tu, Zhikang Niu, Yuzhe Liang, Chunyu Qiang, Chen Zhang, Kai Yu, Xie Chen.
    Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis.
    [Link] [PDF]

  • Wenxi Chen, Dongya Jia, Yushen Chen, Zhikang Niu, Yuzhe Liang, Xiquan Li, Ruiqi Yan, Ziyang Ma, Guanrou Yang, et al.
    WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling.
    [Link] [PDF]

  • Guanrou Yang, Tian Tan, Qian Chen, Zhikang Niu, Yakun Song, Ziyang Ma, Yushen Chen, Zeyu Xie, Tianrui Wang, et al.
    WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling.
    [Link] [PDF]

Benchmark
  • MMAE Team, core contributor.
    MMAE: A Massive Multitask Audio Editing Benchmark.
    [Link] [PDF]

  • MMAR Team.
    MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix.
    NeurIPS Dataset & Benchmark 2025, [Link] [PDF] [Code]

  • Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen.
    URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models.
    EMNLP 2025 Findings, [Link] [PDF] [Code] [BibTeX]

  • Zihan Liu*, Zhikang Niu*, Qiuyang Xiao, Zhisheng Zheng, et al.
    STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence.
    ICLR 2026, [Link] [PDF] [Code] [BibTeX]

Research Projects

 Open-Source Projects:
  • thorough-pytorch: A Chinese PyTorch tutorial and it has already collected 2,300 more stars and 333 forks on GitHub.
  • CSBasicKnowledge: This repo will record some knowledge about computer science, artificial intelligence and EE. It has already collected 560 more stars on GitHub.
  • More open-source contents can be found on my GitHub.
Research Projects

Honors and Awards
  • 2024, Third Prize, 21/1600, Wenxin Cup Entrepreneurship Competition, Baidu.
  • 2024, Stars of Tomorrow, Microsoft Research Asia.
  • 2022, National Scholarship, Ministry of Education in China.
  • 2021, Meritorious Winner, Interdisciplinary Contest In Modeling.
  • 2021, 2023, The First Prize Scholarship, Xidian University.

Activities
  • ICME Reviewer, 2025, 2026
  • 2023.09-2024.9, CS-BAOYAN owner (an open-source CS-BAOYAN organization).
  • 2021.11-Now, Datawhale member (an open-source AI organization), helped data science fans get involved in the AI community.
  • 2021.11-Now, Xmart forum maintainer (an open-source student forum from SJTU X-LANCE Lab), for helping students get involved in the speech AI community.



Last updated: