Music and Audio Computing Lab

Neural Audio Codec


Main Contributors: Hounsu Kim, Hoyeol Sohn

Objective: Developing compact neural audio representations for effective speech transmission and speech tokenization for speech language models.

Background: Neural Audio Codec

Neural audio codecs encode audio into compact latent or discrete representations and reconstruct high-quality waveforms from them. For speech, these representations support efficient transmission and can also serve as speech tokens for speech language models.

Topic 1: Speaker-Decoupled Speech Codec

SDP-Codec is a speaker-decoupled speech codec that jointly quantizes linguistic content and pitch information with a single residual vector quantization codebook, while conditioning the decoder on a separate speaker representation. This design supports low-bitrate speech coding and zero-shot voice conversion with explicit control over speaker identity and pitch.



Topic 2: Variable Frame Rate Codec

DTM-Codec is a variable-frame-rate neural speech codec that adaptively allocates speech tokens across time. Instead of assigning the same number of tokens to silence, stable vowels, and rapid phonetic transitions, it keeps informative encoder tokens and reconstructs the removed positions with a learned mask embedding.


The model combines Dynamic Token Masking (DTM) with Path Length Equalization (PLE), a linear-time boundary selector based on cumulative feature change. Under matched total bitrate evaluation, including both content bits and position bits, DTM-Codec improves reconstruction quality and intelligibility over fixed-frame-rate baselines.


Related Publications

  • SDP-Codec: A Speaker-Decoupled Speech Codec with Pitch Injection for Low-Bitrate Coding and Zero-Shot Voice Conversion
    Hounsu Kim and Juhan Nam
    Proceedings of the 27th INTERSPEECH Conference (Interspeech), 2026
    [paper] [demo] [code]
  • DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection
    Hoyeol Sohn and Juhan Nam
    Proceedings of the 27th INTERSPEECH Conference (Interspeech), 2026
    [paper] [code]