Neural Singing Voice Synthesis and Editing

Main Contributors: Yoonjeong Park, Soonbeom Choi

Topic 1: Singing Voice Synthesis

Singing Voice Synthesis (SVS) is a task that generates vocal sounds from melody and lyrics on computers. It has a long history dating back to the digitized vocal-tract analog model used to synthesize the song "Daisy Bell (Bicycle Built for Two)" in 1961. Since then, researchers have made great efforts to imitate natural human singing voice. This research aims to explore various topics in SVS including high-quality audio generation, controllable vocal expressions, and multilingual singing voice synthesis.

Children's Song Dataset for Singing Voice Research

"Children's Song Dataset (CSD)" is an open-source dataset for singing voice research. The dataset contains 50 Korean and 50 English songs sung by one Korean female professional pop singer. Each song is recorded with two separate keys and each audio recording is paired with transcribed MIDI and lyrics annotations with time alignment.

Related Publications

• Children’s Song Dataset for Singing Voice Research
  Soonbeom Choi, Wonil Kim, Saebyul Park, Sangeon Yong, and Juhan Nam
  Late-Breaking Demo in the 21st International Society for Music Information Retrieval Conference (ISMIR). [paper] [website] [repository]

End-to-End Singing Voice Synthesis

We have developed end-to-end SVS models that generate singing voice using only melody and lyrics as input with deep neural networks and the CSD dataset. This demo shows our recent results from a melody-unsupervised singing voice synthesis model.

Related Publications

• A Melody-Unsupervision Model for Singing Voice Synthesis
  Soonbeom Choi and Juhan Nam
  Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022 [paper] [website]

• Korean Singing Voice Synthesis Based on Auto-Regressive Boundary Equilibrium GAN
  Soonbeom Choi, Wonil Kim, Saebyul Park, Sangeon Yong, and Juhan Nam
  Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020 [paper] [website] [code]

Topic 2: Text-based Singing Voice Editing

Text-based singing voice editing (SVE) aims to generate edited singing audio from original audio, original lyrics, and edited lyrics without manual waveform manipulation. This task is similar to speech editing, which modifies speech segments to match a revised transcript. However, SVE is more challenging due to additional music-specific constraints. Edited segments must remain consistent with the original melody and rhythm, and the total duration of the edited audio must be strictly preserved to maintain synchronization with the accompaniment.

Melody-aware Duration-ratio Infilling for SVE

We propose MeloDISinger (Melody-aware Duration-ratio Infilling for singing voice editing), a flow-matching-based SVE model for melody-aware and duration-preserving editing. Its key component, MeloDRP (Melody-aware Duration Ratio Predictor), enables explicit span-wise phoneme duration control while preserving the original duration of each edit span. A flow-matching-based mel decoder then synthesizes edited regions in an audio-infilling manner while preserving non-edited regions.

Related Publications

• MeloDISinger: Melody-Aware & Duration-Preserving Singing Voice Editing with Audio Infilling
  Yoonjeong Park, Jaekwon Im and Juhan Nam
  Proceedings of the 27th INTERSPEECH Conference (Interspeech), 2026 [paper] [website]