Pianist Multimodal Data Acquisition and Generation

Main Contributors: Joonhyung Bae, Taegyun Kwon, Yonghyun Kim, Kirak Kim

The primary objective of this research is to enhance our understanding of piano performance through the lens of multimodal data analysis. The central problem it addresses is the gap in comprehensive data that integrates both the auditory and physical aspects of piano playing. By focusing on the multimodal aspects of piano performance, this study aims to bridge this gap, providing insights into the nuanced interplay between musicians and their instruments. The expected outcomes include the development of advanced AI algorithms for pianist motion generation and a deeper understanding of the intricate dynamics of musical performance. This research is anticipated to pave the way for significant advancements in AI-driven music synthesis, performance analysis, and the creation of more intuitive and human-like interfaces for musical interaction.

Pianist Motion Data Acquisition

The acquisition phase of this study involves collecting a diverse range of data from both professional and amateur pianists. This includes audio recordings, MIDI outputs, detailed motion capture data, and multi-angle video recordings. The aim is to capture a holistic view of piano performance, encompassing the auditory experience as well as the physical movements and techniques of the pianists. By amassing this multimodal dataset, the study seeks to create a comprehensive resource that can be utilized for in-depth analysis and the development of AI models. This phase is critical in laying the groundwork for understanding the complex interactions and nuances of piano playing.

Related Publications

• Automated Gap-filling Algorithm for Kinetic Data of Finger Movement: Pianist Hand Motion Cleaning Using Spatio-temporal Imputation
  Taegyun Kwon, Dawon Park, Joonhyung Bae, Jonghwa Park, Jaebum Park, and Juhan Nam
  Proceedings of the 29th Congress of International Society of Biomechanics, 2023 [paper]
• Multivariate Kinematic Analysis of Piano Performance: a Case Study of a Professional Pianist
  Dawon Park, Taegyun Kwon, Joonhyung Bae, Jonghwa Park, Juhan Nam, and Jaebum Park
  Proceedings of the 29th Congress of International Society of Biomechanics, 2023 [paper]

Pianist Motion Generation

Building on the data acquired, the next phase of the research focuses on the generation of pianist motion using AI algorithms. These algorithms are designed to replicate the intricate movements and techniques of pianists, based on the analysis of the collected multimodal data. The challenge lies in accurately capturing the subtleties of human motion and translating them into a format that can be understood and replicated by AI. This aspect of the research is vital for advancing our capabilities in synthesizing realistic piano performances through AI. The ultimate goal is to create AI models that not only mimic human performance but also contribute to the development of new tools and methods for musical composition and education.