Course Description

This course aims to learn the new technology to learn machine learning with applications to the music and audio domains. Specificially, we handle various tasks in the topics of music and audio classification, automatic music transcription, source separation, sound synthesis, and music generation. Student will have hands-on experiences using audio processing and machine learning libraries through the assignments and gain experience of the full cycle of research through the final research project.

Textbooks

• Fundamentals of Music Processing (FMP), Meinard Müller, Springer, 2015
( Python Notebooks for Fundamentals of Music Processing )
• Pattern Recognition And Machine Learning (PRML), C. Bishop, Springer, 2006
• Deep Learning, Ian Goodfellow and Yoshua Bengio and Aaron Courville (DL), MIT Press, 2017

KLMS             GitHub (Code)             Resources

Schedule

Week	Topics
1	[Aug-29, online] Course Introduction [slides] [Aug-31, online] Audio Data Representations [slides] [practice] audio_representations.ipynb Suggested Readings The FMP book (Chapter 1: Music Representations) The FMP book (Chapter 2: Fourier Analysis of Signals)
2	[Sep-5, online] Music Classification - Overview and Audio Feature Extraction [slides] [Sep-7, online] Music Classification - Overview and Audio Feature Extraction (Cont'd) [practice] hand-crafted audio features.ipynb Suggested Readings Signal Processing for Music Analysis, Müller et al. (2011) Feature Learning and Deep Architectures: New directions for Music Informatics, Humphrey, Bello and LeCun (2013)
3	[Sep-12] No Class (Chuseok Holiday) [Sep-14, online] Music Classification - Traditional Machine Learning [slides] [practice] Unsupervised Learning.ipynb [Homework #1] Musical Instrument Recognition (Due Sep 25, 11:59 PM) [link] Suggested Readings The PRML book (Chapter 3: Linear Models for Regression) The PRML book (Chapter 4: Linear Models for Classification) The PRML book (Chapter 5: Neural Networks) The PRML book (Chapter 6: Kernel Methods) The PRML book (Chapter 9: Mixture Models and EM) The PRML book (Chapter 12: Continuous Latent Variables) Musical Genre Classification of Audio Signals, Tzanetakis and Cook (2002)
4	[Sep-19, online] Music Classification - Deep Learning Overview [slides] [Sep-21, online] Music Classification - Deep Learning Overview (Cont'd) Suggested Readings The DL book (Chapter 6: Deep Feedforward Networks) The DL book (Chapter 7: Regularization for Deep Learning) The DL book (Chapter 8: Optimization for Training Deep Models) The DL book (Chapter 9: Convolutional Networks) Deep Learning for Audio-Based Music Classification and Tagging: Teaching Computers to Distinguish Rock from Bach, Nam et al. (2018)
5	[Sep-26, online] Music Classification - Recent Advances [slides] [Sep-28, hybrid] Music Recommendation and Retrieval [slides] Suggested Readings The FMP book (Chapter 7: Content-based Audio Retrieval) ISMIR 2021 Tutorial - Music Classification: Beyond Supervised Learning, Towards Real-world Applications Content-Based Music Information Retrieval: Current Directions and Future Challenges, Casey et al. (2008)
6	[Oct-3] No Class (National Holiday) [Oct-5, hybrid] HW1 review, HW2 introduction, and Music Recommendation and Retrieval (Cont'd) [Homework #2] Music Auto-Tagging (Due Oct 16, 11:59 PM) [link]
7	[Oct-10] No Class (National Holiday) [Oct-12, online] Automatic Music Transcription - Overview and Monophonic Pitch Estimation [slides] Suggested Readings Automatic Music Transcription: An Overview, Benetos et al. (2018) The DL book (Chapter 10: Sequence Modeling: Recurrent and Recursive Nets)
8	[Oct-17, online] Automatic Music Transcription - Polyphonic Pitch Estimation and Note Transcription (Make-Up) [slides] Suggested Readings The FMP book (Chapter 8: Musically Informed Audio Decomposition)
9	[Oct-24, hybrid] Automatic Music Transcription - Audio-to-Score Alignment [slides] [Oct-26, hybrid] HW2 review, HW3 introduction, and Automatic Music Transcription - Audio-to-Score Alignment (Cont'd) [Homework #3] Polyphonic Piano Transcription (Due Nov 9, 11:59 PM) [link] Suggested Readings The FMP book (Chapter 3: Music Synchronization)
10	[Oct-31, hybrid] Automatic Music Transcription - Rhythm Transcription [slides] [Nov-2 (Wed, 7pm), hybrid] Automatic Music Transcription - Chord Recognition [slides] Suggested Readings The FMP book (Chapter 5: Chord Recognition) The FMP book (Chapter 6: Tempo and Beat Tracking) ISMIR 2021 Tutorial - Tempo, Beat, and Downbeat Estimation
11	[Nov-7] No Class (Conference Trip) [Nov-9, video] Music Source Separation [slides] Suggested Readings The FMP book (Chapter 8: Musically Informed Audio Decomposition) The DL book (Chapter 14: Auto-Encoder) ISMIR 2020 Tutorial - Open Source Tools & Data for Music Source Separation
12	[Nov-14, hybrid] Symbolic Music Generation - Overview and RNN Models [slides] [Nov-16, hybrid] HW3 Review, HW4 Introduction, Symbolic Music Generation - RNN Models (Cont'd) [Nov-18, video] Symbolic Music Generation - Transformer Models [slides] Individual Project Meeting with Professor [Homework #4] (Optional) Symbolic Music Generation [link]
13	[Nov-21, hybrid] Audio Generation - Overview and Deep Learning Models [slides] [Nov-23, hybrid] Singing Voice Synthesis (Invited Talk: Soonbeom Choi, PhD Student) Individual Project Meeting with Professor
14	[Nov-28 and Nov-30, hybrid] Student Presentation: Project Introduction
15	[Dec-7, online] Invited Talk: Dr. Taesu Kim (Neosapience CEO)
16	[Dec-14, hybrid] Student Presentation: Final Project