VirtualPianoLesson (2): Mixed Reality Framework for Interactive Piano Playing

Theoretical (Analytical):

Practical (Implementation):

Literature Work:


  • The overall goal of the VirtualPianoLesson project is to create a mixed reality piano lesson, that shows the student a visual representation of the musical piece they want to learn, tracks the progress during the virtual practice session, and adapts the difficulty level and visual representation of the lesson to the student’s ability.
  • This project/thesis focusses on the development of mixed reality piano playing.
  • During the project, different parts of the mixed reality environment need to be synchronized (real/virtual keyboard, user vision, user’s hands).
  • The mixed reality environment will be implemented with a HTC Vive using Unity.

Problem Statement

  • How can we synchronize a virtual and real keyboard using a HTC Vive?
  • How can we trace the user's vision and hand movement on the keyboard using a HTC Vive?


  • Assess existing Mixed Reality frameworks
  • Derive different levels of virtualization of a piano lesson
  • Implement the Mixed Reality Environment using a HTC Vive and a Komplete Kontrol S88 keyboard


  • Interest in music/piano education
  • Knowledge about Augmented Reality
  • Good programming skills in C#/Java (Unity3D) and basic web programming skills (HTML/CSS/JS,D3)


  • Scope: Bachelor / Master
  • Project/Thesis Duration: 3 months/3 months (Bachelor), 6 months / 6 months (Master)
  • Start: immediately



  • Desnoyers-Stewart, J., Gerhard, D., & Smith, M. (2017). Mixed Reality MIDI Keyboard. In Proceedings of the 13th International Symposium on CMMR (pp. 376-386).
  • Desnoyers-Stewart, J., Gerhard, D., & Smith, M. L. (2018). Augmenting a MIDI Keyboard Using Virtual Interfaces. Journal of the Audio Engineering Society, 66(6), 439-447.
  • Liang, Hui, et al. "Barehanded music: real-time hand interaction for virtual piano." Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 2016