MUSICARE-VR & Adaptive AI for Dementia Care

Music reaches people with dementia when little else does. MUSICARE-VR — a collaboration led by Prof. Xiaopeng Zhao (UT Mechanical, Aerospace & Biomedical Engineering) with Duke University nursing researcher Darina Petrovsky, in which AICIP contributes the adaptive-AI core — builds a music intervention inside virtual reality that responds to the person experiencing it.

The system, built in Unreal Engine 5 for Meta Quest 3, places a person with dementia and a trained interventionist in a shared virtual space for sing-along, dance-along, and metronome activities. Biometric sensors stream heart-rate data into a real-time comfortability score; an AI “conductor” — architected by AICIP PhD student Bryce Bible — uses that signal to adapt tempo, pacing, and activity intensity moment to moment. Multiplayer networking lets caregivers join sessions from a distance. The project is funded by an NIH National Institute on Aging a2 Pilot Award through the PennAITech collaboratory.

A companion research line, co-authored by Dr. Qi, extends the same goal to socially assistive robots: integrating reinforcement learning with LLM-based agents so a Pepper humanoid robot can model a person’s cognitive and emotional state and respond with context-aware, personalized assistance. The framework uses probabilistic state modeling and LLM-driven behavior simulation to train interaction policies before they ever meet a patient.

The work is at the prototype/pilot stage: implemented activities and biometric adaptation were demonstrated at the 2025 a2 National Symposium. No clinical deployment claims are made.