off-the-shelf.

off-the-shelf is an augmented reality interface for collaborative music-making with an AI musician. It visualises notes, controls, and sound parameters so performers can follow contributions and respond while improvising.

Keywords: Interaction design, functional prototyping, performance-based iteration, and user research.

[-] The design challenge

Electronic music can be difficult to follow through sound alone. When several instruments overlap, musicians may struggle to identify who is contributing and which parameters are changing.

I explored how visual feedback could help musicians understand this activity while staying engaged in the creative process.

[-] Designing the interface

I adapted my arMIDI system into a working AR prototype connected to the IMPSy generative music system. Three decisions shaped the interface:

• Represent musical interaction data. A generic keyboard-and-control layout organises note events, control changes, and parameter values across different electronic instruments. It represents musical activity at the data level, rather than reconstructing bodily gestures.
• Make changes visible as they happen. Keys light up yellow when notes play, while seven virtual knobs reflect changes to sound parameters. Performers can connect what they hear with what is changing.
• Distinguish collaborators. In group sessions, grey and purple representations separate the human and AI contributors, with yellow highlights showing current activity. The visualisations sit within the performance space through an AR headset.

I built the interface in Unity with Microsoft MRTK for HoloLens 2, using Node.js and WebSocket to exchange MIDI data in real time.

[-] Prototyping through performance

I explored the prototype through recorded demos, a live research concert, and trio improvisation. Each setting exposed different constraints.

The first setup visualised notes but offered limited control over sound character. A later configuration expanded the system’s parameter control, creating more varied textures and opportunities for musical call and response. Trio sessions added visualisations for both collaborators and explored shared versus separate instruments, revealing how the setup shaped the AI’s perceived role.

These experiences also exposed a persistent interaction problem: fast-changing activity could be visible without being easy to follow.

[-] Evaluating with 16 musicians

I conducted a study with 16 experienced electronic musicians to understand how visual feedback affected their creative experience, both alone with the AI and alongside another human musician.

[>] Outcome

Visibility needs to support action. Seeing notes and parameter changes helped some musicians identify contributions and respond deliberately. Rapid activity could still be difficult to interpret, and clearer information did not consistently lead to better collaboration.

Design for people creating together. A second human often provided stability and shared understanding. The interface supported this connection as well as awareness of the AI, highlighting the importance of supporting human coordination in creative tools.

The study informed design directions for readable, attributable feedback and flexible collaboration. It provides qualitative insights from music sessions with experienced musicians.

[>] Papers and performance

• AI See, You See, CHI Extended Abstracts 2025 — interface design and performance-based exploration.
• Practice-based research paper, AIMC 2024.
• Negotiating Human–AI Co-Creativity in Music with In-Process Visualisation — manuscript reporting the 16-musician study.
• Watch the performance recording above.

Back to all projects