arMIDI is a collaborative augmented reality interface that visualises musicians’ hand positions, eye gaze, and instrument activity. It gives electronic musicians a shared view of their interaction to support awareness and coordination while playing together.
Keywords: Interaction design, spatial visualisation, and functional prototyping.
Electronic instruments often have small controls that are obscured during performance. A musician’s movements may also reveal little about the resulting sound, making it difficult for collaborators to follow changes or anticipate one another’s actions. I explored how AR could make this information visible while musicians continued playing their existing instruments.
Each performer sees a virtual representation of their collaborator’s keyboard through a HoloLens 2 headset. The design combines two layers of information: bodily cues that communicate movement and attention, and instrument data that shows notes and control changes.
Yellow spheres labelled “L” and “R” represent the collaborator’s palm positions. During iterative in-lab development, these offered more stable tracking and less visual clutter than fully rendered hands, which were prone to jitter.
A blue sphere shows gaze position. When the collaborator looks towards the virtual instrument, the marker freezes and enlarges to signal their attention. This approach communicates gaze while limiting distracting movement in the performer’s view.
The keyboard representation highlights active notes and controls in yellow. Animated knobs and faders reflect changes, with pink annotations displaying their current values. Bright colours were chosen for visibility through the headset under office lighting.
I built the interface in Unity and C# with Microsoft MRTK, with 3D modelling in Blender. A QR code anchors the virtual keyboard beside the physical instrument. A Node.js and WebSocket backend exchanges MIDI data between performers’ computers and headsets in real time.
A study with eight musicians in four pairs compared bodily cues, instrument activity, combined feedback, and no visualisation through improvisation, questionnaires, and interviews.
The combined feedback was preferred overall and supported greater self-reported awareness than bodily cues alone or no visualisation. Instrument feedback was considered easiest to understand and respond to, while hand and gaze cues helped some musicians anticipate actions and coordinate roles.
The study also highlighted an attention trade-off: additional information could distract from listening or be difficult to translate into a musical response. This points towards more selective feedback that prioritises meaningful changes and limits unnecessary detail.