// HACKER NEWS — CYBERSECURITY
Show HN: I trained a 125M model to autocomplete piano on-device
TL;DR: I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The biggest improvements came from finding the right MIDI representation, cleaning the training data aggressively, and adding DPO post-training.
Almost a year ago, I started tinkering with an idea: connect my MIDI piano to my phone, play something, and have AI autocomplete the song for me. Think GitHub Copilot, but for piano.
It turned out to be a deeper rabbit hole than I expected. Fourteen experiments later, it is finally at a point where I am happy enough with it to write about.
The app, RollTab, is available for free here if you have a MIDI keyboard and an iPhone/iPad. 1
Each audio starts with a short prompt, followed by the model's continuation.
A MIDI file is quite different from an MP3 or other audio formats. Rather than storing recorded sound, it stores music as a sequence of events: a key is pressed at a certain pitch and velocity, a key is released, the sustain pedal changes state, and so on. Other events include switching instruments or changing volume.
These events are often organised into multiple tracks. A pop or game MIDI might have melody, chords, bass, drums, strings, and several synth parts. This project is focused on piano continuation, so I mostly kept piano-like material and removed or reduced the rest.
To train a transformer on these performances, I first needed to turn the MIDI events into a discrete sequence the model could read and predict.
The most obvious mapping is to make a token for every MIDI event:
If you include pitch and velocity directly in a NOTE_ON token, the vocabulary can grow quickly. There are 128 MIDI pitches and 128 velocity values, so the naive combined note-on vocabulary has up to:
tokens just for note-on and note-off. In practice you would probably bucket
velocity, but the basic issue remains: many combinations are rare, and the
model has to learn a lot of structure from sparse tokens.