AI Audio to MIDI: how we trained a model on synthetic data with checked labels
We built the MIDI extractor, a tool in the Beatloo ecosystem that turns a recorded guitar, piano or synth part into MIDI. What sets it apart is the data: the base of training is 160 hours of synthetic audio rendered with 2,271 presets of a software synthesizer, where every labeled note was checked against the sound. We then fine-tuned the big model on 85 hours of live guitar and piano recordings.
Inside are two models: Midify Mini, with 16,796 parameters, modeled on Spotify’s Basic Pitch (16,782), and Midify, with 4.1 million. On synthetic audio with unseen presets Midify scores a note F1 of 0.848 against 0.557 for Basic Pitch; on live guitar from GuitarSet, 0.918 against 0.771; on live piano from MAESTRO, 0.855 against 0.649. The comparison is uneven both ways: Basic Pitch was trained on GuitarSet, and we tuned thresholds only for our own models (section 1.2).
Below: how we picked 2,856 presets out of 10,052, how the models are built, and which experiments didn’t work.
Some results surprised us. Most of the model’s mistakes are not wrong pitches: it adds a ghost harmonic, a note an octave above or below the one played. Live recordings improved note durations on live guitar more than everything else combined: F1 with offsets rose from 0.55 to 0.78. All the audio augmentations together added less than 0.01 F1. And the model itself disproved our check for “inaudible” repeated notes.
This post is for musicians who transcribe parts from recordings, and for anyone deciding whether synthetic data is enough to train transcription. Spoiler: it is enough for the base, and a little live audio gets the durations right on real recordings.
Results and comparison with Basic Pitch
1.1 Overall results
Basic Pitch is Spotify’s latest public model for turning audio into MIDI; its weights haven’t changed since it was published at ICASSP. The metric is onset note F1 from mir_eval: a tolerance of ±50 ms on the note start and ±50 cents on pitch. We measure it on four sets: synthetic audio with unseen presets and melodies (1,267 examples), pop melodies from a MIDI pack (section 3.1) on unseen presets (715), live guitar: GuitarSet, guitarist 05 (60 recordings), and live piano: 40 performances from the MAESTRO test split (5 h). Figure 1 shows the average over the four sets. Basic Pitch runs with its default thresholds. Midify Mini’s thresholds were tuned on the synthetic validation set, and for live recordings on GuitarSet guitarist 04 and MAESTRO validation. Midify uses one set of thresholds and post-processing rules for every set, tuned on the same guitarist 04 and MAESTRO validation.
Midify averages 0.868 note F1, Spotify’s Basic Pitch 0.622
Average note F1 over four sets: synthetic, pop melodies, live guitar, live piano; scale from 0 to 1
* Includes GuitarSet, which Basic Pitch was trained on, so probably too high
The mini model nearly matches Basic Pitch in architecture and parameter count, so its lead (0.17 on synthetic audio, about 0.10 on guitar and 0.07 on piano) is mostly the effect of the data. The big model adds another 0.12 on synthetic audio, 0.05 on live guitar and 0.14 on live piano. On pop melodies Basic Pitch writes many extra notes (precision 0.427 with recall 0.634); on synthetic audio it misses and adds notes about equally (0.545 and 0.569).
Figure 2 breaks the results down by instrument category. The synthetic categories count every note of the first two sets, 285 unseen presets in all; live guitar and live piano have rows of their own.
Midify leads in every category; pads and distorted guitar are the weakest
Onset note F1, 50 ms tolerance; synthesizer categories on unseen presets
* Probably too high: Basic Pitch was trained on GuitarSet. The test has only 6 distorted guitar presets
Midify leads in every category. Its lead over Basic Pitch is largest on strings and winds, pads and bass (0.35–0.37) and smallest on live guitar (0.15) and live piano (0.21). All three models share the same weak spots: pads and distorted guitar (section 1.3).
1.2 Caveats
Basic Pitch was trained on GuitarSet, MAESTRO, Slakh, iKala and MedleyDB, and GuitarSet wasn’t split by guitarist, so it has probably heard guitarist 05’s recordings. Midify was fine-tuned on GuitarSet guitarists 00–03 and 500 performances from the MAESTRO training split; guitarist 05 and the MAESTRO test performances were never used to train any of our models. On synthetic audio the domain helps us: the test presets are unseen, but come from the same synthesizer. Tuned thresholds matter for the mini model: with thresholds from synthetic audio it gets 0.847 on GuitarSet instead of 0.870. Midify with the thresholds from training scores almost the same as with tuned ones: 0.914 on GuitarSet and 0.854 on MAESTRO. MAESTRO labels hold a note while the sustain pedal is down, as is standard in piano transcription work, so F1 with offsets on piano is low for every model.
1.3 Where the models go wrong
On synthetic audio Midify is weakest on distorted guitar (0.71) and pads, presets with a slow attack (0.73); see figure 2. Pads are also the largest group in the test. With distortion there are many extra notes, apparently intermodulation frequencies of the chord’s notes. On live piano Midify misses 22% of the notes, and nine in ten of those misses are chords of four or more notes and repeated notes merged with the previous one. On live recordings 6–8% of the output notes are extra, and ghost harmonics make up 2–3% of the output; what we did about them is in sections 4.6–4.8.
Models and training
Both models take the same input: a CQT (a spectrum with a logarithmic frequency scale) of 22,050 Hz audio from the note A0 up, 3 bins per semitone, with a hop of about 11.6 ms. Then harmonic stacking: copies of the spectrum are shifted so that harmonics 0.5, 1, 2, …, 7 of each note line up with its fundamental, and the overtones of one note fall under one convolution.
Midify Mini is a fully convolutional network with outputs for onsets (88 keys), frame-by-frame note activity (88) and the pitch contour (264). Midify is a CRNN of four convolutional blocks (48–96 channels) and a bidirectional GRU with onset, note activity, offset and velocity outputs for 88 keys, 4.1 million parameters in all. The decoder opens a note at an onset peak above a threshold and holds it while the activity stays above a second threshold. Midify adds two rules: a weak note an octave, a twelfth or two octaves above a more confident one is dropped, and keys from C6 up get their own onset threshold. The thresholds and rules are found by a grid search on the dev sets.
Table 1. The models; speed is how many times faster than real time on a CPU
| Midify Mini | Midify | |
|---|---|---|
| Parameters | 16,796 | 4,107,728 |
| File size | 84 KB | 16 MB |
| Speed on CPU | ~160× | ~60–90× |
Before a long run, the model is overfit on a single batch: if F1 doesn’t get close to 1, we look for a bug in the data, not in the hyperparameters (now 0.994 in 300 steps). We train on random 2–4 s windows: BCE for onsets (with a higher weight), note activity and offsets, MSE for velocity in onset frames; AdamW, a cosine learning rate, bf16. Midify trains in four stages. First, 150 thousand steps from scratch on synthetic audio, about 5.5 hours on one RTX 5080, with the checkpoint picked by note F1 on the validation set’s unseen presets. Then 30 thousand fine-tuning steps where 60% of the windows are synthetic and 20% each are live guitar and live piano (section 3.4). Then another 40 thousand steps focused on the weak spots: 30% of the windows are live piano and 10% are pads (section 4.7). The last stage is another 40 thousand steps with melodies, chord progressions and 808s from a beatmaker archive added to the synthetic audio (section 4.8). The fine-tuning checkpoints are picked on GuitarSet guitarist 04.
Augmentations are applied on the fly, to synthetic audio and live recordings alike: reverb, delay, distortion, chorus, phaser, EQ, compression, noise and 50/60 Hz hum, a high cut and mp3 compression. Delay echoes aren’t labeled: the model writes out only the notes that were played.
Data: melodies, presets, labels
Below, a clip is a 10–30 s piece of MIDI, and an example is a clip rendered with one preset, together with its labels. A synthesizer doesn’t always play what it was sent: a preset may sound an octave higher or lower, “swallow” a note when it runs out of voices, delay the attack, or go silent before the note off. So we move the labels into the preset’s range and correct them for all of this, then check every note for audibility; examples where even one note can’t be heard are thrown out, 8–14% of them.
3.1 Melodies
The generator produced 20,000 clips in two styles:
- random, 30%: random pitches, durations from 30 ms to 4 s, 1–8 notes at once, onsets off the grid; this keeps the model from memorizing patterns instead of listening;
- musical, 70%: progressions with inversions, arpeggios, a melody over chords, bass, repeated notes, trills, staccato, rests.
Humanization: tempo 50–200 BPM, swing, timing shifts of ±10–20 ms, varied velocity. The guitar profile plays on a virtual fretboard: 6 strings, frets 0–19, only playable fingerings, one note per string. Techniques: strums with 10–40 ms between strings, fingerpicking, muted strokes.
The second source is a pack of real MIDI melodies: 2,755 files, 2,102 clips, about 8.7 hours. Both the generated clips and the pack’s melodies are split into training, validation and test; copies of one melody, even transposed, always land in the same part.
The third source is a beatmaker archive: 65,535 MIDI files of melodies, chords and drums. The selection dropped drums (by channel, by file name, such as hi-hats, kicks, claps and snares, and by note pattern, such as hi-hat rolls: one note in a narrow register with fast hits), parts on one or two notes, single chords and scales, and exact and transposed duplicates. That left 8,889 files; more than half of them are chord progressions, which we thinned out by half, so 6,220 went into training. Short loops of 1–8 bars are repeated up to 20 s, giving 6,709 clips, about 35 hours.
3.2 Presets
A bot clicked through the preset browser of a software synthesizer and captured 10,052 presets. Each one takes an exam:
- polyphony and legato: it plays chords and doesn’t merge notes;
- voice count: how many notes sound at once without dropping any;
- tuning and octave: the pitch shift measured with pYIN, an estimate of the fundamental frequency f0;
- range, latency, decay: corrections for the labels;
- hidden layers: energy at 0.5·f0 and 1.5·f0;
- junk: a long note gives exactly one onset;
- velocity: whether loudness depends on how hard a note is struck;
- duplicates: by spectral fingerprints.
After what we found in section 3.3 we later widened the layer check (0.33·f0 and 0.25·f0) and added an echo check: repeats in the tail of a short note at the same delays for two different notes.
Of 10,052 presets, 2,856 are fit for training: 28%
Presets left after each check; in brackets, the share of those captured; on the right, how many were dropped
The main exam drops about 70% of the presets. 227 presets had an echo, but some of them had already been dropped, so the last step removes 214. The split is by preset: 2,284 for training, 284 for validation, 288 for testing.
3.3 Rendering and alignment checks
Rendering runs in 12 processes, 70–95 times faster than real time in total. More than 200 hours have been rendered, 160 of them used for training: 27,295 examples on 2,271 of the 2,284 training presets.
For 20 random examples from each render batch we lay the labels’ piano roll over the CQT spectrogram; AI agents review the images, and we look at some ourselves. That is how we found a preset with a hidden layer two octaves below the note and 15–19 dB quieter: the sound is there, but the note is not in the labels. We also found a saxophone with an echo about every 0.5 s.
3.4 Live recordings
To fine-tune Midify we took two open datasets with accurate labels. GuitarSet is acoustic guitar labeled from a hexaphonic pickup: guitarists 00–03 (2 h) go to training, 04 to threshold tuning, and 05 only to the test. MAESTRO v3 is performances on a Disklavier grand piano, where the instrument itself records the MIDI. From the 108 GB archive we pulled 555 performances (90 h) over HTTP Range, following the official split: 500 for training, 15 for tuning, 40 for the test; a piece of music always stays in one part. We hold a MAESTRO note while the sustain pedal is down, because the sound really does ring on.
Experiments: what worked and what didn’t
We compared generations of the big model on a fixed test: the test presets among the first 815 that passed the exam. That is why its numbers differ in the third decimal place from the synthetic test in section 1.
Midify by version: v1 0.785 → v2 0.815 → v3 0.833 → v4 0.840 with 34 → 72 → 112 → 178 h of training audio. Midify Mini gained 0.002 (0.725 → 0.727) as the data grew from 34 to 112 h: 17 thousand parameters can’t take in more data. The big architecture on the same data gave +0.060, and the next +0.055 came from three steps in data with the same architecture. In the “presets and augmentations” step (+0.018), nearly all of the gain came from the new presets (section 4.2). Next, v5, an architecture one and a half times wider trained from scratch, scores 0.851; v6, which is v5 fine-tuned on live recordings, scores 0.844 on this synthetic test but gains +0.02 on live guitar and +0.07 on live piano (section 4.5); v7, fine-tuning focused on piano and pads, adds another +0.025 on live piano with no losses elsewhere (section 4.7); v8, fine-tuning with a beatmaker archive, adds another +0.006 on live piano and +0.003 on pop melodies (section 4.8).
4.1 The model memorized clips
The first big model scored 0.875 on training clips and 0.785 on the test. It handled new clips on training presets almost as badly as unseen presets: about 0.08 below the training clips. Our hypothesis: the model memorizes clips, having seen each one about 24 times. We added 15,000 clips (they are part of the 20,000 in section 3.1): the test rose to 0.815 and the gap shrank to 0.05.
4.2 Audio augmentations
Real recordings come with reverb, distortion and mp3 compression. We expected a gain on live guitar and trained the same model with the augmentations from section 2 and without them. The gain: +0.006 on synthetic audio and +0.008 on GuitarSet. Not what we expected, but there is an explanation: GuitarSet is recorded clean and doesn’t test what the augmentations protect against. We kept the augmentations.
4.3 Real melodies from a MIDI pack
The generator doesn’t know how real songs are built: density, rhythm, voice leading. Hypothesis: real melodies in training would raise the quality on pop melodies. The pack from section 3.1 gave +0.031 on pop melodies, and the main synthetic test didn’t drop (v3 → v4: 0.833 → 0.840). A caveat: the test and training pop melodies come from the same pack, though they share no melodies, so part of the gain is in-domain. We kept the pack in training.
4.4 Repeated notes in pads: an experiment that didn’t work
Two thirds of the notes in pads are repeated notes of the same pitch, played while the previous note is still in its release; the model misses about half of them. Hypothesis: the synthesizer doesn’t give some of them a new attack, so the labels lie. We built a check: a repeated note with no rise in energy at its onset is removed from the audio and the labels, and the example is rendered again. In unit tests on artificial signals the check worked, but on the dataset’s renders it marked 55% of repeated notes as inaudible, even on piano.
We tested the check against the model: the model finds 68% of the “inaudible” notes (95% for bass and organ) and 83% of the “audible” ones. The check barely tells the two groups apart, so we turned it off rather than delete about a million correct notes. The argument isn’t airtight: the model could have guessed the repeats from the rhythm.
4.5 Wider, from scratch, and with live recordings
We trained two versions from scratch on all the data: the previous architecture (2.6 million parameters) and one one and a half times wider (4.1 million). At equal steps the wide one stayed 0.01–0.015 ahead and won every test: synthetic 0.851 against 0.843, pop melodies 0.840 against 0.830, live guitar 0.888 against 0.874. Then we fine-tuned the wide model for 30 thousand steps on a mix with live recordings. Synthetic dropped by 0.008, live guitar rose to 0.910, live piano from 0.763 to 0.834, and F1 with offsets on guitar from 0.55 to 0.76: the live recordings taught the model where a note really ends.
4.6 Ghost harmonics
Breaking down the extra notes showed that the model’s most common mistake is not a wrong pitch but an added harmonic: a note an octave, a twelfth or two octaves away from the one played. For the previous Midify these were 5% of all output notes on synthetic audio, mostly an octave up (an overtone), and 7–8% on live guitar, mostly down (a subharmonic). We tried two remedies. The first is a loss penalty: silent keys at these intervals from a sounding note weigh four times as much. Harmonics dropped by a third, but misses on piano grew and the average F1 didn’t change, so the model without the penalty went into the release. The second is a decoder rule: a weak note above a more confident one at a harmonic interval is dropped. Together with the wide model and the live recordings, ghost harmonics on live guitar fell from 5.4% to 1.9% of the output notes.
4.7 More piano and pads
After the first fine-tune, the weak spots were still live piano (misses in dense chords and repeated notes merged into one) and pads. We downloaded more of MAESTRO, up to 500 training performances (83 h instead of 25), raised piano’s share of the training windows from 20 to 30%, and added a separate stream of pads, 10% of the windows; the test and validation performances stayed the same. Another 40 thousand fine-tuning steps gave +0.025 on live piano (0.824 → 0.849), +0.007 on live guitar and +0.004–0.008 on synthetic audio and pop melodies; pads rose from 0.717 to 0.729. There are fewer wrong notes everywhere: on live piano their share fell from 9.7% to 6.3% of the output notes, on live guitar from 10.4% to 8.9%.
4.8 Beatmaker melodies
People most often convert parts from beats into MIDI: melodies, chords and 808s. The generator and the MIDI pack have few such parts, so we took a beatmaker archive (section 3.1), rendered 41 hours of it with the training presets (a fifth of all the synthetic audio) and fine-tuned the model for another 40 thousand steps. We held out the archive's test part: 578 examples on unseen presets. On it F1 rose from 0.842 to 0.853; on pop melodies from 0.846 to 0.849, on live piano from 0.849 to 0.855, on live guitar from 0.916 to 0.918; synthetic audio didn't change. There are fewer extra notes again: 5.6% of the output notes on live piano instead of 6.3%, 8.3% on live guitar instead of 8.9%.
How to use it
Drag an audio file (mp3, wav, flac, ogg or m4a) into the MIDI extractor window, and drag the finished MIDI from the window into your DAW. The threshold sliders rebuild the notes without running the model again, so you can tune the note density by ear.
Midify transcribes a three-minute track in 2–3 s on a CPU. Midify Mini is 84 KB and takes about 1 s.
If there are several instruments, turn on guitar or piano separation: an open stem separation model pulls out that track. It separates piano with some bleed, and the app warns you about it.
Try it yourself
You can turn audio into MIDI right in your browser, with nothing to install. Drop in an MP3 or WAV and get MIDI with tempo and key, ready for any DAW. The file is used only for the transcription: the Beatloo server deletes it as soon as it has been read.