Stem Splitter · Research

How we teach a computer to hear every instrument in a song

Beatloo · 7 min read

Drop a finished song into Beatloo’s Stem Splitter and you get it back in pieces: vocals, drums, bass, guitar, piano and everything else. This article explains how we train the AI model behind it — without code and without math.

The short version: a model is only as good as the examples it learns from. So instead of scraping music off the internet, we build our own training songs, where we know exactly what every instrument plays.

It’s the same idea behind our MIDI extractor. At the end you’ll also find an honest list of what the model still finds hard.

What a stem splitter does

A finished song is a bit like a smoothie. Once the fruit is blended, you can’t just pick the strawberries back out. A stem splitter tries to do exactly that: listen to the mix and rebuild each instrument on its own track, called a stem.

Beatloo splits a song into up to six stems:

Vocals — lead and backing vocals together;
Drums — kicks, snares, hats, cymbals and percussion;
Bass — bass guitar, 808s and synth bass;
Guitar — acoustic, clean and distorted;
Piano — acoustic and electric pianos;
Other — synths, pads, strings, brass and effects.

It listens first and returns only the stems it actually finds. A song without a guitar won’t come back with an empty guitar track.

Producers use stems to remix and flip samples, to learn a part by soloing it, to make acapellas and instrumentals, and to get cleaner MIDI: one instrument on its own converts much better in Audio to MIDI than a full mix.

One mix in, a separate track for each instrument out

The splitter hears the whole song and rebuilds every instrument it finds

Figure 1. Illustration. The splitter returns only the instruments it finds in the song.

How a model learns

An AI model doesn’t come with rules about what a snare or a bass guitar sounds like. It learns from examples, a little like training your ear: hear a mix, guess what the drums alone would sound like, compare the guess with the right answer, adjust. Then repeat that millions of times.

Every training example has two parts: the full mix — the question — and every instrument recorded separately — the answer key. The better and more varied the answer keys, the better the model gets.

A picture of sound

The model doesn’t listen to sound the way we do. It looks at a picture of it called a spectrogram: time runs from left to right, pitch from bottom to top, and brightness shows how loud each pitch is at each moment.

For each instrument, the model learns to draw a stencil over that picture: these bright spots belong to the bass, those short bursts to the hi-hats. Cut the picture along a stencil and turn it back into sound — that’s a stem.

The model learns a stencil for each instrument

Left: the mix as a picture of sound. Right: the parts the model keeps for the bass and for the drums

Mix

Bass stencil

Drums stencil

Figure 2. Illustration. Time runs left to right, pitch bottom to top. The bass lives in long low notes; drums are short bursts across many pitches.

Where our training songs come from

To learn, the model needs songs where every instrument is available on its own. Real multitrack recordings are rare and expensive, and most of them aren’t licensed for training. The best-known research collection, MUSDB18, has 150 songs — so few that a big model starts memorizing them instead of learning to listen.

So we make our own songs. Because we create every part ourselves, we always have a perfect answer key:

01Write the partsDrum grooves, bass lines, chords and melodies in many genres, tempos and keys.

02Play each partEvery part is played on its own by a real-sounding instrument from our checked library.

03Check itEach part has to pass automatic checks before it can be used.

04Mix itLevels, panning, EQ, compression, reverb and distortion, chosen at random.

05TrainThe model hears the mix and is graded against the clean parts.

Figure 3. How one training song is made.

Real-sounding instruments

The parts are played by the same library of checked instrument sounds we built for our MIDI extractor: thousands of synthesizer presets, each one tested so that what it plays matches the notes it was given. The MIDI extractor article explains how we picked 2,856 of them out of 10,052.

Mixed like an engineer would

A clean demo and a loud, dense master sound very different, and the splitter has to handle both. So every song is mixed with random settings: some parts loud, some quiet, panned left or right, with more or less reverb, compression and distortion. Whatever happens to the mix, we keep the clean version of each part as the answer.

Vocals

Synthesizers can’t sing, so vocals are the one stem we can’t generate. For vocals we use [licensed recordings of real singers — describe the source], mixed over our generated backing tracks the same way.

The result: every training song comes with a perfect answer key, and every sound in it is ours or licensed. No music is scraped from the internet.

Checking every stem

A wrong answer key teaches the wrong lesson. If the “bass” answer secretly contains a bit of kick drum, the model learns that kicks belong in the bass. So before a song goes into training, every part is checked automatically:

every note in the part can actually be heard;
no part is silent or clipped;
echoes and reverb tails stay with their own instrument;
the same song doesn’t appear twice, even transposed;
levels stay within a realistic range.

We also look and listen ourselves. For random songs from every batch we line up the picture of the mix with the pictures of the stems and check that nothing is missing or in the wrong place. Songs that fail a check are thrown away rather than fixed by hand.

How we know it works

We test the splitter on songs it has never heard: new arrangements, new instrument sounds, and real recordings that were never part of training.

The main score is measured in decibels and is called SDR. In plain words, it shows how much louder the instrument you asked for is than the leftovers of everything else. Higher is better: every 3 dB more means the leftovers are about half as loud.

Numbers don’t tell you how a stem sounds, so we also run blind listening tests with producers: they hear two versions of the same stem and pick the cleaner one, without knowing which model made which.

We’ll publish the full results, including a comparison with open-source models, when Stem Splitter launches.

Where it still struggles

Some songs are hard for every stem splitter, ours included:

Lots of reverb. The sound of the room smears across all instruments, so a little of it ends up in every stem.
Instruments in the same range. Rhythm guitar and piano chords, or pads and strings, can trade notes between stems.
Heavy distortion. A distorted guitar adds overtones that sound like other instruments.
Backing vocals and harmonies. They stay in the vocals stem together with the lead.
Low-quality files. Heavily compressed MP3s add artifacts the model can’t undo.

These are exactly the cases we now generate more of in training.

What’s next

Stem Splitter is coming to Beatloo Tools. You drop a song, it returns the stems it finds, and you can send any stem straight to Audio to MIDI.

Until then, try AI Audio to MIDI: drop any MP3 and get its notes back as MIDI, with tempo and key.

Try Audio to MIDI More from the blog