Skip to main content

Blog

Essays & Engineering Notes

On artificial intelligence, computer science, and software engineering

  1. A Billion-Parameter Music Model: What 23× More Parameters Bought

    I scaled a text-to-music transformer from 45 million to 1 billion parameters, trained it on two GPUs, and measured what the extra parameters actually bought.

  2. Building a Small Transformer That Makes Music

    Building a 45M-parameter text-to-music model by hand in PyTorch — EnCodec tokens, cross-attention, home-made pipeline parallelism on two GPUs, and the silent PCIe bug that nearly stopped it. With real training curves and generated audio to judge.