-
7 October 2026 · 9 min read
Building a Small Transformer That Makes Music
Building a 45M-parameter text-to-music model by hand in PyTorch — EnCodec tokens, cross-attention, home-made pipeline parallelism on two GPUs, and the silent PCIe bug that nearly stopped it. With real training curves and generated audio to judge.