tl;dr: Dust is a zeroth-order optimization method that trains transformers by perturbing activations independently per token, turning each token into a virtual population member evaluated in a single forward pass. It matches or exceeds backprop on pretraining in several settings and is 1000-10,000x more compute-efficient than weight-space ES methods like EGGROLL. Counterintuitively, larger models (up to 243M params tested) are more population-efficient than smaller ones, and Dust's gradient estimates increasingly align with backprop's as population grows—though it's not yet compute-competitive with backprop at today's scales.