I trained a small transformer in 1.5hrs and it beats many LLMs

	I trained a small transformer in 1.5hrs and it beats many LLMs(mvakde.github.io)
	651 points by porridgeraisin 9 days ago \| 162 comments
	tl;dr: A researcher trained a small transformer from scratch on a 5090 GPU in 1.5 hours for $0.67, scoring 45% on ARC-1 and 7% on ARC-2, matching TRM/HRM without recursion. Key improvements over the prior version include modern architecture (SwiGLU, RMSNorm), NorMuon optimizer, 3D RoPE with per-task embeddings, and supervised training on output tokens only. The author argues recursion isn't necessary, criticizes reliance on synthetic data and augmentations in other ARC approaches, and believes 65% is achievable within the transformer framework.
	HN Discussion: ↑Author clarifies the work is a small transformer, not an LLM, demonstrating complex problems can be tackled without LLMs ↓The improvements are 'squeezing the lemon' tactics that should come after achieving near-SoTA with the novel method •Requests clarification on the training-on-eval defense for less informed readers ↑Praises author's achievement and suggests it will lead to career opportunities ↓Raises concern about input leakage during pretraining as a flaw in the approach