Can gzip be a language model?(nathan.rs)
398 points by networked 22 days ago | 161 comments
tl;dr: Building on the "compression is prediction" equivalence, the author uses gzip (via zlib's DEFLATE) as a language model by scoring candidate continuations by their compressed length against a corpus-primed context. Naive next-byte selection fails due to integer-length quantization, so a beam search over multi-byte spans is used, with only a recent tail retained to avoid verbatim loops. Primed on tiny Shakespeare, it produces recognizably Shakespeare-ish (if incoherent) output—no weights, no training, pure standard-library Python.
HN Discussion:
  • ↑Compression-based classification/prediction has established precedent that supports the article's premise
  • ↑Compression and next-token prediction are fundamentally related concepts, as the article suggests
  • ~The search space limitation means results only show a lower bound of gzip's capability
  • ~Cautions against overstating equivalence between simple compression models and large neural networks
  • •Curious about extending the idea to other domains like video compression or reverse direction (LLMs as compressors)