Progress on provable inference for large language models

	Progress on provable inference for large language models(anthropic.com)
	236 points by ltononro 7 days ago \| 121 comments
	tl;dr: Anthropic researchers released FlashProve, an open-source zkSNARK-based system for provable inference that lets users cryptographically verify a response came from a specific committed model without revealing its weights. On Llama 4 Maverick (400B MoE, 4-bit), a single H100 generates proofs at ~110 tokens/sec with 19MB proofs verifiable in seconds on CPU—roughly 70-95x slower than vLLM inference, but orders of magnitude faster than prior open-source zkML work. The system includes Lean formal verification of soundness, completeness, and zero-knowledge properties, though limitations include 1,024-token context, integer-only greedy decoding, and no architecture hiding.
	HN Discussion: ↓Claims the work builds on their prior research without proper attribution ~Welcomes the advance as meaningful progress but notes cheaper probabilistic alternatives exist ↓Questions practical feasibility given batched inference and heterogeneous hardware reality ↓Argues the proven property isn't very useful and doesn't address real concerns like data exfiltration or quantization ↓Competing practitioners note their solutions are faster and less limited than FlashProve