Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

	Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU(github.com)
	475 points by joshcsimmons 3 days ago \| 205 comments
	tl;dr: llama-modchip is a set of ggml/llama.cpp patches, custom quantization formats (Fractal-64 at 1.71-3.75 bpw), and CPU kernel optimizations that fit DeepSeek-V4-Flash 284B (13.17B active MoE) into 64GB of RAM with no GPU, achieving ~44 tok/s prefill and 9-15 tok/s decode on a 16-core Zen 5. It also supports Nemotron-3.5-30B and Qwen3-30B across x86_64 (AVX-512/AVX2), ARM64, and Apple Silicon, with hybrid GPU offload and speculative decoding via a Stockfish-style n-gram drafter. Note: the custom GGUF tensor type (ID 40) requires the patched binary—stock llama.cpp, Ollama, and LM Studio won't load these files.
	HN Discussion: ↑Local AI is becoming increasingly accessible and affordable even as hardware prices rise ↓Aggressive sub-4-bit quantization significantly degrades model quality and shouldn't be trusted ↓The README is poorly written AI-generated word salad that obscures key details like quantization level ↓Skepticism that the repo actually delivers meaningful improvements over stock llama.cpp ↑Hardware scarcity drives creative optimization work like this, which benefits the ecosystem