Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs(github.com)
276 points by Argonautlabs 1 day ago | 152 comments
tl;dr: A fork of the Deltafin engine runs the full unpruned 2.8T-parameter Kimi K3 MoE model on an M5 Max MacBook Pro (128GB) at ~1 token/sec decode, streaming 1.45TB of expert weights from four SSDs. Scaling tests show the slowest drive per layer bottlenecks throughput (one drive gets 52%, three drives 90% of four-drive speed), and prefill is painfully slow—~6.3 minutes to first token on a 512-token prompt—because each layer's experts are re-read 8 times, a known but unfixed bug.
HN Discussion:
  • Impractically slow performance makes this more novelty than usable tool
  • Prefill/prompt loading bottleneck matters more than decode speed reported
  • Modular/sparse model architectures could avoid needing full weights loaded
  • Hardware limitations (Apple RAM, SSD speed) are the real constraint here
  • Achievement is impressive as a starting point for local large-model inference