My local model setup on an M4 Pro Mac Mini(lws.io)
323 points by raybb 8 days ago | 192 comments
tl;dr: The author runs a local LLM setup on a 48GB M4 Pro Mac mini using Qwen3.6-35B-A3B and Gemma-4-E4B (both 4-bit quantized) via the oMLX inference server, with Tailscale connecting his iPhone and MacBook to access it through clients like Hermes, Apollo, and Raycast. He argues MoE models are key for consumer hardware since only active parameters (3B of 35B) matter for inference speed, achieving ~34 tok/s generation. The motivation: avoiding cloud API pricing volatility, privacy risks, and geopolitical restrictions, while handling ~80% of daily tasks locally at zero marginal cost.
HN Discussion:
  • ~Local LLM setup is still too complex; waiting for turnkey solutions
  • Questions the cost motivation since simple AI tasks are free in the cloud
  • ~Privacy, not cost, should be the primary motivation for local models
  • Sharing performance benchmarks that validate MoE model efficiency claims
  • Prefill latency makes Mac local LLMs inferior to dedicated hardware like DGX Spark