Apple Silicon and macOS VMs: Faster LLM Inference with llama.cpp

	Apple Silicon and macOS VMs: Faster LLM Inference with llama.cpp(github.com)
	302 points by frabonacci 10 days ago \| 43 comments
	tl;dr: Cua's team built a process-scoped Metal capability shim that overrides conservative GPU capability answers reported inside macOS VMs on Apple's Virtualization.framework, allowing llama.cpp to select newer Metal kernels (SIMD-group matrix, bfloat16, etc.). On an M1 Ultra, this yielded 7-16× speedups across TinyLlama 1.1B, Gemma 4 12B, and Muse Glimmer 30B, reaching 94-99% of bare-metal prompt processing speed. The shim is released under a permissive license, but relies on private, version-sensitive Metal behavior and only affects the injected process.
	HN Discussion: ~Title is misleading; speedup only applies to VMs, not llama.cpp generally on Apple Silicon •Questioning why Apple's Virtualization.framework exposes reduced Metal capabilities in the first place •Requesting additional benchmarks on other Apple Silicon chips like M1/M3 Pro •Broader commentary that Apple's hardware enables AI work despite Apple's software/ecosystem neglect ~Confusion about technical terminology used in the article (Apple GPU family numbering)