Mesh LLM: distributed AI computing on iroh

	Mesh LLM: distributed AI computing on iroh(iroh.computer)
	344 points by tionis 60 days ago \| 92 comments
	tl;dr: Mesh LLM pools GPUs across multiple machines into a single OpenAI-compatible API endpoint (localhost:9337/v1), letting requests run locally, route to peers, or split large models pipeline-style across nodes by layer ranges. It's built on iroh for authenticated, NAT-traversing QUIC connections between nodes identified by public key, with no central server and a custom gossip layer over ALPN-negotiated streams. The 18MB client ships with 40+ models ranging from laptop-sized to 235B MoE, aimed at teams wanting to avoid API lock-in and use hardware they already own.
	HN Discussion: ↑Ease of setup and joining the mesh worked flawlessly on first try ↓Network bandwidth makes distributed inference too slow for practical interactive use ~Most users lack the hardware described in examples to meaningfully participate ~Interest in applying this to smaller purpose-built models rather than large LLMs ~Installation and compatibility have rough edges with older hardware