Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
Apple's MLX team shows how to run frontier-class LLMs locally on Mac Studio, scaling a nearly trillion-parameter MoE model across four M5 Ultra machines via RDMA over Thunderbolt to reach over 36 tokens per second.