Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText
Google introduces Streaming DiLoCo in MaxText, a distributed optimization algorithm that enables efficient LLM training across geographically distributed datacenters by replacing massive periodic weight synchronizations with a continuous stream of parameter fragments.