Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang， OpenAI
OpenAI inference engineers Lu Zhang and Qianru Lao recount how IRB evolved from a feedback-loop load balancer into a control-plane and data-plane design whose global optimizer assigns routing weights to minimize end-to-end latency under capacity, health, and cache constraints, with penalties, capped retries, and load shedding for production stability.