Proudly brought to you by SerpAPI: Today’s release covers a LOT of ground, but you can scrape all this into markdown for your agents! 50% token cost of JSON.
We last covered the Inference Inflection in April, and historically most of the advances have been closed source decacorns from Fireworks to Baseten to Modal.
Well, time for the open source response. It is day 1 of PyTorch Conf, and DeepSeek timed an intercontinental ballistic missile into the discussion by open sourcing DeepInfer, their production stack, capable of up to 72 hour prompt caching to the 12 hours OpenAI just announced:
This is their customization of vLLM with credit to SGLang and TRRTLLM, but there is a lot added… minus their crown jewels - their ultra optimized KV cache (compact + >8hour lifetime), PD/disggregation, and load balancing (their 3FS was open sourced a while ago). We have translated their blogpost and published it here together with a complete research guide and explainers and codemap and subsystem guide:
As we point out in their additional disclosure, their claim to serve 155T tokens/day gets them $38 per SM100/A950 card per hour at off peak prices (currently $0.15/$0.60 per million input/output tokens: 155000000 tokens * $0.14 blend cost / 24 hours is ~$1m per hour). Taking their production data (which we translated) of 1.5 million cache-inclusive input tokens/second per prefill GPU, 96% input cache hit, and 14,000 output tokens/second per decode GPU (over ~2400 GPUs):
with all-in costs $5-8 per card per hour that is an >80% inference margin!
Doing some cluster math, that is roughly $166B per GW, with 81 day payback period on inference margin (DeepSeek famously plans for 10 month payback). Imagine what the Whale could do with more GWs.
Sponsored by SerpAPI!
Give Your AI Agents Live Search Data with Fewer Tokens
Receive results from any of SerpApi’s 100+ APIs as clean, structured Markdown, using roughly 50% fewer tokens than JSON on average. Simply add output=md or use the /search.md endpoint. It requires zero configuration and is included with every plan at no additional cost.
AI News for 10/19/2026-10/20/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!
AI Twitter Recap
Top Story: DeepSeek DeepInfer—Production Inference, Open-Source Boundaries, and the Economics of Cached Tokens
What happened
DeepSeek released DeepInfer, the inference engine behind V4.1 Flash, alongside production-serving details that make the engineering report as important as the code.
The release covers the inference engine and API-processing layer. DeepInfer supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output on NVIDIA and Huawei Ascend platforms. The accompanying
deepseek-recipecomponent implements production token encoding/decoding and official-API-compatible processing. DeepSeek says future model inference implementations and optimizations will also appear in this repository. @deepseek_ai, @deepseek_ai, announcement translationThe reported production figures are substantial—but describe a heavily cached service. DeepSeek’s engineering report, as summarized and quoted in the supplied posts, reports 155 trillion tokens/day from one cluster, 96% input-cache hits, 400 ms average time to first token, and 230 tokens/s generation, reaching 300 tokens/s for coding. It also reports $38 revenue per accelerator-hour at off-peak prices. These are first-party production claims, not independently audited measurements. @ZhihuFrontier, @swyx
“Fully open source” does not mean the entire production service is reproducible from this release. Although DeepSeek describes DeepInfer as fully open source, the technical account says the KV-cache management and load-balancing components are not yet fully released, because they remain coupled to internal infrastructure. The engine is open; important service-level orchestration remains a deployment responsibility. @deepseek_ai, @ZhihuFrontier
vLLM supplied the most concrete comparative response. After acknowledging that DeepInfer was initially built primarily on vLLM and released under Apache 2.0, the vLLM team reported decode throughput within 6% of DeepInfer’s published results on GB200 under a matched configuration. It also published AgentX results and reproduction recipes, materially qualifying claims that DeepSeek’s efficiency is inaccessible to existing open-source engines. @vllm_project, @vllm_project, reproduction resources
The release triggered two distinct debates: whether DeepSeek has a uniquely strong serving stack, and whether its disclosures demonstrate high inference profitability more generally. The first now has comparative benchmark evidence; the second remains sensitive to workload mix, cache accounting, hardware cost, and which expenses are included. @Tim_Dettmers, @zephyr_z9, @Yuchenj_UW
Architecture: separate stages, schedule compute rather than tokens
DeepInfer’s disclosed architecture separates vision encoding, prefill, and decode, allowing each stage to scale against its own bottlenecks. The recommended deployment shapes are ordinary eight-accelerator machines for prefill and 64–128-accelerator supernodes for decode. Decode uses wide expert parallelism to distribute MoE experts while maintaining large effective batches. These are deployment recommendations, not sufficient evidence of the exact hardware composition of DeepSeek’s production cluster. @ZhihuFrontier
The scheduling details explain why a conventional “tokens per batch” comparison can miss substantial differences:
Prefill scheduling estimates attention-compute cost. Work is allocated using estimated compute rather than raw request length alone. Chunk sizes vary to fit a time budget, instead of enforcing one fixed chunk size across heterogeneous requests. @ZhihuFrontier
Prefill routing does not require cache affinity. Because cached prefixes are pooled in shared storage, a request can go to an available worker rather than the worker that previously served its prefix. In a detailed Chinese-language analysis, inference engineer Su Jin Yan highlights this as a particularly useful production design: prefill data-parallel groups balance on “compute tokens,” while shared storage supplies reusable state. The same analysis highlights request bucketing and FIFO scheduling with a runtime estimator. @SuJinYan123
Decode routing balances both running-request count and KV-cache length. Request count approximates scheduling pressure; accumulated KV length captures memory/state pressure that request count alone misses. The engineering commentary describes prefill/decode scaling signals as aggregate compute load for prefill and running requests for decode. @ZhihuFrontier, @SuJinYan123
Expert-parallel load balancing replicates popular experts. Replicas are periodically redistributed to reduce inter-device waiting caused by routing skew. The trade-off is additional HBM consumption against higher throughput, with diminishing returns as replication increases; the useful replication factor is workload-specific. @ZhihuFrontier, @SuJinYan123
CPU scheduling overlaps accelerator execution. Asynchronous scheduling removes idle gaps between decode steps rather than treating host-side orchestration as free. @ZhihuFrontier
One technically notable observation is the apparent absence of tensor parallelism in the disclosed Blackwell configuration. A commentator contrasts this with default vLLM/SGLang recipes and traces the approach back to DeepSeek V3’s H800-era interconnect constraints. That is an interpretation of the configuration—not evidence that tensor parallelism is generally inferior. @nrehiew_
Scale also matters. Su Jin Yan argues that wide expert-parallel prefill needs enough traffic and workers to pay off, and favors pipeline parallelism for small-to-medium deployments with only a few prefill workers. The author also notes that layer heterogeneity can complicate pipeline partitioning. This is practitioner judgment, not a benchmark establishing a universal crossover. @SuJinYan123
Cache hierarchy: persistent prefixes and cheap reconstruction of transient state
The cache design is the central connection between the throughput figures, agent workloads, and reported revenue per accelerator.
Persistent block KV lives in a separate 3FS storage cluster with roughly 100 PiB of SSD capacity. Reported retention is three days for agent traffic and seven days for chatbot traffic. Any prefill worker can retrieve a cached prefix, decoupling cache locality from compute placement and allowing storage and compute to scale independently. @ZhihuFrontier
That retention is useful for agents that repeatedly resend a growing history of instructions, repository content, tool outputs, and previous responses. It also means 155T served tokens/day must not be r