hiw3c 日报精选
arXiv cs.AI/CL/LG 网页快照

TideCache:高效混合LLM服务,具备透明缓存管理与层级流式传输功能

原文标题 · TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

作者: Jinbin Luo, Xiaozhe Yao, Shang Yang | 混合架构在开放权重语言模型中已成为主流。其全局注意力层与有界状态层维持异构缓存,这些缓存的需求随上下文长度与并发程度而变化。我们提出TideCache,一种基于CPU--GPU超芯片的服务系统,包含三个组件:用于在缓存之间分配GPU内存以及管理其在GPU与主机内存中的驻留机制。

来源:arXiv cs.AI/CL/LG · arxiv.org

正文为英文,可一键机器翻译(仅首次需要等待)

Computer Science > Distributed, Parallel, and Cluster Computing

Title: TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

Abstract: Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.

Submission history

Access Paper:

Additional Features

Current browse context:

References & Citations

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .