Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes
Speculative decoding in hybrid models becomes inefficient at large batch sizes because the Mamba layers require full state copies for each draft token, leading to excessive memory traffic that outweighs the benefits of faster decoding.