
SpecInfer accelerates large language model serving by leveraging tree-based speculative inference and verification, which significantly reduces memory accesses to the LLM's parameters and end-to-end inference latency while preserving the same generative performance as incremental decoding.


coremsg

specinfer-accelerating-large-language-model-serving-with-tree-based-speculative-inference-and-verification


SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification


title_rewrite


AttentionStore, a new attention mechanism, enables the reuse of key-value caches across multi-turn conversations, significantly reducing the repetitive computation overheads and improving the inference performance and cost-efficiency of large language models.


efficient-attention-reuse-for-cost-effective-multi-turn-conversation-inference-in-large-language-models


Efficient Attention Reuse for Cost-effective Multi-turn Conversation Inference in Large Language Models
