CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters
Speculative decoding accelerates large language model inference by drafting future tokens cheaply and verifying them with the target model in parallel. Block drafters score a whole block of future tokens in one forward pass, yet standard decoding verifies only the top-scoring chain and discards the…
arXiv - cs.CL (NLP)2 min read