PolyGraph: Plan-Buffered CUDA Graphs for Adaptive Speculative Decoding
Adaptive speculative decoding changes the active batch size B and draft length d from one decode step to the next, whereas a CUDA Graph replays a fixed launch configuration.
arXiv cs.LG2 min read