vLLM vs SGLang vs TensorRT-LLM: a serving benchmark
Self-reported Llama-3.1-8B results for three LLM inference engines. Published unique-prompt throughput and latency rows cover H100 bf16. FP8 and L40S appear only in derived cost results. The harness and raw data are not publicly available.
RunInfra Blog9 min read
