Optimizing LLMs: Comparing vLLM, LMDeploy, and SGLang | Clarifai Blog
Large Language Models (LLMs) are at the forefront of AI innovation, offering remarkable capabilities in natural language processing tasks. However, their impressive performance comes with a significant trade-off: inference efficiency, which impacts both cost and time for model owners and users. To address these challenges, extensive research has focused on optimizing caching techniques, memory allocation, GPU kernel performance, and more. Among open-source solutions, frameworks like vLLM, LMDeploy, and SGLang stand out, delivering exceptional performance compared to others. In this blog, we will explore the foundations of these frameworks, provide sample code, and compare their performance.
Background
The attention algorithm lies at the heart of the remarkable capabilities of LLMs, revolutionizing natural language processing by addressing the limitations of earlier sequential techniques like RNNs and LSTMs. These older methods struggled with handling long contexts, were slow to train, and lacked scalability. Attention effectively overcomes these challenges.
However, as the saying goes, "Life is essentially an endless series of problems. The solution to one problem is merely the creation of another.” quoted from this book. While attention offers significant advantages, it also introduces new considerations, such as increased computational demands. The algorithm requires extensive matrix calculations and caching of processed tensors for the decoding step, which can lead to slower inference times.
Solutions
Common approaches to improve LLM efficiency include running models with lower precision formats, such as FP16 or even more compact formats like INT8 or 4-bit quantization, instead of the standard FP32, and utilizing more powerful hardware. However, these methods do not fundamentally address the inherent inefficiencies of the algorithm itself.
A more effective alternative focuses on optimizing one of the core bottlenecks: the KV cache in LLMs. Key strategies include:
- Smarter Cache Management: Efficiently manage caching across batched requests to minimize memory waste.
- Optimized Memory Allocation: Structure memory usage to store more data within limited memory capacity.
- Enhanced Processing Efficiency: If memory is not the constraint, leverage system resources to accelerate processing.
- Optimized Kernel Implementations: Replace naive Torch implementations with robust, inference-optimized kernels.
The Frameworks
A key pioneer in addressing LLM inefficiency is vLLM, followed by LMDeploy and SGLang. While these frameworks share common foundational ideas to tackle inefficiencies in LLMs, each employs distinct, customized methods to achieve its goals.
vLLM
vLLM optimizes LLMs by enhancing memory efficiency and enabling parallel computation. It reduces the overhead associated with large-scale model inference, allowing for faster processing and better resource utilization without compromising accuracy.
LMDeploy
LMDeploy focuses on simplifying the deployment process of LLMs at scale. It integrates model parallelism and fine-tuning techniques, improving the speed and scalability of deploying models for real-world applications, particularly in distributed settings.
SGLang
SGLang leverages structured programming techniques to optimize LLMs by focusing on efficient resource management and computation. It introduces specialized language abstractions and tools for fine-grained control over model execution, leading to enhanced performance in specific tasks or environments.
The table below provides an overview of vLLM, LMDeploy, and SGLang, including their specifications, supported architectures, and GPU compatibility.
| Framework | Specs | Supported architectures | Supported GPU |
|---|---|---|---|
| LMDeploy | Delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch (a.k.a. continuous batching), blocked KV cache, dynamic split & fuse, tensor parallelism, high-performance CUDA kernels, and more. Quantizations: Supports weight-only and k/v quantization; 4-bit inference performance is 2.4x higher than FP16. | Transformers, Multimodal LLMs, Mixture-of-Expert LLMs | Nvidia |
| vLLM | A fast library for LLM inference and serving: Cached PagedAttention, continuous batching, distributed inference, fast model execution with CUDA/HIP graph. Quantizations: GPTQ, AWQ, INT4, INT8, and FP8. | Transformers, Multimodal LLMs, Mixture-of-Expert LLMs | NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron. |
| SGLang | Builds upon open-source LLM engines like LightLLM, vLLM, and Guidance, incorporating high-performance CUDA kernels. Introduces innovations like RadixAttention for KV cache reuse and a compressed state machine for fast constrained decoding. | Almost all transformer-based models | Nvidia, AMD (recently supported) |
Benchmark
Environment setup
Hardware CPU RAM (GB) GPU VRAM (GB) AMD EPYC 7J13 64-Core Processor 216 A100-SXM4 40 Metrics: The benchmarking was conducted using the open-source test framework llmperf with a custom fork llmperf multimodal to enable testing of multimodal models.
Test config: Input tokens (i): 2048, Output tokens (o): 2048, Number of concurrent requests: 1 and 100.
Results
- Single request:
With one request at a time, SGLang handles best in terms of ttfs, 22.3% faster than the slowest (lmdeploy-pytorch). On the other hand, lmdeploy-turbomind outperforms the rest with 88.6 tok/s on average and 8.12% better than the worst one (vllm).
100 requests:
For TTFS, SGLang performs exceptionally well for 2 out of 3 models but falls significantly short for Mistralv0.3, even after multiple retests yielding consistent results. This suggests the framework is not well-optimized for the Mistral architecture.Throughput per second is led by lmdeploy-turbomind, outperforming the worst-performing framework by over 20%.
Conclusion
In this blog, we have benchmarked various models using different inference frameworks. SGLang demonstrates strong performance in handling single requests efficiently. However, its optimization appears architecture-specific, as it struggles with the Mistral model under concurrent load. Meanwhile, lmdeploy-turbomind consistently leads in throughput across both single and concurrent request scenarios, proving to be the most robust framework overall.