What is vLLM?
vLLM is a fast and easy-to-use library for LLM inference and serving, released under the Apache-2.0 license. It was originally developed in the Sky Computing Lab at UC Berkeley. It handles text-generation models and also multi-modal, embedding, retrieval, reward and classification models from Hugging Face.
The engine manages attention key and value memory with PagedAttention, and schedules requests with continuous batching, chunked prefill and prefix caching. It supports quantization formats such as FP8, NVFP4, INT8, INT4, GPTQ/AWQ and GGUF, speculative decoding, and tensor, pipeline, data, expert and context parallelism for distributed inference. Output features include streaming, structured outputs, tool calling and reasoning parsers.
For applications, vLLM provides an OpenAI-compatible API server, plus Anthropic Messages API and gRPC support. Hardware support covers NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, with plugins for other accelerators. Install it with uv or pip using uv pip install vllm, or build from source.
Key capabilities
- Scheduling Continuous batching, chunked prefill and prefix caching.
- Memory management PagedAttention for attention key and value memory.
- API serving OpenAI-compatible server, Anthropic Messages API and gRPC support.
- Distributed execution Tensor, pipeline, data, expert and context parallelism.
- Output features Streaming, structured outputs, tool calling and reasoning parsers.
- Model coverage Language, multi-modal, embedding, retrieval and classification models.
Where vLLM fits
- API serving Run models behind an OpenAI-compatible API server, with Anthropic Messages API and gRPC support also listed.
- Quantized inference Load models in formats such as FP8, NVFP4, INT8, INT4, GPTQ/AWQ and GGUF.
- Speculative decoding Choose between n-gram, suffix, EAGLE and DFlash methods.
- Distributed inference Spread a model across GPUs with tensor, pipeline, data, expert and context parallelism.
- Structured output and tool calling Constrain output with xgrammar or guidance, and use tool calling and reasoning parsers.
- Multi-modal and embedding models Serve models such as LLaVA, Qwen-VL and Pixtral, plus embedding and retrieval models such as E5-Mistral, GTE and ColBERT.
- Multi-LoRA and hardware Apply multi-LoRA adapters on NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs.
Install
uv pip install vllmReviews of vLLM
Be the first to share your experience with vLLM.
Write a review
Loading review form…
Alternatives to vLLM
DeepInferDeepInfer is an Apache-licensed DeepSeek inference engine for Linux, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, supporting offline inference.
LocalAILocalAI is an MIT-licensed inference server that runs language, speech, image, video and vision models on CPUs or GPUs and serves them through OpenAI, Anthropic, Ollama and ElevenLabs-compatible APIs, with backends pulled on demand.
OllamaOllama is an MIT-licensed tool for downloading and running open models such as Gemma, Qwen, DeepSeek and gpt-oss on your own machine. It ships a desktop app, CLI and local REST API, and offers cloud-hosted models on a pay-as-you-go or subscription basis.