What is llama.cpp?
llama.cpp is an open-source inference project built on the ggml library and distributed under the MIT license. Its main product is a library with a C-style interface, accompanied by command-line tools and an OpenAI-compatible HTTP server with a built-in web interface. It is intended for developers and technical teams running language and vision-language models locally or in the cloud.
Deployment options include prebuilt binaries, Docker and compilation from source with CMake. The project supports CPU execution, hardware-specific GPU backends and hybrid CPU/GPU inference. Quantization can reduce model memory use, while deployment requirements depend on the selected model, hardware and backend.
Key capabilities
- Inference library A C/C++ implementation built on ggml with no dependencies.
- CLI and server Command-line tools and an OpenAI-compatible API server with a built-in web UI.
- Hardware backends CPU, Metal, CUDA, HIP, Vulkan, SYCL and others.
- Quantization 1.5-bit to 8-bit integer formats.
- Hybrid inference CPU and GPU together for models larger than VRAM.
- Install options Install script, Docker, pre-built binaries or source builds.
Where llama.cpp fits
- Local LLM and VLM inference
- Quantized models
- OpenAI-compatible serving
- Apple silicon, NVIDIA, AMD and Vulkan acceleration
- Hybrid CPU and GPU execution
- Embedding in C/C++ applications
Install
curl -LsSf https://llama.app/install.sh | shReviews of llama.cpp
Be the first to share your experience with llama.cpp.
Write a review
Loading review form…
Alternatives to llama.cpp
DeepInferDeepInfer is an Apache-licensed DeepSeek inference engine for Linux, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, supporting offline inference.
LocalAILocalAI is an MIT-licensed inference server that runs language, speech, image, video and vision models on CPUs or GPUs and serves them through OpenAI, Anthropic, Ollama and ElevenLabs-compatible APIs, with backends pulled on demand.
OllamaOllama is an MIT-licensed tool for downloading and running open models such as Gemma, Qwen, DeepSeek and gpt-oss on your own machine. It ships a desktop app, CLI and local REST API, and offers cloud-hosted models on a pay-as-you-go or subscription basis.