Running LLMs locally: Ollama, Strata, Xolotl, GPT4All and AnythingLLM compared

Engines and apps: two layers of a local LLM setup

A local setup has two parts. The engine loads the weights into memory and generates tokens. The application is what a person uses: a chat window, a document library, an agent or a coding tool. Some projects ship both, and some focus on one layer and connect to others over an API.

The five projects share a common lineage in the GGUF file format and the llama.cpp and ggml code base. The GPT4All models page says it runs Hugging Face models with a llama.cpp backend and that many of them use the .gguf file type. Strata's how it works page credits parts of llama.cpp and ggml. Xolotl's quick start fetches llama.cpp at a pinned commit and applies its own patch. AnythingLLM's Docker guide describes its optional built-in engine as Lemonade running llama.cpp. Ollama's import guide covers bringing in both GGUF files and Safetensors weights through a Modelfile.

The engines differ in focus. Ollama documents a general model library and a broad GPU list. Strata and Xolotl are built around mixture-of-experts (MoE) models, where each token only uses a few of the model's experts, and they spread those experts across graphics memory, system RAM and the SSD so that models larger than the graphics card can run on a desktop PC.

Ollama: model library, CLI and local REST API

Ollama describes itself as a way to start building with open models, locally or in the cloud. The repository is MIT licensed, and the latest release at the time of writing is v0.40.16, published on 9 November 2026.

Install paths and system requirements

The README lists a shell installer for macOS and Linux (curl -fsSL https://ollama.com/install.sh | sh), a PowerShell installer for Windows (irm https://ollama.com/install.ps1 | iex), manual downloads, and the official ollama/ollama image on Docker Hub. The Docker page publishes port 11434, with variants for NVIDIA and AMD (ROCm).

The Windows page requires Windows 10 22H2 or newer, NVIDIA drivers 551.61 or newer for NVIDIA cards, and at least 4 GB of disk for the binaries, with models needing tens to hundreds of GB more. The installer does not need Administrator rights. The macOS page requires macOS Sonoma (v14) or newer and states that Apple M-series Macs get CPU and GPU support while x86 Macs run on the CPU only.

Supported GPUs

According to the hardware support page, Ollama supports NVIDIA GPUs with compute capability 5.0 or higher and driver 550 or newer, with cards of compute capability 5.0 to 6.2 needing driver 570 or newer. The table runs from the GTX 750 Ti to the RTX 50 series and the GB10 in DGX Spark. AMD Radeon cards are supported through ROCm, which on Windows requires a ROCm v7 or HIP7-capable driver stack, and additional AMD and other GPUs through Vulkan on Windows and Linux. Apple GPUs use the Metal API.

The Ollama local API

The local server binds to 127.0.0.1 on port 11434 by default, and the FAQ explains how to change the bind address with the OLLAMA_HOST environment variable or place the server behind a proxy such as Nginx. The API introduction lists three base URLs on a local server:

• Native Ollama API http://localhost:11434/api
• OpenAI compatibility http://localhost:11434/v1
• Anthropic compatibility http://localhost:11434

Local requests need no API key. The native API reference covers generate and chat completions, creating, listing, showing, copying, deleting, pulling and pushing models, embeddings, listing running models and the server version. Generate and chat responses stream unless the request sets "stream": false, and the keep_alive parameter controls how long a model stays in memory after a request, five minutes by default.

The OpenAI compatibility page describes a subset of the OpenAI API, with /v1/chat/completions, /v1/completions, /v1/responses, /v1/models and /v1/embeddings. The Anthropic compatibility page covers a subset of the Messages API and shows how to point tools such as Claude Code at a local server by setting ANTHROPIC_BASE_URL=http://localhost:11434.

Models, storage and context length

Models are pulled from the library at ollama.com, for example with ollama run gemma4, and the ollama launch command connects integrations such as Claude Code, Codex, Copilot CLI and OpenCode. The FAQ documents the OLLAMA_MODELS variable to move the model folder. The context length page sets the default context by available graphics memory: 4k below 24 GiB of VRAM, 32k from 24 to 48 GiB and 256k from 48 GiB, adjustable with OLLAMA_CONTEXT_LENGTH. The same page says agents and coding tools need at least 64000 tokens.

Ollama also offers cloud models. The API introduction states that cloud requests need an API key, and that a local server can use cloud models after signing in, in which case those requests run on Ollama's service rather than on the local machine.

Strata: an MoE engine with a desktop app and terminal launcher

Strata is an MIT-licensed project by Niko1221 whose README describes it as a way to run powerful AI on your own computer, with no account needed. The latest release is Strata 0.2.13, published on 9 November 2026, whose notes list new DFlash2 drafting defaults per card, a faster app start and groundwork for Qwen4.

Strata supported models

The README documents three model families, with ISTA-DASLab files recommended first, then Unsloth and other providers:

• Qwen3.8-Flash-Next ISTA-DASLab Q2_0 to IQ3_S plus an IQ1_M coder build, Unsloth UD-IQ4_XS to Q8_0, and agentionai Gyro files.
• Qwen3.6-35B-A3B Unsloth UD-Q4_K_XL, UD-Q6_K_XL and Q8_0.
• DeepSeekV4.1-Flash-1017 ISTA-DAS sets at 1.75, 2.00 and 2.25 bits per weight, downloads of 239 to 274 GB, and a 191 GB coder set with half of the experts removed.

The models guide gives memory figures. For Qwen3.8-Flash-Next, the weights need 37.6 GB of RAM plus VRAM at Q2_0 and 54.8 GB at IQ3_S. DeepSeek V4.1-Flash needs a graphics card with 12 GB or more, at least 46 GB of RAM (64 GB is described as the tested comfortable size), an NVMe SSD and the disk space of the chosen set. Qwen3.6-35B-A3B is listed at 32 to 52 GB of RAM. Model files are not part of the repository, and the project states that each model's own licence applies to its files.

How Strata fits large models on a desktop

Strata's how it works page explains the approach with Qwen3.8-Flash-Next, which has 24,576 small experts, of which each token uses 10. The graphics card handles the work needed for every token and caches the most-used experts, all experts sit pinned in system RAM where the CPU computes those the card does not hold, and a 28.8 GB n-gram lookup table stays on the SSD. A built-in draft layer guesses up to three tokens that the main model then checks in one pass; the project reports that this gives the same answer 1.6 to 1.8 times sooner.

Install, hardware and API

The desktop app has one installer per system (a Windows setup file, a Linux AppImage and a macOS arm64 DMG), and on first start it downloads the engine for the detected graphics card. The terminal app installs with a shell script on Linux and macOS or a PowerShell command on Windows, then runs with strata launch. Docker and source builds are also documented.

The README's hardware table lists NVIDIA GeForce RTX 20, 30, 40 and 50 series cards; AMD Radeon RX 7700 to 7900, RX 9060 and 9070 and the AI PRO R9700, with RX 6800 and 6900 marked experimental; Intel Arc on Linux; Apple Silicon M1 or newer; AMD Ryzen AI Max (Strix Halo); and NVIDIA DGX Spark on Linux.

For tools, the README names Claude Code, Codex CLI, OpenCode, oh-my-pi, pi.dev, Hermes Agent and OpenClaw, connected through the OpenAI- and Anthropic-compatible APIs, and the strata connect command prints setup steps for a given tool. The terminal guide uses http://127.0.0.1:8080 as the default server address and offers access modes for this computer only, the local network, a Cloudflare quick tunnel or Tailscale. According to the safety and privacy page, the server listens on 127.0.0.1, sharing beyond the PC requires an API key, the shared port answers only /v1/models, /v1/chat/completions and /v1/messages, and Strata sends no telemetry, crash reports or analytics.

Xolotl: a Strata-based engine for very large MoE models

Xolotl, published by Indra's Mirror, is built from Strata and extends it with its own engines. The README states that Niko1221's engine, server and core kernels remain at its heart and that Xolotl adds more than 70,000 lines on top: engines for DeepSeek-V4.1, V4-Flash and GLM-5.3-Flash, a Qwen3.8-27B dense engine grown from Mirko Perrone's port, an LFM3-Omni engine, a fast mode and an experimental path called Xolotl Direct. The AGENTS.md file records the last upstream merge on 31 October 2026, bringing in Strata 0.2.8 with engine 0.1.54. The code is MIT licensed. The README describes it as research code from one developer, measured on one PC.

Xolotl supported models

The README lists DeepSeek-V4.1-Flash (552B MoE), GLM-5.3-Flash (320B MoE), Qwen3.8-Flash-Next (125B MoE), the dense Qwen3.8-27B and Liquid AI's LFM3-Omni-2B, which takes text, pictures, audio and video in and produces speech. The README states that all of them sit behind one OpenAI- and Anthropic-compatible server. The quantised DeepSeek and Qwen3.8-Flash-Next builds linked from the README are hosted on Hugging Face under the IndrasMirror account, and several of those builds carry "Uncensored" in their names. For Qwen3.8-27B, the README accepts any dense GGUF file of that model.

How Xolotl tiers memory, and its published figures

The How Xolotl works document describes three tiers: the most-used experts in VRAM, tens of GiB in a pinned RAM arena that the GPU reads by DMA, and the rest on an NVMe drive, with experts moved a layer ahead of when the router asks for them.

The project publishes measurements from one machine: an RTX 4090 with 24 GB, a Ryzen 7 5700X, 110 GB of DDR4 and one NVMe drive. On that PC the README reports 32.6 tokens per second of decode for DeepSeek-V4.1 during its own agentic coding test set, and 41.1 tokens per second in an opt-in fast mode that reads a 2-bit copy of the largest experts and, by the project's description, changes about one token in a hundred. These figures are the developer's, from that single configuration.

Install and requirements

Xolotl is distributed as source code. AGENTS.md states that there is no Xolotl installer and that the Strata app and strata launch install upstream Strata's engine and models, not Xolotl's. The quick start requires Linux, an NVIDIA GPU and CUDA 12.x or 13.x, and builds with CMake and Ninja after a script fetches llama.cpp at the pinned commit. Each model is then served by its own script on its own port (the README's examples use 8150 for DeepSeek-V4.1, 8144 for GLM-5.3-Flash, 8099 for Qwen3.8-Flash-Next and 8090 for Qwen3.8-27B), and clients point at http://127.0.0.1:<port>/v1.

The Windows guide covers the DeepSeek-V4.1 engine only, built natively on Windows 11 with MSVC and CUDA, without WSL. It lists an NVIDIA GPU with 24 GB, 128 GB of RAM recommended for the example configuration, about 370 GB on an NVMe drive, Visual Studio 2022 and CUDA Toolkit 12.8 or newer (13.1 or newer for RTX 50 cards). The repository has no tagged releases; changes are tracked in the changelog, newest first, with pull request numbers.

GPT4All: a desktop app with LocalDocs and a local server

GPT4All from Nomic AI is described in its README as running LLMs privately on everyday desktops and laptops, with no API calls or GPUs required. The repository is MIT licensed. Its latest release is v3.10.0, published on 25 February 2025, and the most recent commit to the repository dates from 27 May 2025.

Installers and requirements

The README links installers for Windows, Windows ARM, macOS and Ubuntu. It states that the Windows and Linux builds require an Intel Core i3 2nd Gen or AMD Bulldozer CPU or better, the Windows ARM build supports Qualcomm Snapdragon and Microsoft SQ1 and SQ2 processors, the Linux build is x86-64 only, and the macOS build requires Monterey 12.6 or newer. The system requirements page recommends 16 GB of RAM and an NVIDIA GTX 1080 Ti or RTX 2080 class GPU with 8 GB or more of VRAM on PCs, or an M2 Pro on Apple hardware. Its minimum is 16 GB of RAM, or 8 GB for 3B models, and any Direct3D 11/12 or OpenGL 2.1 capable GPU.

Models and LocalDocs

The models page says GPT4All is optimised for models in the 3 to 13 billion parameter range on consumer hardware, searchable from Hugging Face inside the app. Its example table lists Llama 3 Instruct 8B as a 4.66 GB file needing 8 GB of RAM.

LocalDocs indexes a linked folder into text snippets with Nomic's on-device embedding models, finds snippets that match a question and adds them to the prompt, with a Sources view below each answer showing which files were used.

GPT4All API server and Python SDK

The API server page describes an OpenAI-compatible server that is switched on in Settings, Application, Advanced, with the "Enable Local API Server" option. It listens on port 4891 by default at http://localhost:4891/v1, accepts HTTP only and listens only on 127.0.0.1. Documented endpoints are /v1/models, /v1/completions and /v1/chat/completions. LocalDocs collections can be attached to the server chat, with references returned in the API response. For scripts, pip install gpt4all installs a Python client around llama.cpp.

AnythingLLM: a document and agent workspace for local or hosted models

AnythingLLM from Mintplex Labs is an MIT-licensed application for chatting with documents, running agents and connecting to local or cloud LLMs. The latest release is v1.19.0, published on 26 October 2026. It comes as a desktop app for macOS, Windows and Linux, a Docker image and a hosted instance, and the README marks multi-user permissions, white-labelling and the embeddable chat widget as Docker-only features.

Model providers and the built-in engine

The README lists any llama.cpp-compatible open model plus a long set of providers, including Ollama, LM Studio, LocalAI, KoboldCPP, Microsoft Foundry Local, OpenAI, Anthropic and AWS Bedrock. Ollama is also listed as an embedder, and LanceDB is the default vector database. The model storage notes state that local chat models must be GGUF files compatible with current llama.cpp.

The Docker guide describes an optional built-in engine, turned on with ENABLE_BUILT_IN_LLM=true, that runs Lemonade and llama.cpp inside the container so that no separate Ollama or LM Studio server is needed. The engine is reachable only from inside the container, and the README lists CPU, NVIDIA (CUDA) and AMD and Intel (Vulkan) as supported hardware.

Install, requirements and API

The Docker image is mintplexlabs/anythingllm, run with a mounted storage folder and published on port 3001, after which the app is available at http://localhost:3001. The Docker guide recommends at least 2 GB of RAM and 10 GB of disk on a cloud VM. For the desktop app, the system requirements page recommends 16 GB of RAM and an 8-core CPU for a basic experience, notes that the local model is the main factor in hardware needs, and suggests a GPU with 8 to 12 GB or more of VRAM on Windows.

The API page says the full developer API is documented on each instance at /api/docs and is protected by API keys created by accounts with the right access level. The server source includes OpenAI-compatible routes under /v1/openai for models, chat completions, embeddings and image generation.

On privacy, the README states that AnythingLLM collects anonymous usage telemetry through PostHog, such as the installation type and the LLM provider, without chat content. It can be turned off with DISABLE_TELEMETRY="true" or in the app's Privacy settings.

Side-by-side comparison from the official documentation

How these local LLM tools connect to each other

Because the engines expose OpenAI-compatible endpoints, the projects can be layered. AnythingLLM lists Ollama as an LLM and embedding provider, so an Ollama server can supply the model while AnythingLLM handles documents, agents and users.

Ollama, Strata and Xolotl also document Anthropic-compatible endpoints, which is how coding tools that expect the Anthropic API reach a local model. Ollama's compatibility page sets ANTHROPIC_BASE_URL to the local server, Strata's strata connect prints the steps per tool, and Xolotl's README points Anthropic-compatible clients at the same local port as OpenAI-compatible ones. GPT4All's server documents OpenAI-style endpoints only.

What the documentation flags before a local LLM install

• Disk space Ollama's Windows page puts models at tens to hundreds of GB, Strata's DeepSeek sets need about 197 to 280 GB of free disk, and Xolotl's Windows guide lists about 370 GB on NVMe.
• System RAM as well as VRAM For Qwen3.8-Flash-Next, Strata's models guide states that RAM must hold the experts plus about 10 GB for the system, and that a bigger graphics card makes it faster without lowering the RAM needed, except in its low-RAM mode.
• Network exposure Ollama, Strata and GPT4All bind to the local machine by default. Ollama documents OLLAMA_HOST for wider binding, Strata requires an API key for sharing, and GPT4All's server listens on 127.0.0.1 only.
• Cloud fallbacks Ollama cloud models, GPT4All remote providers and AnythingLLM cloud providers send prompts to an outside service, as each project's documentation states.
• Model licences Strata states that each model's own licence applies to its files, and GPT4All's models page notes that licences vary for personal and commercial use.

The local LLM inference category on iThub lists these engines next to related projects such as llama.cpp, and the AI assistants and chat category covers GPT4All, AnythingLLM and other chat applications.

Sources

• Ollama: README, API introduction, API reference, OpenAI compatibility, Anthropic compatibility, hardware support, FAQ, context length, Windows, macOS, Docker, import, v0.40.16 release
• Strata: README, how it works, models, terminal guide, safety and privacy, Docker, v0.2.13 release
• Xolotl: README, AGENTS.md, how Xolotl works, Windows guide, changelog
• GPT4All: README, system requirements, API server, LocalDocs, models, v3.10.0 release
• AnythingLLM: README, Docker guide, model storage, system requirements, API access, OpenAI-compatible routes, v1.19.0 release