What is Xolotl?
Xolotl is an open-source inference engine and local model server for very large mixture-of-experts (MoE) language models on a single consumer PC. The repository description states the main target: DeepSeek-V4.1-Flash, all 552 billion parameters, running as a local coding agent on one gaming PC with an RTX 4090 and 110 GB of RAM. Alongside DeepSeek-V4.1-Flash it serves GLM-5.3-Flash (320B MoE), Qwen3.8-Flash-Next (125B MoE), the dense Qwen3.8-27B and Liquid AI's LFM3-Omni-2B, each behind one OpenAI- and Anthropic-compatible server. The project is published by Indra's Mirror on GitHub as Indras-Mirror/Xolotl, and the README describes it as research code from one developer, measured on one PC.
Xolotl is built from Strata, the MoE engine by Niko1221, and was previously published as Strata-DSV4.1 (and before that Strata-DS4). Strata's engine, server and core kernels remain part of it, merged at Strata 0.1.44.1; according to AGENTS.md, the last upstream merge on 31 October 2026 brought in Strata version 0.2.8 with engine 0.1.54. Xolotl adds more than 70,000 lines on top: its own DeepSeek-V4.1, V4-Flash and GLM-5.3-Flash engines, the Qwen3.8-27B dense engine (grown from Mirko Perrone's port), the LFM3-Omni engine, a fast mode, Xolotl Direct, and the quantisation, expert-tiering and serving work for each model. Upstream llama.cpp and ggml are fetched at a pinned commit and patched.
The core idea is memory tiering. An MoE model only uses a few experts per token, so Xolotl keeps every expert somewhere: the most frequently used ones in GPU memory, tens of GiB in a pinned system-RAM arena that the GPU reads by DMA, and the rest on an NVMe drive, read with O_DIRECT and prefetched a layer ahead. The How Xolotl works document describes the three tiers, the per-model engines and the shared Python server. Each model adds its own techniques, such as a calibrated mixed-precision quantisation, speculative decoding with exact sampling and a 4-bit KV cache for 512K context on DeepSeek-V4.1.
Xolotl is distributed as source code only. AGENTS.md states that there is no Xolotl installer, and that the Strata app and strata launch install upstream Strata's own engine and models rather than Xolotl's. Linux with an NVIDIA GPU is the main platform, built with CMake, Ninja and CUDA 12.x or 13.x; the DeepSeek-V4.1 engine also builds and serves natively on Windows 11 with MSVC and CUDA, as described in the Windows guide. The repository has no tagged releases; changes are tracked in the Xolotl changelog, newest first, with pull request numbers and contributors.
Recent changelog entries from November 2026 include faster short agent turns on DeepSeek-V4.1 and GLM-5.3-Flash by computing some experts on the CPU while the GPU runs the rest, an opt-in DeepSeek-V4.1 fast mode (7 November), the experimental Xolotl Direct path that lets the NVMe drive write experts straight into VRAM (7 November), a DeepSeek-V4.1 load time cut from 96 to 44 seconds, and native LFM3-Omni-2B support (2 to 3 November). Quantised model builds, several labelled as uncensored or abliterated, are published separately on Hugging Face under the IndrasMirror account. The code is released under the MIT license, with copyright held by Niko1221 and the Strata contributors.
Write-ups and benchmark posts are published on the Indra's Mirror website, starting with Xolotl: Four Frontier Models on One Gaming PC. Related engines listed on iThub include the upstream Strata project and llama.cpp, whose ggml library Xolotl builds on.
Key capabilities
- Three-tier expert storage Experts are spread across a VRAM LRU cache (sized from the VRAM actually free, census-ranked, growing and shrinking per request), a pinned RAM arena registered with
cudaHostRegisterand fed to the GPU by DMA, and an NVMe file tier read with O_DIRECT at around 5 GB/s. The experts each token's router selects are moved a layer ahead. Details are in How Xolotl works. - DeepSeek-V4.1-Flash engine A dedicated engine (
ds41_generate) runs the 552B MoE model with its own GSQ quantisation, where each expert gets a format from calibration (ternary PTQ1_0, Q2_0 or Q3_K), plus the Engram n-gram memory read from disk. It supports 512K context through a bit-exact 4-bit KV cache (--kv-fp4) and DSpark speculative decoding with exact speculative sampling and confidence-gated verification. The engine documentation is in DEEPSEEK_V41.md. - Conversation-adaptive arena Because an agent re-sends its whole conversation each step, the server keeps the previous request's KV cache and prefills only the new tail. On DeepSeek-V4.1 a background reader moves the conversation's frequently used file-tier experts into the RAM arena during idle NVMe time, while routing stays exact. The project reports small-turn time falling from 8.48 to 7.60 seconds in its measurement;
STRATA_ARENA_REBAL=0turns the feature off. - DeepSeek-V4.1 fast mode An opt-in mode reads a 2-bit (Q2_K) copy of the two largest expert formats from a sidecar file of about 40 GB, built once with
ds41_generate --build-shadow(about 30 minutes of CPU time) or downloaded. It is enabled through theXOLOTL_SHADOWenvironment variable and leaves the model file unchanged. The project states it is lossy, changing about one token in a hundred; its table shows agent-task decode rising from 32.6 to 41.1 tok/s on an RTX 4090. See XOLOTL_FAST.md. - Xolotl Direct An experimental, opt-in Linux path in which the NVMe drive writes experts directly into a GeForce card's VRAM over PCIe, without passing through system RAM, using the normal NVIDIA driver. It requires a kernel with
CONFIG_PCI_P2PDMA=y, Resizable BAR enabled and the model on an NVMe drive; it is not available on Windows. The project measured 3.1 ms per 18 MiB expert instead of about 11 ms, and Xolotl falls back to the normal path if a requirement is missing. See XOLOTL_DIRECT.md. - GLM-5.3-Flash engine The GLM engine (
glm_generate) handles the 320B MoE model with KDA plus MLA hybrid attention and 288 experts with top-8 routing. It applies an abliteration LoRA to the dense half and the routed experts on every path (GPU, MMQ prompt chunks and the CPU pool), uses soft pruning that keeps every expert reachable, and has an elastic expert cache that lends VRAM to long prompts. Serving configurations target 512K context. - Qwen3.8 engines Qwen3.8-Flash-Next (125B MoE) runs on the upstream Strata 0.1.44 path with MTP speculative decoding, an int8 KV cache and image input. The dense Qwen3.8-27B runs on
xolotl-dense(formerlystrata-dense, kept as a symlink) with MMQ prompt prefill, MTP speculative decoding and an int8-K / int4-V cache for 200K context. The optional--ngram-chainsetting adds an n-gram draft chain that keeps the output identical. - LFM3-Omni-2B engine
xolotl-lfm3serves Liquid AI's LFM3-Omni-2B with text, picture, audio and video input and text or speech output. It is designed to run beside a large model on the same card and can park and unpark in tens of milliseconds, handing back about 1.4 GB of VRAM. The LFM3 documentation lists its measured prefill, decode and speech figures on an RTX 4090. - OpenAI- and Anthropic-compatible server Every model is served by the same Python server with the model's own chat template and tool-call format (Qwen XML, GLM
<arg_key>, DeepSeek DSML). It exposes OpenAI Chat Completions and Responses at/v1and the Anthropic Messages endpoint at/v1/messages, with streaming, tool calls and conversation-prefix reuse. The setup steps in AGENTS.md keep the server on127.0.0.1unless the user asks for more. - Thinking budget and decision endpoint Requests can set a
reasoning_budget_tokensthinking budget, 32K tokens by default. A/v1/decideendpoint returns a one-token decision from the model's log-probabilities, for example for voice turn-taking without a second model. A server loop guard closes runaway thinking runs, a behaviour that can be restored to the older ending with"repeat_wrap_up": false. - Voice sidecar The server proxies
/v1/audio/transcriptionsand/v1/audio/speechto an ASR and TTS server that starts on the first request and stops when idle. While it runs, the expert cache shrinks to make room. LFM3-Omni can also act as the speech input and output for Qwen3.8-Flash-Next on one RTX 4090. - Build from source on Linux The README build clones the repository, runs
tools/xolotl/fetch-llama.shto fetch llama.cpp at a pinned commit with the Xolotl ggml patch, then configures CMake with CUDA options and builds with Ninja intobuild-xolotl/. The CUDA architecture is set per card: 86 for RTX 30, 89 for RTX 40, 120 for RTX 50 (with CUDA 13.x). Steps are in the README quick start. - Native Windows 11 build for DeepSeek-V4.1 The DeepSeek-V4.1 engine builds with
build-win.batusing Visual Studio 2022 Build Tools, CUDA Toolkit 12.8 or newer (13.1 or newer for RTX 50, with architecture120a), Git for Windows and Python 3.12, without WSL. The guide lists about 370 GB of NTFS NVMe space for the model and the official Engram shards, and a 24 GB NVIDIA GPU. A bundled script downloads Hugging Face files with resume and SHA-256 checks. See WINDOWS.md. - Hardware requirements Will it run on my machine? lists an NVIDIA GPU of compute capability 7.5 or newer (RTX 20, 30, 40 or 50); Tesla V100 has an experimental path only. The measured setup is an RTX 4090 with 24 GB, 110 GB of DDR4 and one NVMe drive. DeepSeek-V4.1 needs about 152 GiB for the quant plus about 200 GB of Engram tables on a fast NVMe formatted ext4 or NTFS, not exFAT and not a hard drive; less RAM means more NVMe reads per token, and around 256 GB holds every expert.
- Smaller GPUs The engine sizes its work to the card, so 12 GB GPUs can run with smaller prompt batches and fewer experts in VRAM. Community runs listed by the project include DeepSeek-V4.1 at 8 to 10 tok/s on an RTX 3060 12 GB with 46 GB of RAM and Qwen3.8-Flash-Next at 39 to 48 tok/s on RTX 3060 and RTX 4070 cards. Community results for further cards, including AMD and Intel, are collected in COMMUNITY_BENCHMARKS.md.
- Published benchmarks and tools The README reports decode, prefill and agent-task scores for each model, measured on one RTX 4090 machine with the Gauntlet v3-small suite of six agentic coding tasks graded by hidden tests, with reference scores from each model's full-precision API through OpenRouter. BENCHMARKS.md holds every measured number and a speed log. Repository tools include
tools/throughput.py,tools/quant_quality.py,tools/turnbench.pyandxolotl-dense --selftest. - Model files on Hugging Face Quantised builds are published separately, including DeepSeek-V4.1-Flash-1017 XQ and Qwen3.8-Flash-Next XQ 45 GB. For Qwen3.8-27B, any dense GGUF works, and
tools/dense_setup.pycan download one and write its run configuration. Model weights originate from DeepSeek, Zhipu AI, Qwen and Liquid AI under their own terms.
Where Xolotl fits
Running DeepSeek-V4.1-Flash as a local coding agent
The project's main documented workflow serves the 552B DeepSeek-V4.1-Flash model on one gaming PC and connects a coding agent to it. After building, the user downloads the 1017 XQ build (V4 recommended) and its Engram data, edits the model paths in tools/v41/serve/strata-ds41.json and starts serve_ds41.py on port 8150. The README table lists 32.6 tok/s decode on agent tasks, 289 to 409 tok/s prefill and 512K context on the RTX 4090 test machine, with the opt-in fast mode reaching 41.1 tok/s.
Serving GLM-5.3-Flash with long context
GLM-5.3-Flash, a 320B MoE model, is served through tools/glm/serve/serve_glm.py with a 512K configuration on port 8144. The changelog describes long thinking runs in coding agents, prefix reuse that resumes a retried conversation in about 1.2 seconds instead of re-reading 7,400 tokens, and a decode rate of 28.5 to 29 tok/s on the test machine after its RAM upgrade. AGENTS.md notes that DeepSeek-V4.1 and GLM-5.3 each take most of the RAM, so one large model runs at a time.
Fast local models on mid-range GPUs
Qwen3.8-Flash-Next (125B MoE, XQ 45 GB build) is started with scripts/xol-qfn.sh and listens on port 8099; the README lists 157 tok/s decode on an RTX 4090 and community runs of 39 to 48 tok/s on 12 GB RTX 3060 and RTX 4070 cards. The dense Qwen3.8-27B runs through python3 -m serve.server on port 8090, where the --mtp and --ngram-chain engine arguments speed up decoding while keeping output exact.
Connecting OpenAI and Anthropic clients
Each server prints its address when ready, and clients connect to http://127.0.0.1:<port>/v1 for OpenAI Chat Completions and Responses or /v1/messages for the Anthropic Messages API. Tool calls are translated to and from each model's native format, and the thinking budget is set per request with reasoning_budget_tokens. Server behaviour is documented in How Xolotl works.
Voice and multimodal input beside a large model
LFM3-Omni-2B can run on the same card as a large model and act as its speech input and output, with text, pictures, audio and video accepted as input. The voice sidecar exposes transcription and speech endpoints, and the /v1/decide endpoint returns one-token decisions for voice turn-taking. The LFM3 guide describes park and unpark behaviour and the VRAM it releases.
Measuring a machine and reporting results
The repository ships the Gauntlet agent-task suite, throughput and quant-quality scripts and per-engine self-tests, and the README invites reports from other hardware. Contributions and issues go to the GitHub repository, with GitHub issues as the reporting channel; changes to the upstream Strata core can also be sent to Niko1221/Strata. Similar MoE offloading projects listed on iThub include Project Maya and colibri.
Reviews of Xolotl
Be the first to share your experience with Xolotl.
Write a review
Loading review form…
Alternatives to Xolotl
StrataOpen-source local LLM engine with a desktop app and terminal app for Windows, Linux and Mac. It splits mixture-of-experts models across GPU, RAM and SSD and serves OpenAI- and Anthropic-compatible APIs to coding tools.PMProject MayaProject Maya runs large mixture-of-experts models locally, spreading experts across GPU memory, RAM and SSD, with browser chat and OpenAI-compatible APIs.
colibricolibri is an Apache-2.0 inference engine in C that runs large mixture-of-experts models locally by keeping dense weights in RAM and reading experts from disk.