Search tools, AI agents, MCP servers
Submit a tool

Search iThub

Search tools, categories and collections. Type at least two characters.

Explore all tools
↑ ↓ to explore resultsEnter to open · Esc to close

Edge0

Open-source streaming MoE inference framework and model family for local hardware, from phones to one NVIDIA GPU

Local LLMs & InferenceOpen sourcev0.2.0

Edge0 at a glance

Pricing
Open source
Deployment
Self-hosted
License
Open source
Version
v0.2.0November 11, 2026
Platforms
WindowsmacOSLinux
Language
C++
Data location
Local only: models run on the user's device or GPU; the local server binds to 127.0.0.1 by default, and network access is needed only to fetch source and model weights from GitHub, Hugging Face or ModelScope.
Maker
Edge0-AI
Founded
2026

What is Edge0?

Edge0 is an open-source streaming mixture-of-experts (MoE) inference framework from the company of the same name, published in the Edge0-AI/Edge0 repository together with a family of sparse language models. The repository describes its goal as running large sparse models on consumer hardware: instead of holding every expert in memory, the engine keeps a working set of experts resident and reads the rest from SSD or host RAM on demand. The official website presents Edge0 as on-device AI with local processing and no per-token cost, and its About page names Samuel Zeng as founder and CEO. The company also publishes Audio8 speech models on its Hugging Face organization; this profile covers the inference framework and the Edge0 language models.

The project was first released on September 8, 2026 with two preview checkpoints, Edge0-35B-A3B-preview and Edge0-8B-A1B-preview, followed on September 16 by the technical report The Other Half of the Memory Wall on serving 35B MoE models from SSD with trained routing prediction. Per-platform engines for iOS, macOS, Android and Windows followed on September 30 and are archived at the pre-edge0-infer tag. On November 4, 2026 the team released edge0-infer, a single C++ engine built on GGML with one native API, which replaces the earlier Python framework and the four separate platform apps.

The latest release is Edge0 0.2.0, published on November 11, 2026. It adds Edge0 145B and Edge0 120B, two latent-MoE models derived from Qwen3.8-Flash-Next and trained natively at 4 bits, which run on edge0-strata, a fork of the Strata engine, on a single NVIDIA GPU under Linux or Windows. The release also introduces edge0-infer as the shipped engine for Edge0 35B and adds Anthropic Messages API endpoints to the local server. Its binary assets are Windows only: Edge0 Chat packages for x64 and Arm64. Earlier releases 0.1.0 and 0.1.1 shipped an ad-hoc signed macOS app, a sideload Android APK and an unsigned iOS IPA for the previous engines.

Platform support depends on the model. Edge0 35B runs on the native engine on macOS (Metal, CPU), iOS (Metal), Android (CPU), Linux (CPU) and Windows (CPU, Vulkan). Edge0 145B and 120B run only on Linux or Windows with an NVIDIA GPU and CUDA 13. The 8B checkpoint is published, but native-engine support for it is listed as planned. The repository description summarizes the project as 145B on a laptop and 35B on a phone; the README states that its Apple M2 Ultra measurements for 145B do not establish macOS release availability.

The framework code is licensed under Apache-2.0, while bundled components keep their own licenses, including MIT for GGML and for Strata. Model weights are covered separately by the Edge0 Community Model License 1.0 plus the upstream Qwen license. That model license permits personal, research, educational and internal use and commercial use of generated outputs, but requires written permission from Edge0 for paid model distribution, commercial inference or fine-tuning services and independent commercial coding or office assistants.

Key capabilities

  • edge0-infer command line The edge0-infer program provides run, chat, bench, serve and info commands with the same syntax for every Edge0 model. It reads model_name from the model folder's model.json and selects the matching engine itself. For 145B and 120B only serve and chat (plus inspect) are available; run belongs to the native engine.
  • Local OpenAI- and Anthropic-compatible server edge0-infer serve listens on http://127.0.0.1:8000 by default and answers /v1/chat/completions, /v1/responses, /v1/messages, /v1/messages/count_tokens and /v1/models. The local server documentation lists request fields and limits. Files, Batches and the Anthropic Models endpoints are not implemented.
  • LAN access with token authentication Loopback clients may use any nonempty API key. A non-loopback listener requires --api-key-file, and remote clients must send the token as Authorization: Bearer or x-api-key. The server sets no CORS headers.
  • Native engine on GGML The native engine for Edge0 35B depends only on GGML, vendored unmodified at a fixed version, and the build downloads nothing. The same code runs on macOS, iOS, Android, Linux and Windows with Metal, Vulkan or CPU backends, and prefill and decode can use different backends on the same device.
  • Native C++ API Applications link the CMake target edge0::edge0 and include the headers in include/edge0/. The native API guide covers runtime lifecycle, streamed generation, multi-turn sessions, concurrency and cancellation, and examples/native_chat.cpp shows a full two-turn session. The 145B and 120B models are not available through this API.
  • Experts streamed on demand Routed experts are read from storage into a fixed pool of expert slots per layer, so a model larger than the device memory budget runs without loading all of its weights. The project states that peak memory is bounded by the active expert set rather than the parameter count.
  • Prerouter routing prediction Edge0 35B ships with a trained prerouter head that predicts the next token's experts so loads overlap computation. The predicted routing mode is the default and actual runs the model's original computation for comparison. The project reports up to 59 percent higher decode throughput from the prerouter, with the gain growing with storage latency, model size and routed width.
  • Recover-LoRA quantization recovery For the 35B and 8B tiers, the int4 base is frozen and LoRA adapters are trained by distillation from the full-precision teacher. Adapters are stored separately and can be switched on or off. Edge0 reports an average loss of 3.9 points for 35B and 2.8 points for 8B against the fp16 base models on its own OpenCompass runs.
  • Edge0 145B and 120B latent-MoE models Both models have 48 layers with 512 routed experts plus one shared expert and route each token to 10 experts. Expert inputs are compressed from 2560 to 1792 dimensions (145B) or 1280 dimensions (120B). The model page lists 10.12B active parameters per token for 145B, 8.64B for 120B, a 256K context and one multi-token-prediction layer for speculative decoding.
  • Q3_K and NVFP4 builds Each large model is published in two formats on Hugging Face, for example Edge0-145B-q3k and Edge0-145B-NVFP4. The 145B builds take 40.81 GiB (Q3_K) and 47.79 GiB (NVFP4) of weights, the 120B builds 27.98 GiB and 32.96 GiB. Model weights are never included in the release packages.
  • VRAM and host RAM capacity controls Flags in the model's model.json control the edge0-strata engine: --vram-reserve-mib (default 700), --expert-cache, --kv-resident, --spec for the speculative verify window (default 4), --resident-budget-gib and --mmap-experts. Expert residency keeps VRAM and RAM expert sets disjoint so no expert is held twice.
  • SSD offload mode --mmap-experts reads uncached experts from the pack file on SSD on demand. Combined with --resident-budget-gib 20 and a static expert profile, host RAM used for the expert cache is capped at 20 GiB. The README notes that this flag does not limit total process memory.
  • Edge0 Chat desktop app for Windows Release 0.2.0 ships edge0-chat-0.2.0-windows-x64.zip and edge0-chat-0.2.0-windows-arm64.zip with published SHA256 checksums. On x64, Edge0 35B runs on the CPU (AVX2) or through Vulkan, and 145B or 120B need an NVIDIA GPU of compute capability 7.5 or newer with at least 16 GiB of GPU memory, 20 GiB of system memory and CUDA Toolkit 13. The Arm64 package targets compute capability 12.1 GPUs such as the NVIDIA RTX Spark N1X.
  • Chat app examples for other platforms The repository includes chat application examples for desktop (macOS and Windows, built with Tauri), Android and iOS under edge0-infer/examples/. They are built from source; the current release does not ship mobile binaries, and the older 0.1.x Android APK and iOS IPA were sideload builds outside Google Play and the App Store.
  • Build from source The native engine needs CMake 3.21 or newer, Ninja, a C++17 and C11 compiler, uv and Git, with presets such as mac-metal and cpu. For 35B, the project lists about 18 GB for the original weights and about 20 GB for converted assets, plus the same amount free during conversion. The 145B and 120B path also requires building edge0-strata with CUDA 13 for Linux or Windows on x64 or Arm64.
  • Model asset tools Python tools under tools/assets download the 35B checkpoint from Hugging Face, check it against a locked revision and digests, convert it into engine assets and verify the result. The same folder documents how to add a new model to the engine.
  • Status and measurement tooling The server exposes /health and /status, and models report context_length and thinking capabilities through /v1/models. A benchmarking guide and measurement scripts describe how the project produced its reference figures. For 145B and 120B, parts of /status are null and concurrent requests are queued instead of answered with 429.

Where Edge0 fits

Running a 35B MoE model on a laptop or phone

The Edge0 35B tier is based on Qwen3.6-35B-A3B with LoRA adapters and a trained prerouter. On the native engine, expert weights stay on storage and only the active set is loaded. Edge0 reports about 24.8 tok/s decode with a 3.43 GiB peak footprint on an Apple M5 Mac with Metal, 13.2 tok/s on an iPhone 17 Pro Max and 11.4 to 11.8 tok/s on an Android phone using the CPU, measured with edge0-infer over three processes. The 35B and 8B release guide documents setup per platform.

Serving a 145B or 120B model from one NVIDIA GPU

For the latest models, a user builds edge0-strata and edge0-infer, downloads one model folder such as Edge0-145B-q3k and starts edge0-infer serve --model with that folder. The server prints the listening address once weights are loaded. Edge0 reports 121.2 tok/s agent decode and 3554.9 tok/s prefill at a 16K input for 145B Q3_K on an RTX 5090 D with 24 GB, and 91.7 tok/s agent decode on an RTX 4090.

Running under tight VRAM and RAM budgets

The project publishes measurements of the same RTX 5090 D capped to 16 GiB VRAM and 20 GiB system RAM through --vram-reserve-mib and a cgroup limit, where 145B Q3_K reached 44.9 tok/s agent decode by the project's figures. The README describes this budget as a test limit rather than a minimum requirement. SSD offload, a host RAM expert budget and a static expert profile are the documented memory-saving options.

Local backend for agents and coding tools

Because the local server answers both OpenAI Chat Completions and Responses requests and the Anthropic Messages API, existing SDK clients can point at 127.0.0.1:8000. Edge0 positions 145B and 120B for local agent workloads and publishes model-level scores on DeepSWE 1.1, Terminal-Bench 2.1, HLE, GPQA-Diamond, IFBench and AA-LCR; for example 86.5 on Terminal-Bench 2.1 for Edge0-145B, as reported by the project. The known limitations page lists behavior such as occasional recomputation in multi-turn reuse.

Embedding the engine in an application

Developers can link the edge0::edge0 CMake target and call the native API in process, with streaming, multi-turn sessions and cancellation. The Android, iOS and desktop chat examples in the repository use this engine. The Edge0 documentation site covers app builds and privacy for the earlier 8B and 35B releases.

Research on SSD streaming and routing prediction

The technical report and the routing and prerouter documents describe SSD expert offload, the dual shift prediction across layers and tokens, and the benchmark method. The roadmap for Q4 2026 lists support for next-generation architectures with hybrid linear attention and latent thinking with batched expert pre-prediction; these items are announced, not released. Similar local MoE projects on iThub include Strata, on which edge0-strata is based, colibri and Xolotl.

Reviews of Edge0

No reviews yet

Be the first to share your experience with Edge0.

Loading review form…

Found Edge0 useful? Upvote it todayYour vote helps other IT teams find it in the daily ranking.

Similar to Edge0

Compare:Edge0 vs colibriEdge0 vs XolotlEdge0 vs Project Maya

  1. colibriLocal inference in C that streams model experts from RAM and diskLocal LLMs & Inference2 views
  2. XolotlLocal MoE inference server that tiers model experts across VRAM, pinned RAM and NVMeLocal LLMs & Inference32 views
  3. Project MayaLocal engine and server for GLM, DeepSeek and MiniMax MoE models, with experts tiered across GPU, RAM and SSDLocal LLMs & Inference7 views
  4. FreeTokenOpen-source engine to serve MoE models on consumer GPUs and CPUsLocal LLMs & Inference1 views
  5. StrataRun large open models like Qwen3.8-Flash-Next locally on consumer GPUs, with an OpenAI-compatible APILocal LLMs & Inference507 views
  6. OllamaRun open-weight LLMs locally on macOS, Windows or Linux with a REST API, plus optional paid cloud modelsLocal LLMs & Inference40 views
  7. DeepInferRun DeepSeek model inference with an accelerator-focused engine and explicit distributed deployment requirementsLocal LLMs & Inference8 views
  8. InferSSDOpen-source llama.cpp-based engine that streams MoE experts from SSD to run models larger than RAM or VRAMLocal LLMs & Inference8 views