Submit a tool

Search iThub

Search tools, categories and collections. Type at least two characters.

Explore all tools
↑ ↓ to explore resultsEnter to open · Esc to close

Local LLMs & Inference

Engines and runtimes that run open-weight language models on your own hardware, from laptops to GPU servers.

Choosing guide

16 results in Local LLMs & Inference

Active Pro listings from this category appear first. Your sort applies within Pro and Free results.

Advertisements
Advertisements
Focused collections

Shortlists for Local LLMs & Inference

Explore collections with tools relevant to this category, grouped by deployment, source model or workload.

Self-hosted IT tools →

14 matching tools from this category

IT tools with an API →

14 matching tools from this category

Open-source IT tools →

14 matching tools from this category

How to choose local llms & inference tools

Local inference software loads open-weight language models onto hardware you control, from a laptop CPU to a multi-GPU server, and serves them through a CLI, a desktop app or an HTTP API. It keeps prompts and outputs on your own machines and removes per-token billing. Unlike MLOps platforms, which manage training and the model lifecycle, these engines focus on loading weights, quantization and fast token generation.

Read the guide

Who uses these tools

  • Developers prototyping against a local model endpoint
  • Platform engineers serving models on internal GPUs
  • Privacy-bound teams keeping prompts on premises

Capabilities to compare

  • Supported model architectures and quantization formats
  • CPU, GPU and Apple silicon backends
  • OpenAI-compatible HTTP serving endpoint
  • Batching, KV cache and throughput tuning
  • Offloading for models larger than VRAM

Operating considerations

  • VRAM and RAM size set the largest usable model
  • Model licences differ from the engine's own licence
  • Driver and toolkit upgrades can break builds
  • Exposed inference ports need authentication in front
  • Power draw and cooling for always-on GPU boxes

Selection checklist

  1. List the models and context lengths you need, then confirm the engine supports those architectures and quantization formats.
  2. Benchmark tokens per second and time to first token on your actual hardware, not on published reference machines.
  3. Check whether the engine exposes an API your existing clients already speak, so applications need no rewrite.
  4. Decide between a single-user desktop runtime and a multi-user server with batching before you buy GPUs.

What belongs in this category: A product belongs here when its main job is executing model weights on user-owned hardware, as an engine, runtime, serving layer or driver tweak, rather than training models, tracking experiments or calling a hosted model API.

Local LLMs & Inference on iThub in figures

iThub lists 16 tools in Local LLMs & Inference. 14 of 16 are open source, 14 of 16 can be self-hosted, 15 of 16 have a free, freemium or open-source pricing model and 14 of 16 offer an API.

Tools listed
16
Open source
14
Self-hostable
14
Free to start
15
With an API
14
Web app
6
Desktop app
15

Recently added: Ollama, AnythingLLM, GPT4All, Xolotl

Typical use cases

  • Running a coding assistant against a model on a workstation
  • Serving an internal chat endpoint on shared GPU servers
  • Processing confidential documents without sending them to a cloud API
  • Testing quantized models before committing to hardware purchases

Trade-offs to model

  • Desktop runtimes are quick to install and switch models, while server engines need more setup but handle many concurrent requests efficiently.
  • Aggressive quantization lets larger models fit on modest hardware, but it can reduce answer quality on reasoning and code tasks.

Common mistakes

  • Sizing hardware for the model weights alone and forgetting the memory needed for long context windows.
  • Binding a local inference server to every network interface without authentication, exposing it to the whole network.

Compare Ollama and vLLM →Read about running large models on consumer PCs →Browse MLOps and LLMOps tools →

Decision resources

Continue the technical evaluation

Use the resources that match your decision stage. Profile data and linked sources were refreshed through November 10, 2026.

Evaluation methodology · Corrections process