What is NVIDIA NeMo Switchyard?
NVIDIA NeMo Switchyard is an open-source library that helps an AI agent choose which model handles each request. It combines efficient models with more capable models so you can balance task success, cost and latency on your own workload. The code is published by NVIDIA under the Apache 2.0 licence.
You adopt it by embedding its Rust core, libsy, in your gateway or agent harness. You own the model clients, credentials and service, and Switchyard supplies the routing decision. You configure the models an agent can use and choose a routing algorithm. The algorithm examines the request or the agent's recent tool activity, and some algorithms call a model to judge the task. Routing can change as the agent continues its work.
Four routing choices are documented: Auto, Task, Execution and Composite. Task has a model judge whether the efficient model can handle the work, while Execution uses recent tool activity and outcome signals. Switchyard is also available through OpenRouter, an experimental LiteLLM example and a NeMo Relay plugin, and there are Python bindings and a pypi package. The project is pre-1.0, so APIs and routing behaviour can change between releases.
Key capabilities
- A Rust core library, libsy, for embedding routing in your own gateway or agent harness.
- Routing algorithms: Auto, Task, Execution and Composite.
- Gateway paths through OpenRouter, a LiteLLM example and a NeMo Relay plugin.
- Python bindings and TypeScript bindings.
- A standalone OpenAI- and Anthropic-compatible demo proxy for local evaluation.
- Benchmark results and configurations for Terminal-Bench 2.1 and DeepSWE.
Where NVIDIA NeMo Switchyard fits
Switchyard fits gateways and agent harnesses that need to choose between efficient and more capable models during coding or other tool-driven work. Teams can embed libsy into a service they operate, use a documented gateway integration, or compare routing configurations locally with their own model pool.
Before deployment, teams should verify that routing actually changes the selected model, check gateway-specific algorithm support, and test provider formats, credentials, timeouts, and session identifiers. Evaluation should compare completed tasks and total run cost against fixed-model baselines. Judge calls, lost prompt caches, and repeated escalation can make routing more expensive; execution-based presets also lack useful signals on chat-only traffic.
Install
cargo install --locked switchyard-server --version 0.3.0Reviews of NVIDIA NeMo Switchyard
Be the first to share your experience with NVIDIA NeMo Switchyard.
Write a review
Loading review form…
Alternatives to NVIDIA NeMo Switchyard
AgentaAgenta is an open-source workspace for building AI agents by chatting, sharing them with a team, running them on schedules or app events, and tracing every run. MIT core, self-hosted with Docker or Kubernetes, or Agenta Cloud.
BifrostBifrost is an Apache-2.0 AI gateway written in Go. It unifies 20+ providers behind one OpenAI-compatible API with fallbacks, virtual keys, semantic caching, logging and an MCP gateway, run through NPX or Docker.
DifyDify by LangGenius combines a visual workflow builder, knowledge pipelines, sandboxed agents, model management and app APIs. It runs as Dify Cloud or self-hosted with Docker Compose under a modified Apache 2.0 license.