Search tools, AI agents, MCP servers
Submit a tool

Search iThub

Search tools, categories and collections. Type at least two characters.

Explore all tools
↑ ↓ to explore resultsEnter to open · Esc to close

NVIDIA SkillEvaluator

NVIDIA tool that validates AI agent skills and measures their effect with sandboxed with/without-skill agent runs

MLOps & LLMOpsOpen sourcev0.9.0

NVIDIA SkillEvaluator at a glance

Pricing
Open source
License
Open source
Version
v0.9.0November 13, 2026
Language
Python
Data location
Runs locally or in CI; static Tier 1 checks need no network model. LLM checks and Tier 3 send skill content to the configured provider and sandbox.
Maker
NVIDIA
Founded
2026

What is NVIDIA SkillEvaluator?

SkillEvaluator is an open-source Python tool from NVIDIA that checks AI agent skills and measures their effect on task results. Agent skills are folders of instructions and supporting files defined by the Agent Skills specification, built around a SKILL.md file. According to the README, SkillEvaluator starts with quality checks and then compares agent runs with and without a skill. It is part of the NVIDIA Verified Skills pipeline, and the method is described in an arXiv paper and an NVIDIA Developer blog post.

The tool is organised in three independent tiers. Tier 1 (validation) runs deterministic checks for schema, quality, security, PII, license, code integrity, Unicode, scripts and harness compatibility, with optional LLM judging. Tier 2 (deduplication) uses embedding similarity to find repeated guidance inside one skill and overlap across a skill collection. Tier 3 (live evaluation) runs real agent CLIs in sandboxes on evaluation cases, once with the skill and once without, and reports the difference as Skill Lift. Besides skills, the validator recognises plugins, rules and workflows, applying only the checks that fit each content type.

SkillEvaluator is distributed as the skillevaluator package on PyPI, installable with pip or uv tool install, from a source checkout with uv, and as a container. Optional extras keep the footprint small: llm, security, tier2, tier3, all and dashboard. It requires Python 3.12, 3.13 or 3.14; Docker is needed only for the container install and Tier 3 Docker-mode runs. Tier 1 fetches pinned SkillSpector and Gitleaks releases when missing, Semgrep is optional, and Tier 3 trials are executed by Harbor, an open-source agent evaluation framework. The README notes that the project downloads and installs additional third-party open-source software whose licenses users should review.

The latest release is SkillEvaluator 0.9.0, published on November 13, 2026, with a wheel, source archive, CI bundle, SHA256 checksums and an artifact manifest. Its notes describe evaluation dataset generation grounded in skill instructions, author intent or traces, version comparison with compare-versions and reusable no-skill baselines, token, judge-call, attempt and cost budgets, more detailed evidence in reports, grader qualification with tier3 validate-grader, opt-in Tier 2 and Tier 3 CI checks for GitHub and GitLab, an upgrade to Harbor 0.25.0 and experimental Hermes agent support. Several grading outcomes changed, so the notes say scores from older releases need re-evaluation before comparison.

SkillEvaluator is licensed under the Apache License 2.0, with copyright held by NVIDIA Corporation and affiliates. The README states a support level of Experimental: community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement, and issues go through GitHub. The tool itself is free; model calls and managed sandboxes used by LLM-backed checks and Tier 3 can incur charges with the chosen provider, while eligible NVIDIA Developer Program members can use NVIDIA Build endpoints for free prototyping under their access terms. Related listings on iThub include SkillOpt and LazySkills.

Key capabilities

  • Keyless quality check skillevaluator quality-check ./my-skill scores a skill out of 100 across four weighted categories without an API key, scanner, GPU or Docker. The bundled text-analyzer example scores 88.8/100 (B) in the README, and the passing threshold defaults to 70. The Tier 1 guide notes that this static result leaves publication evidence incomplete until security scanning and live evaluation have run.
  • Tier 1 security, PII and license checks Tier 1 runs SkillSpector static security scanning, a PII scan for credentials, secrets and local identifiers, license and dependency checks, Unicode and code-integrity checks and advisory AST-based linting of Python scripts. Gitleaks and SkillSpector are fetched as pinned, verified releases when missing; a required scanner that cannot run is reported as INCOMPLETE and the command exits nonzero.
  • LLM rubric and finding verification With a provider configured, rubric-eval applies a nine-criterion LLM-as-judge rubric, --llm adds LLM security analysis, and --llm-verify reviews each static finding in context. High-confidence false positives stay visible but are downgraded to INFO with the verifying provider and model recorded.
  • Harness compatibility checks Tier 1 includes checks that a skill is compatible with the agent harnesses it targets, alongside schema validation of the SKILL.md frontmatter. For plugins, rules and workflows the skill-only quality, lint and version checks are skipped and the content is validated against its own public contract.
  • Tier 2 semantic deduplication Tier 2 commands such as similarity-check, context-optimization-check and dedup-scan use an embeddings API to find redundant content inside a skill and, with --catalog, overlapping skills across a collection; intra-skill candidates are verified by a chat LLM. Both checks read local files only and need no external vector database, and a fully local setup with an OpenAI-compatible server is documented.
  • Tier 3 live agent evaluation Tier 3 runs agent CLIs on evaluation cases inside a sandbox, with and without the skill, using the same cases, inputs, model and grader for both arms. The agents guide lists Claude Code, Codex and OpenCode as supported agents and Hermes as experimental, with Docker as the default environment plus Kata, local and cloud backends provided by Harbor.
  • Skill Lift and per-dimension scores Results compare the skill-assisted run against its no-skill baseline. Skill Lift is the headline comparison, with pass@k as supporting reliability, and the benchmark card breaks scores down by security, correctness, discoverability, effectiveness and efficiency for each agent. A scored attempt fails when goal accuracy or security is below 1.0.
  • Evaluation dataset generation tier3 create-eval-dataset builds evaluation cases from substantive evals/EVAL.md guidance, a prompt file of task requests and outcomes, or matching retained traces. Since 0.9.0 grounding defaults to auto, production traces can be imported as redacted review candidates, and expectations are promoted only after explicit human review. Deterministic checks can be declared for script execution, workspace files and final responses.
  • Version comparison and baselines Retained skill versions can be compared with --compare-to and compare-versions, run history can be browsed, and a compatible no-skill baseline can be reused with --reuse-baseline. Version comparisons are kept separate from Skill Lift against the baseline.
  • Budgets and cost control Token, judge-call, attempt and priced-cost budgets can cap an evaluation. When a limit stops a run, reports keep the partial results, and token usage is shown per case and agent, with missing counters and reused-baseline usage identified.
  • Reports in several formats Runs write terminal, JSON, HTML, Markdown and SARIF reports, plus a BENCHMARK.md card for publication. Reports show met, unmet, unchecked and inapplicable expectations, dataset coverage, skill activation and next steps, and HTML charts work offline. The reports guide documents the JSON contract and the optional performance dashboard.
  • Repository CI for GitHub and GitLab skillevaluator ci init opens a setup pull request in a GitHub repository; GitLab CI/CD is also supported. Tier 1 runs as a required gate with no model key, while Tier 2 and Tier 3 can be added as advisory checks that run only when a pull request changes a skill. The repository guide covers pinned images, workflow versions and secrets.
  • Agent Skill for coding agents SkillEvaluator ships its own skill, added with npx skills add NVIDIA/SkillEvaluator to Claude Code, Codex or another agent. Asked "Is my skill any good?", the agent picks the tier and flags, runs the CLI locally and reports the verdict, scores and fixes, as shown in the agent guide.
  • Model providers LLM-backed stages use one provider selected with SKILL_EVAL_LLM_PROVIDER: openai, anthropic, nv_build, bedrock or openai-compatible, each with a default model that SKILL_EVAL_LLM_MODEL overrides. Tier 2 embeddings use an OpenAI-compatible API, by default text-embedding-3-small for OpenAI or nvidia/nemotron-3-embed-1b for NVIDIA Build.
  • Readiness checks with doctor skillevaluator doctor checks that scanners, providers, agents and the sandbox are ready before compute is spent, for example doctor --env-mode docker or doctor --tier 1. The README notes that it checks readiness, not inference success.
  • Combined pipeline command skillevaluator validate ./my-skill runs all three tiers for a skill, with autopilot creating one starter evaluation case when no dataset exists. --tiers 1,2 skips live evaluation and --tiers 1 keeps the run static. Tier 1 always gates the exit code, Tier 2 gates unless --no-block-on-dedup is set, and Tier 3 gates only with --block-on-agent-eval.

Where NVIDIA SkillEvaluator fits

Checking a new skill before sharing it

A skill author points skillevaluator tier1 ./my-skill --no-llm or quality-check at the skill directory and gets a schema, security, PII, license, quality and harness result with a numeric quality grade, all on the local machine without an API key. The terminal report lists findings and the JSON and HTML reports are written to reports/ by default.

Measuring whether a skill helps an agent

With a provider, agent and Docker sandbox configured, skillevaluator tier3 evaluate ./my-skill runs the evaluation cases with and without the skill on agents such as Claude Code, Codex or OpenCode. The report shows Skill Lift per agent and per dimension, so an author can see whether the skill raises correctness or efficiency against the no-skill baseline.

Gating skill changes in a repository

Teams that keep skills in Git run skillevaluator ci init to add the CI template. Every pull request then runs Tier 1 as a required check with no model key, and optional Tier 2 and Tier 3 evaluations run as advisory jobs when a skill changes, with results summarised per skill in the CI output.

Finding overlap in a skill collection

Maintainers of a large skill catalog run Tier 2 with --catalog ./skill-catalog.json to compare skills against each other. The report flags pairs whose guidance overlaps by embedding similarity and repeated content inside a single skill, and states when inter-skill comparison did not run.

Tracking a skill across versions

After revising a skill, compare-versions or --compare-to compares retained versions, while --reuse-baseline avoids paying again for an unchanged no-skill baseline. Budgets on tokens, judge calls and cost bound each run, and partial results are kept if a budget stops it.

Asking from inside a coding agent

With the SkillEvaluator skill installed in an agent, a user asks in plain language whether a skill is good. The skill selects tier 1, 2 or 3 and the right flags, runs the CLI on the user's machine and returns the verdict, quality score and the first fixes to make, with the same scores as the CI path.

Install

pip install skillevaluator

Reviews of NVIDIA SkillEvaluator

No reviews yet

Be the first to share your experience with NVIDIA SkillEvaluator.

Loading review form…

Found NVIDIA SkillEvaluator useful? Upvote it todayYour vote helps other IT teams find it in the daily ranking.

Similar to NVIDIA SkillEvaluator

Compare:NVIDIA SkillEvaluator vs SkillOptNVIDIA SkillEvaluator vs Scale AINVIDIA SkillEvaluator vs Weights & Biases

  1. SSkillOptSkillOpt optimizes reusable natural-language skills for LLM agents without changing model weights.MLOps & LLMOps2 views
  2. Scale AITraining data, RLHF, model evaluations and an enterprise agent platform from Scale AIMLOps & LLMOps60 views
  3. Weights & BiasesExperiment tracking, model registry and LLM evaluation for ML teamsMLOps & LLMOps23 views