What is NVIDIA SkillEvaluator?
SkillEvaluator is an open-source Python tool from NVIDIA that checks AI agent skills and measures their effect on task results. Agent skills are folders of instructions and supporting files defined by the Agent Skills specification, built around a SKILL.md file. According to the README, SkillEvaluator starts with quality checks and then compares agent runs with and without a skill. It is part of the NVIDIA Verified Skills pipeline, and the method is described in an arXiv paper and an NVIDIA Developer blog post.
The tool is organised in three independent tiers. Tier 1 (validation) runs deterministic checks for schema, quality, security, PII, license, code integrity, Unicode, scripts and harness compatibility, with optional LLM judging. Tier 2 (deduplication) uses embedding similarity to find repeated guidance inside one skill and overlap across a skill collection. Tier 3 (live evaluation) runs real agent CLIs in sandboxes on evaluation cases, once with the skill and once without, and reports the difference as Skill Lift. Besides skills, the validator recognises plugins, rules and workflows, applying only the checks that fit each content type.
SkillEvaluator is distributed as the skillevaluator package on PyPI, installable with pip or uv tool install, from a source checkout with uv, and as a container. Optional extras keep the footprint small: llm, security, tier2, tier3, all and dashboard. It requires Python 3.12, 3.13 or 3.14; Docker is needed only for the container install and Tier 3 Docker-mode runs. Tier 1 fetches pinned SkillSpector and Gitleaks releases when missing, Semgrep is optional, and Tier 3 trials are executed by Harbor, an open-source agent evaluation framework. The README notes that the project downloads and installs additional third-party open-source software whose licenses users should review.
The latest release is SkillEvaluator 0.9.0, published on November 13, 2026, with a wheel, source archive, CI bundle, SHA256 checksums and an artifact manifest. Its notes describe evaluation dataset generation grounded in skill instructions, author intent or traces, version comparison with compare-versions and reusable no-skill baselines, token, judge-call, attempt and cost budgets, more detailed evidence in reports, grader qualification with tier3 validate-grader, opt-in Tier 2 and Tier 3 CI checks for GitHub and GitLab, an upgrade to Harbor 0.25.0 and experimental Hermes agent support. Several grading outcomes changed, so the notes say scores from older releases need re-evaluation before comparison.
SkillEvaluator is licensed under the Apache License 2.0, with copyright held by NVIDIA Corporation and affiliates. The README states a support level of Experimental: community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement, and issues go through GitHub. The tool itself is free; model calls and managed sandboxes used by LLM-backed checks and Tier 3 can incur charges with the chosen provider, while eligible NVIDIA Developer Program members can use NVIDIA Build endpoints for free prototyping under their access terms. Related listings on iThub include SkillOpt and LazySkills.
Key capabilities
- Keyless quality check
skillevaluator quality-check ./my-skillscores a skill out of 100 across four weighted categories without an API key, scanner, GPU or Docker. The bundledtext-analyzerexample scores 88.8/100 (B) in the README, and the passing threshold defaults to 70. The Tier 1 guide notes that this static result leaves publication evidence incomplete until security scanning and live evaluation have run. - Tier 1 security, PII and license checks Tier 1 runs SkillSpector static security scanning, a PII scan for credentials, secrets and local identifiers, license and dependency checks, Unicode and code-integrity checks and advisory AST-based linting of Python scripts. Gitleaks and SkillSpector are fetched as pinned, verified releases when missing; a required scanner that cannot run is reported as INCOMPLETE and the command exits nonzero.
- LLM rubric and finding verification With a provider configured,
rubric-evalapplies a nine-criterion LLM-as-judge rubric,--llmadds LLM security analysis, and--llm-verifyreviews each static finding in context. High-confidence false positives stay visible but are downgraded to INFO with the verifying provider and model recorded. - Harness compatibility checks Tier 1 includes checks that a skill is compatible with the agent harnesses it targets, alongside schema validation of the
SKILL.mdfrontmatter. For plugins, rules and workflows the skill-only quality, lint and version checks are skipped and the content is validated against its own public contract. - Tier 2 semantic deduplication Tier 2 commands such as
similarity-check,context-optimization-checkanddedup-scanuse an embeddings API to find redundant content inside a skill and, with--catalog, overlapping skills across a collection; intra-skill candidates are verified by a chat LLM. Both checks read local files only and need no external vector database, and a fully local setup with an OpenAI-compatible server is documented. - Tier 3 live agent evaluation Tier 3 runs agent CLIs on evaluation cases inside a sandbox, with and without the skill, using the same cases, inputs, model and grader for both arms. The agents guide lists Claude Code, Codex and OpenCode as supported agents and Hermes as experimental, with Docker as the default environment plus Kata, local and cloud backends provided by Harbor.
- Skill Lift and per-dimension scores Results compare the skill-assisted run against its no-skill baseline. Skill Lift is the headline comparison, with pass@k as supporting reliability, and the benchmark card breaks scores down by security, correctness, discoverability, effectiveness and efficiency for each agent. A scored attempt fails when goal accuracy or security is below 1.0.
- Evaluation dataset generation
tier3 create-eval-datasetbuilds evaluation cases from substantiveevals/EVAL.mdguidance, a prompt file of task requests and outcomes, or matching retained traces. Since 0.9.0 grounding defaults toauto, production traces can be imported as redacted review candidates, and expectations are promoted only after explicit human review. Deterministic checks can be declared for script execution, workspace files and final responses. - Version comparison and baselines Retained skill versions can be compared with
--compare-toandcompare-versions, run history can be browsed, and a compatible no-skill baseline can be reused with--reuse-baseline. Version comparisons are kept separate from Skill Lift against the baseline. - Budgets and cost control Token, judge-call, attempt and priced-cost budgets can cap an evaluation. When a limit stops a run, reports keep the partial results, and token usage is shown per case and agent, with missing counters and reused-baseline usage identified.
- Reports in several formats Runs write terminal, JSON, HTML, Markdown and SARIF reports, plus a
BENCHMARK.mdcard for publication. Reports show met, unmet, unchecked and inapplicable expectations, dataset coverage, skill activation and next steps, and HTML charts work offline. The reports guide documents the JSON contract and the optional performance dashboard. - Repository CI for GitHub and GitLab
skillevaluator ci initopens a setup pull request in a GitHub repository; GitLab CI/CD is also supported. Tier 1 runs as a required gate with no model key, while Tier 2 and Tier 3 can be added as advisory checks that run only when a pull request changes a skill. The repository guide covers pinned images, workflow versions and secrets. - Agent Skill for coding agents SkillEvaluator ships its own skill, added with
npx skills add NVIDIA/SkillEvaluatorto Claude Code, Codex or another agent. Asked "Is my skill any good?", the agent picks the tier and flags, runs the CLI locally and reports the verdict, scores and fixes, as shown in the agent guide. - Model providers LLM-backed stages use one provider selected with
SKILL_EVAL_LLM_PROVIDER:openai,anthropic,nv_build,bedrockoropenai-compatible, each with a default model thatSKILL_EVAL_LLM_MODELoverrides. Tier 2 embeddings use an OpenAI-compatible API, by defaulttext-embedding-3-smallfor OpenAI ornvidia/nemotron-3-embed-1bfor NVIDIA Build. - Readiness checks with doctor
skillevaluator doctorchecks that scanners, providers, agents and the sandbox are ready before compute is spent, for exampledoctor --env-mode dockerordoctor --tier 1. The README notes that it checks readiness, not inference success. - Combined pipeline command
skillevaluator validate ./my-skillruns all three tiers for a skill, with autopilot creating one starter evaluation case when no dataset exists.--tiers 1,2skips live evaluation and--tiers 1keeps the run static. Tier 1 always gates the exit code, Tier 2 gates unless--no-block-on-dedupis set, and Tier 3 gates only with--block-on-agent-eval.
Where NVIDIA SkillEvaluator fits
Checking a new skill before sharing it
A skill author points skillevaluator tier1 ./my-skill --no-llm or quality-check at the skill directory and gets a schema, security, PII, license, quality and harness result with a numeric quality grade, all on the local machine without an API key. The terminal report lists findings and the JSON and HTML reports are written to reports/ by default.
Measuring whether a skill helps an agent
With a provider, agent and Docker sandbox configured, skillevaluator tier3 evaluate ./my-skill runs the evaluation cases with and without the skill on agents such as Claude Code, Codex or OpenCode. The report shows Skill Lift per agent and per dimension, so an author can see whether the skill raises correctness or efficiency against the no-skill baseline.
Gating skill changes in a repository
Teams that keep skills in Git run skillevaluator ci init to add the CI template. Every pull request then runs Tier 1 as a required check with no model key, and optional Tier 2 and Tier 3 evaluations run as advisory jobs when a skill changes, with results summarised per skill in the CI output.
Finding overlap in a skill collection
Maintainers of a large skill catalog run Tier 2 with --catalog ./skill-catalog.json to compare skills against each other. The report flags pairs whose guidance overlaps by embedding similarity and repeated content inside a single skill, and states when inter-skill comparison did not run.
Tracking a skill across versions
After revising a skill, compare-versions or --compare-to compares retained versions, while --reuse-baseline avoids paying again for an unchanged no-skill baseline. Budgets on tokens, judge calls and cost bound each run, and partial results are kept if a budget stops it.
Asking from inside a coding agent
With the SkillEvaluator skill installed in an agent, a user asks in plain language whether a skill is good. The skill selects tier 1, 2 or 3 and the right flags, runs the CLI on the user's machine and returns the verdict, quality score and the first fixes to make, with the same scores as the CI path.
Install
pip install skillevaluatorReviews of NVIDIA SkillEvaluator
Be the first to share your experience with NVIDIA SkillEvaluator.
Write a review
Loading review form…
Found NVIDIA SkillEvaluator useful? Upvote it todayYour vote helps other IT teams find it in the daily ranking.






