Stop shipping untested agent skills: hands-on with NVIDIA's SkillEvaluator
Agent skills are the fastest-growing artifact in AI coding — and almost nobody tests them. NVIDIA’s SkillEvaluator scores a skill, checks it against a catalog, and runs a live agent with and without it to measure whether it helps. Here is the install, the first scored run, and a skill of our own taken from a 69.0 fail to a 92.5 pass.
- What it is: NVIDIA’s three-tier framework for evaluating AI agent skills — static quality scoring, deduplication against a catalog, and live agent trials that measure whether a skill actually helps.
- Why it is trending: 1,451 stars, 188 forks, 541 commits, 11 releases, and a spot on GitHub’s daily trending list on November 15, 2026 — the week teams started treating skills as code that needs a gate.
- Cost: free and open source (Apache-2.0). Tier 1 needs no account and no API key; Tiers 2–3 need a model provider and Docker.
- Verified in this tutorial: every command below ran in our sandbox — pip install, the bundled example at 86.2/100, our own skill from 69.0 to 92.5, and the full Tier 1 gate at 10 of 11 checks passed.

Agent skills went from a niche format to the default way teams package agent behavior in a single year. Anthropic publishes an official skills repository, community catalogs list thousands of them, and every coding agent now loads folders of instructions on demand. What almost nobody does is test those folders. A skill ships when someone writes a SKILL.md that reads well — and the first sign of trouble is an agent that ignores it, misuses its scripts, or leaks a secret at runtime.
NVIDIA's SkillEvaluator is the first systematic answer to that gap: a three-tier framework that scores a skill, checks it for duplication against a catalog, and — in the top tier — runs a live agent with and without the skill to measure whether it actually helps. It is version 0.9.0, Apache-2.0, with 1,451 stars and 188 forks, and it holds a spot on GitHub's daily trending list today, November 15, 2026. The method behind it is published as "Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills" (arXiv:2608.20614), which reports a mean composite Skill Lift of 0.2134 across 947 paired live trials — a skill that helps, measured rather than asserted.
This tutorial runs the tool end to end. You will install version 0.9.0, score NVIDIA's bundled example skill, write a deliberately weak skill of your own, watch it fail at 69.0 out of 100, fix exactly what the report tells you to fix, and watch it pass at 92.5. Then you will run the full Tier 1 gate and see where it stays strict. Every command and every quoted output below ran in our sandbox on November 15, 2026.
What you'll need#
- Python 3.12 or later (3.13 used here).
- uv or pip, and Git.
- No account, no API key, no GPU, no Docker for Tier 1 — the whole first tier runs offline on your machine.
- Tiers 2 and 3 need a model provider (NVIDIA Build offers free prototyping for Developer Program members) and Docker for the live-evaluation sandbox. Those tiers are covered in step 6, with the exact commands, but were not run in this sandbox.
1. Install SkillEvaluator#
Two paths, same CLI. The standalone install is what most readers want:
python3 -m venv .venv-skilleval
source .venv-skilleval/bin/activate
pip install skillevaluator==0.9.0
skillevaluator --version
Our run answered:
skillevaluator, version 0.9.0
Or run from a source checkout with uv — the path the README's quickstart uses, and the one we used for the examples below:
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv run --python 3.13 skillevaluator --version
uv provisions Python 3.13 and the dependencies on first run; on our machine that took under a minute after the clone. skillevaluator --help lists the surface: tier1, tier2, tier3, validate, health-check, doctor, models, and dashboard. Pin the version — 0.9.0 is what every output below comes from.
2. Score the bundled example#
The repo ships reference skills under src/skillevaluator/tier3/reference_skills/. Score the text-analyzer one with the keyless quality check — no API key, no scanner, no Docker:
uv run --python 3.13 skillevaluator quality-check \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-quality
Our run answered:
[PASS] Passed required checks
Tier 1: 1 of 1 checks passed; 0 failed, 0 incomplete, 0 skipped, 0 N/A
Publication eligibility: Incomplete
json report: reports/first-quality/skillevaluator-quality.json
The score itself sits in the JSON report: 86.2 out of 100, grade B, against a default pass threshold of 70. The four dimensions — weighted as the tool prints them, Correctness 35%, Discoverability 25%, Reliability 25%, Efficiency 15% — came back 75.0, 85.0, 95.0, and 100.0. Six findings, three medium and three low, all with a fix attached. Three of them:
- Description very long (207 chars, recommend 50-150). Fix: keep descriptions concise for progressive disclosure.
- Description doesn't mention WHEN to use this skill. Fix: add trigger context — "Use for…", "When you need to…".
- No prerequisites/requirements documented. Fix: document dependencies, API keys, or setup needed.
That is the shape of the tool: it does not grade vibes. Each deduction names the dimension, the file, and the exact change. "Publication eligibility: Incomplete" is also doing work — the quality score alone does not clear a skill for publication. The security scan and live evaluation still have to run; a static score is necessary, not sufficient.

3. Write a skill and watch it fail#
A bundled example only proves the tool runs. The real test is your own skill. Create a thin one — a PDF summarizer with a long, vague description, no metadata, no script table, and no examples:
mkdir -p my-skill/scripts
cat > my-skill/SKILL.md <<'EOF'
---
name: pdf-summarizer
description: This skill helps you summarize PDF files and documents of various kinds, extracting the key points, the main arguments, the important data, and producing a concise summary that you can read quickly instead of reading the whole document yourself every time.
---
# PDF Summarizer
Summarize PDFs. Run the script and read the output.
EOF
A minimal scripts/summarize.py that prints a line is enough for the scorer, which grades the skill's documentation, not its runtime behavior. Score it:
uv run --python 3.13 skillevaluator quality-check ../my-skill -r cli -o ../reports/my-skill-v1
Overall: 69.0/100 (Grade: D)
Skill Type: script-based
[FAIL] Validation failed
• Quality score 69.0/100 is below the required 70/100
Tier 1: 0 of 1 checks passed; 1 failed, 0 incomplete, 0 skipped, 0 N/A
Dimension scores: Correctness 65, Discoverability 55, Reliability 70, Efficiency 100. One error, fifteen warnings — and the warnings read like a code review from a strict colleague:
- No documented scripts in table format — add an "Available Scripts" table with Script, Purpose, and Arguments.
- Instructions don't mention
run_script— add explicitrun_script()call examples. - No examples provided; missing
metadata.authorandmetadata.tags. - Description is 256 characters, uses vague words, and is written in first/second person.
Note what failed hardest: Discoverability at 55. The description is the part an agent reads first when deciding whether to load the skill at all — a 256-character ramble with no trigger context is the single most expensive defect in the file.
4. Fix what the report says, then re-run#
Apply the fixes the report listed, nothing more: a short third-person description with a WHEN trigger, author and tags under metadata:, a prerequisites section, a scripts table, and run_script() examples:
---
name: pdf-summarizer
description: Summarize a PDF into key points and action items. Use when a reader needs the essentials of a long PDF without reading the whole file.
compatibility: Python 3.10+, no external dependencies
metadata:
author: Tutorial Demo
tags: [pdf, summarization, documents]
---
# PDF Summarizer
Summarize PDF files into key points and action items.
## Prerequisites
- Python 3.10 or later.
- No external dependencies; the script uses the standard library only.
## Agent Instructions
1. Read this SKILL.md to understand capabilities.
2. Run `scripts/summarize.py` with the PDF path using run_script().
3. Return the summary to the user.
## Available Scripts
| Script | Purpose | Arguments |
| --- | --- | --- |
| `scripts/summarize.py` | Print a short summary of a PDF | `<pdf-path>` |
## Examples
### Usage
```bash
# Summarize a PDF with run_script()
run_script("scripts/summarize.py", ["report.pdf"])
```
Re-run the same command:
uv run --python 3.13 skillevaluator quality-check ../my-skill -r cli -r json -o reports/my-skill-v2
Overall: 92.5/100 (Grade: A)
[PASS] Validation passed
Tier 1: 1 of 1 checks passed; 0 failed, 0 incomplete, 0 skipped, 0 N/A
Correctness 100, Discoverability 95, Reliability 75, Efficiency 100 — a 23.5-point jump from documentation changes alone. Five warnings remain, and they are the honest remainder: no "## Purpose" section, no mention of error handling, the script may lack error handling, no limitations documented. The tool tells you what "good enough" costs next, and leaves the call to you. This fix-and-rescore loop is the workflow: write the skill, score it, fix the top findings, score again, and stop when the remaining warnings are ones you accept deliberately.

5. Run the full Tier 1 gate#
quality-check scores one dimension. validate runs the whole Tier 1 gate — schema and repository governance, security, PII, license, secrets, code risk, lint, and harness compatibility — and it is stricter about layout. Run against the bare folder first:
uv run --python 3.13 skillevaluator validate ../my-skill --tiers 1 -r cli -o reports/my-skill-validate
It failed: "Skill not in standard location (skills/ or team-skills/)". validate expects a repository layout, not a loose folder. Move the skill into one and re-run:
mkdir -p repo/skills
cp -r my-skill repo/skills/pdf-summarizer
uv run --python 3.13 skillevaluator validate ../repo/skills/pdf-summarizer --tiers 1 -r cli -o reports/repo-validate
│ Schema & Repository Governance │ PASS │ 10 checks passed │
│ PII Scan │ PASS │ 2 checks passed │
│ License Compliance │ PASS │ 1 checks passed │
│ Secrets Detection │ PASS │ 1 checks passed │
│ QUALITY │ PASS │ 1 checks passed; quality A │
Tier 1: 10 of 11 checks passed; 0 failed, 1 incomplete, 0 skipped, 1 N/A
Overall: 92.5/100 (Grade: A)
One check stayed incomplete: the Security Scan, because SkillSpector 2.13.4 could not be set up in this sandbox — its download of markdown-it-py==4.3.0 failed on the sandbox proxy. The tool says so plainly and offers the remedies: check network access, install SkillSpector manually, or pass --skip security. The exit code stays 1 while anything is incomplete, and that is the correct behavior for a gate: it refuses to bless what it did not check. The run also prints a harness-compatibility table — Claude Code, Codex, Copilot CLI, Hermes, OpenClaw, and OpenCode all passed for this skill — which is the closest thing to a portability test in the suite.
Two flags worth knowing: --min-score moves the pass threshold (default 70), and --no-llm keeps Tier 1 fully deterministic — no model calls, no key, same score every run.
6. What Tiers 2 and 3 add#
Tier 1 answers "is this skill well written?" Tiers 2 and 3 answer the two questions that follow, and both need a model provider and — for Tier 3 — Docker. They were not run in this sandbox, so treat the commands below as documented, not verified here.
Tier 2 — Deduplication. uv run --extra tier2 skillevaluator tier2 ./my-skill embeds the skill's guidance and compares it against a catalog (--catalog ./skill-catalog.json for inter-skill comparison), reporting semantic overlap. The point is practical: in a repo with dozens of skills, two of them usually cover the same ground, and the agent loads both. Tier 2 finds the pair.
Tier 3 — Live evaluation. uv run --extra tier3 skillevaluator tier3 ./my-skill runs the agent on real tasks with and without the skill inside a sandbox (Docker by default; Harbor powers the runs) and reports Skill Lift — the skill's added value for a fixed task, harness, workspace, and scorer. This is the measurement the paper is built around: across 947 paired cases from 58 of 64 production skills, mean composite Skill Lift was 0.2134, positive in 72.8% of cases — and the paper's warning is that scan-only gates and LLM judges barely agree (Spearman ρ = 0.14), so a skill can look clean on paper and still not help an agent. Tier 3 is how you find out.
To enable the model-backed tiers, set a provider. NVIDIA Build is the documented path for free prototyping (eligible Developer Program members):
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY=<your-key>
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-build
That runs the rubric evaluation — model-backed scores and feedback on the same example skill, saved to reports/first-build/skillevaluator-rubric.json. Before committing to the full pipeline, skillevaluator doctor --env-mode docker checks readiness; skillevaluator ci init opens a setup pull request that runs the gate on changed skills. The dashboard command, which compares saved JSON reports in tables, needs the dashboard extra (uv run --extra dashboard skillevaluator dashboard) — it is not installed by default.
When to use this vs alternatives#
Use SkillEvaluator when skills are a shared asset. The moment more than one person writes skills, or one repo holds more than a handful, an untested skill is a silent tax: agents load it, misuse it, and nobody can say why. The keyless Tier 1 check takes seconds and belongs in CI next to the linter.
Use SkillSpector when the question is security, not quality. NVIDIA's SkillSpector scans a skill for prompt injection, secret exfiltration, and supply-chain risk — Tier 1 fetches it as part of the security check, and we covered it on its own in October. A skill can score 92.5 on quality and still be unsafe; run both.
Use an LLM rubric when you want judgment, not just structure. tier1 --llm adds model-backed rubric scores to the deterministic checks. It costs a model call and needs a provider key, but it catches the things static rules cannot — instructions that are technically complete and practically confusing.
Use live evaluation when the claim is "this skill helps". Only Tier 3 measures that, with paired trials and a Skill Lift number. If a skill exists to make an agent faster or more accurate on a real task, a static A grade is not evidence — run the paired trial.
Skip it for a skill only you will ever use. A personal scratch skill does not need a gate. But the check is free and takes seconds, so even there it is the cheapest way to catch a missing script table or a description the agent will never trigger on.
The takeaway#
Agent skills are code that happens to be written in Markdown, and they have been shipping with the testing practices of a README — none. SkillEvaluator treats them like what they are: versioned artifacts with a schema, a score, and a gate. The useful part is not the grade; it is the fix list attached to every deduction, and the fact that the cheapest tier needs no key, no GPU, and no Docker. Our demo skill went from 69.0 to 92.5 in one edit cycle, and the remaining five warnings are a to-do list, not a mystery.
The workflow to keep: score every skill before it lands, keep the threshold at 70 until your catalog earns a higher one, and put validate in CI so the gate runs on every pull request. When a skill matters enough to claim it improves results, run Tier 3 and ask for the Skill Lift number — that is the only score that says the skill works, rather than that it reads well.
Sources: the SkillEvaluator repository, NVIDIA's SkillEvaluator documentation, and the paper "Evaluating Skills, Not Just Agents" (arXiv:2608.20614). Star counts and trending placement as observed November 15, 2026; version 0.9.0 and all command outputs verified by running them in our sandbox.