eval-run

Executes a skill against test cases, collects artifacts, scores with judges, and generates an HTML report. Orchestrates via scripts: preflight checks for stale artifacts, workspace creation with isolated per-case directories, resolution of tool-interception handlers, headless skill execution (case mode: once per case with case-specific arguments; batch mode: single invocation via batch.yaml), artifact collection into per-case dirs, scoring with four judge types (builtin, inline checks, LLM prompts, external modules), optional pairwise comparison against a baseline for regression detection, and report generation. Supports concurrent case execution via the parallelism setting, tool interception for AskUserQuestion and external APIs, configurable reasoning effort, and a --gold flag to save outputs as gold references. Background-launches execute.py and monitors progress; persists state via state.py and leads its analysis with a decisive recommendation.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Run a skill or prompt evaluation end-to-end against a configured dataset: preflight-clean state, prepare an isolated workspace, execute cases headlessly, collect artifacts, score them with configured judges (deterministic and LLM), optionally run pairwise comparison against a baseline to detect regressions, and produce a decisive, evidence-backed analysis plus an HTML report. Requires an eval.yaml config produced by /eval-analyze; the skill orchestrates the pipeline scripts rather than reimplementing their work.

Identity

Functions

execute verify

Success

• Discovers or accepts an eval.yaml config, runs preflight, and sets up an isolated workspace with the resolved test cases before executing.
• Executes the eval headlessly in the correct auto-detected mode (skill mode with arguments, or prompt mode) and confirms run_result.json exit_code is zero before scoring.
• Collects per-case artifacts and runs every configured judge, honoring --no-llm-judges by skipping LLM judges while still running deterministic ones.
• Produces summary.yaml with per-judge means/pass-rates and per-case results, and when --baseline is given adds a pairwise regression comparison.
• Writes a decisive analysis.md leading with a self-contained Recommendation and generates the HTML report.

Optimization Targets

task_success verifier_backed score.py @ 1559af5→

evidence_completeness judge analyze-results.md @ 1559af5→

Invariants

Must Preserve

• Orchestrate by calling the pipeline scripts (preflight, workspace, execute, collect, score, report); never duplicate or reimplement their scoring/execution logic.
• Launch execute.py in the background with no output redirection (no >, |, tee, or 2>&1) and poll until completion; do not end the turn while it runs.
• Fail fast: if execution produces no artifacts or a non-zero exit_code, report it and stop rather than scoring empty outputs.
• Never read large artifact files into context; rely on summary.yaml and delegate content analysis to agents.
• When inputs.tools is configured, resolve every tool handler (input_filters for Bash) before executing; do not skip this mandatory step.
• Honor run isolation: do not overwrite a prior run's results without cleaning, and persist state via state.py at each step.

Fixed Context

toolsRead, Write, Edit, Bash, Glob, Grep, Agent, Skill, AskUserQuestion

clipython3

knowledgerepository_contentpublic, task_inputtask_private, tool_outputtask_private

Traceability

Skill

↗skills/eval-run/SKILL.md

Supporting

• ↗skills/eval-run/scripts/execute.py
• ↗skills/eval-run/scripts/score.py
• ↗skills/eval-run/scripts/collect.py
• ↗skills/eval-run/scripts/report.py
• ↗skills/eval-run/scripts/preflight.py
• ↗skills/eval-run/prompts/analyze-results.md
• ↗skills/eval-run/prompts/comparison-judge.md
• ↗skills/eval-run/references/execution-modes.md

Diagram

Arguments

/eval-run [--config <path>] [--model <model>] [--run-id <id>] [--baseline <run-id>] [--cases <id> ...] [--no-llm-judges] [--gold] [--effort <level>] [--subagent-model <model>] [--skill <name>]

Argument	Required	Default	Description
--config		auto-discover	Path to eval config. If missing, bootstraps via /eval-analyze.
--model		models.skill from config	Model for skill execution. Required if models.skill is unset in eval.yaml.
--subagent-model		models.subagent, falls back to skill model	Model for subagents (e.g., claude-sonnet-4-6 while main is claude-opus-4-7).
--run-id		YYYY-MM-DD-<model>	Identifier for this run.
--cases		-	Exact case IDs to run (space-separated). Defaults to all cases.
--baseline		-	Previous run to compare against for regression detection via pairwise comparison. Must exist under the same eval-name directory.
--no-llm-judges		false	Skip LLM judges (prompt, prompt_file, LLM builtins). Run deterministic judges only (check, Python builtins, external code).
--gold		false	Save collected outputs as gold reference files in the dataset case dirs after the run.
--effort		runner.effort from config	Claude Code reasoning effort level (Claude Code only; ignored by other runners).
--skill		from config	Override the skill to test.

Usage

/eval-run --model claude-opus-4-6
/eval-run --model claude-opus-4-6 --baseline 2026-05-01-opus
/eval-run --cases case-001 case-002 --no-llm-judges
/eval-run --gold