Skip to content

eval-review

Interactive human-in-the-loop review of evaluation results. Loads summary.yaml and any /eval-run analysis.md, presents judge scores and skill outputs case by case, collects qualitative feedback, and delegates transcript analysis to Explore sub-agents to identify inefficiencies (roundabout paths, multiple approaches, unnecessary tools, wasted turns). Identifies judge-human alignment gaps and suggests new judge candidates, persists feedback to review.yaml (keyed by case directory name for /eval-optimize and /eval-mlflow to consume), and proposes targeted SKILL.md edits as before/after diffs grounded in feedback evidence -- applied only with explicit approval. Complements /eval-optimize (automated) by catching tone, intent, and UX issues that judges cannot measure.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Guide an interactive review of an eval run: present judge scores and skill outputs, collect human feedback on what judges missed, persist that feedback as review.yaml, and propose targeted, evidence-grounded improvements to the artifact under test (SKILL.md or a prompt-mode artifact) and to the judge config.

Identity

Functions
review
Success
  • Loads the specified run's summary, eval.yaml, and per-case results, and presents pass rates and case-level judge scores with rationale.
  • Collects and persists human feedback to $AGENT_EVAL_RUNS_DIR/<eval-name>/<id>/review.yaml with feedback keys matching case directory names exactly.
  • Identifies judge-human alignment and systematic vs edge-case patterns across the reviewed feedback.
  • Proposes specific before/after edits to the artifact under test, each grounded in cited case feedback, and applies them only after explicit user approval.

Optimization Targets

task_success judge review-results.md @ 1559af5
evidence_completeness judge review-results.md @ 1559af5

Invariants

Must Preserve
  • Do not edit the artifact under test without explicit user approval; propose diffs, do not impose them.
  • Keep human feedback separate from judge scores; the skill's value is catching what judges miss.
  • Write review.yaml directly with the Write tool (not state.py) and keep feedback keys identical to case directory names so /eval-optimize and /eval-mlflow can consume them.
  • Do not flood context: summarize outputs and delegate large transcript analysis to an Agent rather than loading full files.
  • Ground every proposed change and finding in concrete case evidence; do not invent issues unsupported by feedback.
Fixed Context
toolsRead, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion, Skill
clipython3
knowledgerepository_contentpublic, tool_outputtask_private, task_inputtask_private

Traceability

Skill
skills/eval-review/SKILL.md
Supporting

Diagram

eval-review diagram

Arguments

/eval-review --run-id <id> [--config <path>] [--cases <name> ...]
Argument Required Default Description
--run-id - Which eval run to review.
--config auto-discover Path to eval config.
--cases - Exact case directory names to review (space-separated). Defaults to all cases.

Usage

/eval-review --run-id 2026-05-01-opus
/eval-review --run-id 2026-05-01-opus --cases case-003 case-005