eval-compare

Cross-model / cross-run comparison report generator. Takes a directory of eval run artifacts (summary.yaml, run_result.json, optional report.html and anova.json) and produces a self-contained, tabbed HTML comparison report: per-model cards, quality/cost tables, per-case score breakdowns, and embedded copies of each run's original report for iframe drill-down. Runs compare.py in two phases -- discover (recursively find every subdirectory containing a summary.yaml, aggregating repeated models as averages with min/max ranges) and generate (emit index.html plus per-run report.html copies) -- then replaces the placeholder analysis sections (Bottom Line verdict, Where Each Model Shined, Shared Weaknesses, Recommendations) with prose grounded in the per-case scores and cost data. Adds Best Value / Highly Variable / Not Viable badges under mutual-exclusivity rules, degrades gracefully when run_result.json or report.html is missing, and surfaces an ANOVA/Pareto Statistical Significance section automatically when an anova.json (written by /eval-anova) is present.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Take a directory of eval run artifacts (summary.yaml, run_result.json, optional report.html and anova.json), discover and aggregate runs per model, and produce a self-contained tabbed HTML comparison report whose LLM-written analysis sections (verdict, badges, strengths, shared weaknesses, recommendations) are grounded in the per-case scores and cost data.

Identity

Functions

analyze generate

Success

• Runs the discover and generate subcommands of compare.py to find every subdirectory containing summary.yaml and emit index.html plus per-run report.html copies for iframe embedding.
• Replaces the generated placeholder sections (Bottom Line verdict, Where Each Model Shined, Shared Weaknesses, Recommendations) with analysis grounded in each run's summary.yaml per-case scores, judge breakdowns, and cost data.
• Adds Best Value / Highly Variable / Not Viable badges only when they clearly apply, honoring the mutual-exclusivity rules (a highly variable model is never Best Value).
• Includes runs with missing run_result.json or report.html via graceful degradation, and aggregates repeated models as averages with min/max ranges.

Optimization Targets

task_success judge SKILL.md @ 1559af5→

evidence_completeness judge SKILL.md @ 1559af5→

Invariants

Must Preserve

• Do not modify or write to any input/source file; only write to the output directory.
• Do not run evaluations or compute statistics -- render pre-computed anova.json numbers only, never import scipy/statsmodels/pingouin.
• Do not explore the input directory manually (no ls/find); let compare.py handle all run discovery.
• Do not invent findings, badges, or verdicts unsupported by the per-case scores and cost data.
• Preserve graceful degradation and per-model aggregation (averages with min/max ranges) so partial or repeated runs are still included.

Fixed Context

toolsRead, Write, Edit, Bash, Glob, Grep, AskUserQuestion

clipython3

knowledgerepository_contentpublic, task_inputtask_private, tool_outputtask_private

Traceability

Skill

↗skills/eval-compare/SKILL.md

Supporting

• ↗skills/eval-compare/scripts/compare.py

Diagram

Arguments

/eval-compare <input-dir> [--output <path>] [--title <text>] [--overview <text>]

Argument	Required	Default	Description
input-dir		-	Directory scanned recursively for eval runs -- every subdirectory containing a summary.yaml is discovered and aggregated per model.
--output		<input-dir>/comparison-report	Output directory for the HTML report (index.html + per-run report.html copies).
--title		Model Comparison	Report title.
--overview		-	Optional context paragraph shown at the top of the report (section omitted if absent).

Usage

/eval-compare eval/runs/my-eval
/eval-compare eval/runs/my-eval --title "Opus vs Sonnet"