eval-anova

Design-of-Experiments (DoE) evaluation with ANOVA. Reads a matrix: block in eval.yaml (factors such as model, thinking-effort, or prompt, plus a replications count), designs the full-factorial grid with a cost estimate (--dry-run), then fans /eval-run out once per matrix cell (condition x replication) -- each cell landing as a standard run with its own summary.yaml tagged by a condition.json. It is not its own executor: eval-run stays the single-condition primitive. analyze.py then computes each case composite via the harness's canonical reward composition and runs repeated-measures ANOVA (single-factor) or a mixed-effects model (multi-factor), restricted to cases present under every condition, plus a cost/quality Pareto frontier, into anova.json. /eval-compare renders the cross-condition comparison (surfacing the statistics section from anova.json) and report.py gives a statistics-forward deep view (condition means, F / p / effect size, per-case matrix). Because the stats read standard summary.yaml runs, --analyze-only can re-analyze runs produced elsewhere (e.g. a CI fan-out) without re-executing.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Run a full-factorial Design-of-Experiments comparison of agent configurations (models, thinking-effort, prompts, or other factors) across a fixed set of test cases by fanning /eval-run out over the matrix, then analyze the resulting standard runs with repeated-measures / mixed-effects ANOVA (F, p, effect size) and a cost/quality Pareto frontier, and render the comparison and statistics reports.

Identity

Functions

orchestrate analyze

Success

• Reads the eval.yaml matrix: block, expands the full-factorial grid (conditions x cases x replications), and with --dry-run prints the design and a cost estimate without executing.
• Drives /eval-run once per condition x replication so each cell lands as a standard run with its own summary.yaml, stamped with a condition.json recording its factor levels.
• Computes each case composite via the canonical harness reward composition (compose_reward) and writes anova.json containing the ANOVA result (method, F, p, effect size, significance), condition summaries, Pareto frontier, and per-case matrix.
• Selects repeated-measures ANOVA for single-factor designs and the mixed-effects model for multi-factor designs, restricting the analysis to cases present under every condition.
• Renders the /eval-compare comparison report and the statistics-forward report.py deep view from on-disk artifacts; --analyze-only re-analyzes existing runs (including externally produced ones) without re-executing.

Optimization Targets

task_success deterministic

Invariants

Must Preserve

• eval-run stays the single-condition primitive: eval-anova only loops it and must not re-implement workspace/execute/collect/score.
• Repeated-measures ANOVA assumes the same cases run under every condition; keep the design balanced by restricting to common cases and recording excluded cases rather than silently dropping them.
• Compute case composites via the canonical harness reward composition (compose_reward honouring eval.yaml reward:, else boolean-gate + normalised-numeric average); do not invent a scoring scheme.
• Do not auto-select plain one-way ANOVA when cases are reused across conditions.
• Report renderers (report.py, /eval-compare) read only on-disk summary.yaml / anova.json artifacts and never re-run the experiment.
• A failed matrix cell is logged and skipped so a partial matrix still yields an analysis over the cells that succeeded.

Fixed Context

toolsRead, Bash, Write

clipython3

knowledgerepository_contentpublic, task_inputtask_private, tool_outputtask_private

Traceability

Skill

↗skills/eval-anova/SKILL.md

Supporting

• ↗skills/eval-anova/scripts/orchestrate.py
• ↗skills/eval-anova/scripts/analyze.py
• ↗skills/eval-anova/scripts/report.py
• ↗skills/eval-anova/scripts/design.py
• ↗skills/eval-anova/references/matrix-schema.md
• ↗skills/eval-anova/prompts/interpret-anova.md

Diagram

Arguments

/eval-anova --config <path> [--dry-run] [--analyze-only] [--cases <id> ...] [--output <dir>] [--no-report]

Argument	Required	Default	Description
--config		-	Path to the eval.yaml containing the matrix: block.
--dry-run		false	Print the factorial design and a cost estimate without executing any runs.
--analyze-only		false	Skip execution; re-run ANOVA + Pareto over existing runs (including externally produced ones) and re-render the report.
--cases		-	Restrict the experiment to specific case IDs (space-separated). Defaults to all cases in the dataset.
--output		-	Override the runs/output directory for this experiment.
--no-report		false	Run and analyze but skip the /eval-compare report render.

Usage

/eval-anova --config eval.yaml
/eval-anova --config eval.yaml --dry-run
/eval-anova --config eval.yaml --analyze-only