eval-optimize

Automated skill improvement loop that acts autonomously (no per-case human input). Runs evaluation, identifies judge failures from summary.yaml, reads execution transcripts and failing case outputs via Explore sub-agents to trace root causes to specific SKILL.md instructions, makes surgical edits grounded in evidence, re-runs evaluation with regression baseline checks (targeting failing cases first, then a full run), handles regressions (continue if minor, revert if major), and iterates up to a configurable maximum. Also reads human feedback from review.yaml (from /eval-review) and MLflow annotations to prioritize issues flagged by humans over automated judge failures. Never edits judges, eval.yaml, or builtin judge code -- only the skill under test. Stops when all judges pass or max iterations is reached, then suggests logging results to MLflow.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Autonomously improve a skill (or prompt-mode artifact) by looping over eval results: identify which judges failed and why from rationale and transcripts, form grounded hypotheses, apply surgical edits to the SKILL.md under test, re-run the eval, and check for regressions until judges pass or the max-iteration limit is reached.

Identity

Functions

orchestrate transform

Success

• Diagnoses each failing judge from summary.yaml rationale, transcripts, and case outputs before editing, noting judge_type (builtin/check/llm/code).
• Applies surgical, evidence-grounded edits to the target artifact under test (SKILL.md or prompt-mode artifact), not broad rewrites.
• Re-runs the eval with a --baseline flag and confirms targeted failures now pass while previously passing cases/judges do not regress.
• Iterates until all judges pass or max-iterations is hit, then reports which edits fixed which failures and the final summary.yaml scores.

Optimization Targets

task_success verifier_backed score.py @ 1559af5→

Invariants

Must Preserve

• Every edit must be grounded in a specific failure with evidence from judge rationale and transcripts -- never make broad, generic changes.
• Do not modify test cases, judges, or eval.yaml; the eval harness is ground truth. Never edit builtin judge code -- suggest adjusting arguments: instead.
• Check for regressions after every edit; a fix that breaks other cases is not a fix, and must be reverted or reframed.
• Stop after max-iterations rather than looping forever, and report what could not be fixed.
• Only edit the artifact under test (the SKILL.md or prompt-mode artifact); make minimal, surgical changes and do not rewrite working sections.

Fixed Context

toolsRead, Write, Edit, Bash, Glob, Grep, Agent, Skill, AskUserQuestion

clipython3

knowledgerepository_contentpublic, tool_outputtask_private, task_inputtask_private

Traceability

Skill

↗skills/eval-optimize/SKILL.md

Diagram

Arguments

/eval-optimize [--config <path>] [--model <model>] [--max-iterations <N>] [--run-id <id>] [--target-judge <name>]

Argument	Required	Default	Description
--config		auto-discover	Path to eval config.
--model		models.skill from config	Model for skill execution. Pass the same model on every iteration for comparable results.
--max-iterations		3	Maximum optimization iterations before stopping.
--run-id		auto-generated	Base run ID. Iterations append -iter-N.
--target-judge		-	Focus optimization on a specific failing judge instead of all judges.

Usage

/eval-optimize
/eval-optimize --max-iterations 5 --model claude-opus-4-6
/eval-optimize --target-judge completeness