Skip to content

eval-analyze

Deep-reads a target skill and generates eval.yaml -- the configuration that /eval-run needs. Examines the skill's SKILL.md, follows sub-skill chains recursively (typically 2-5 levels, capped at 5 to avoid circular references) until it finds the skills that produce the final artifacts, explores scripts and test cases, and produces a complete config with execution mode, dataset schema, output descriptions, judges, model defaults, and thresholds. Uses an Explore sub-agent for the recursive skill analysis, validates the result with validate_eval.py, and caches the analysis in eval.md with a content hash of the top-level SKILL.md for staleness detection. The guiding principle is "observe, don't assume" -- every field name and path must come from a file actually read. Auto-invoked by /eval-run when eval.yaml is missing.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Deep-read a target skill (recursively through its sub-skills and scripts) or execute a custom analysis prompt, explore any existing dataset cases, and emit a complete, structurally valid eval.yaml -- dataset schema, outputs, judges, models, thresholds -- plus a cached eval.md analysis, where every field is grounded in files actually observed rather than templates or placeholders.

Identity

Functions
analyze generate
Success
  • Generates an eval.yaml at the resolved config path that passes validate_eval.py config (relative paths resolve, judge references resolve, execution.skill/prompt set, no template-variable errors).
  • dataset.schema and every outputs[*].schema use the real file and field names observed in the skill and its sample case, not generic placeholders.
  • Judges are concrete and runnable -- inline check snippets are valid Python and LLM prompts define per-level scoring; models default to the documented roles.
  • Writes an eval.md caching the analysis with frontmatter (skill, analyzed_at, skill_hash) so freshness checks work.
  • Correctly selects skill mode vs prompt mode and case vs batch execution from the skill's internal logic, asking the user when the mode is ambiguous.

Optimization Targets

task_success verifier_backed validate_eval.py @ 1559af5
evidence_completeness judge eval-yaml-template.md @ 1559af5

Invariants

Must Preserve
  • Observe, don't assume: every field name, file pattern, and directory path in the generated eval.yaml must come from files actually read -- never invent placeholders like <output-dir> or fabricate sub-skill names, schema fields, or judge code.
  • Under --update, preserve the existing file -- only add missing top-level keys; never overwrite user-modified judges, schemas, thresholds, or permissions.
  • Keep dataset.path and outputs[*].path project-relative (never absolute, never ".") since absolute paths break under Harbor/EvalHub and "." would be cleaned between runs.
  • Always validate the generated eval.yaml with validate_eval.py before reporting; fix errors and surface warnings rather than emitting a config full of placeholders.
  • Fail loudly when skill analysis is incomplete or the dataset cannot be found instead of silently generating a degenerate config.
Fixed Context
toolsRead, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion
clipython3
knowledgerepository_contentpublic, task_inputtask_private, tool_outputtask_private

Traceability

Skill
skills/eval-analyze/SKILL.md
Supporting

Diagram

eval-analyze diagram

Arguments

/eval-analyze [--skill <name>] [--prompt <path>] [--config <path>] [--update] [--assess]
Argument Required Default Description
--skill auto-detect Which skill to analyze. If omitted, lists all project skills (excluding eval harness skills) and picks automatically if only one is found.
--prompt none Path to a custom analysis prompt for prompt-based (non-skill) evals -- tests agent capabilities such as documentation effectiveness, pattern understanding, API usage, or constraint compliance instead of examining a SKILL.md. Mutually exclusive with --skill.
--config auto-discover Output path for the eval config file. If omitted, discovers existing layout and scaffolds at the project root or eval//eval.yaml.
--update false Fill in missing sections only, preserving user edits. Useful for upgrading older configs (e.g., adding a models block, migrating check signatures).
--assess false Assess all project skills and recommend which ones need evals, instead of generating a config for one skill (skips config discovery; --skill is ignored in this mode).

Usage

/eval-analyze --skill my-skill
/eval-analyze --prompt examples/openshift-agentic-docs.md
/eval-analyze --update
/eval-analyze --assess
/eval-analyze --skill rfe.create --config eval-rfe.yaml