Skip to content

eval-dataset

Generates realistic test cases based on the eval.yaml dataset schema and judge criteria. Reads eval.md and eval.yaml to derive judge-driven requirements (each case should exercise at least one judge criterion), then generates cases via one of three strategies: bootstrap (from scratch with simple/complex/edge case coverage), expand (fills gaps in existing datasets by analyzing what judges check that no case tests, optionally learning from a previous run's failure patterns), and from-traces (extracts real inputs from MLflow production traces). Handles external-state fields with TODO_ placeholders (so it never fabricates Jira keys, repos, or API endpoints), generates answers.yaml guidance for interactive skills using AskUserQuestion, and creates annotations.yaml for outcome-aware judges (ensuring conditional judges are exercised on both branches). Can invoke /eval-analyze first when no config exists.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Produce evaluation test cases for an eval.yaml that match dataset.schema and exercise judge-driven requirements, sourcing them per generation.strategy (agent-authored from skill analysis, synthetic LLM generation from seeds, or extracted from MLflow production traces), either bootstrapping a fresh starter set or augmenting an existing one to close coverage gaps.

Identity

Functions
generate
Success
  • Detects generation.strategy from the config (defaulting absent to 'skill') and routes to the matching provenance path (skill authoring, synthetic script, or from-traces extraction).
  • Writes case directories under dataset.path whose files and fields conform exactly to dataset.schema, including every {field} referenced by execution.arguments in case mode.
  • Derives fresh-vs-augment from the current dataset state (empty/thin -> fresh starter set; populated -> non-duplicating gap-fillers numbered after the highest existing case).
  • Designs cases to cover distinct execution paths and judge-driven requirements, including simple, complex, and edge cases for a fresh set.
  • Validates generated cases against the schema and reports provenance, coverage, remaining gaps, and any TODO_ external-state placeholders that must be replaced before running.
  • When --harbor is passed, emits self-contained Harbor task packages for the generated cases.

Optimization Targets

task_success judge case-generation.md @ 1559af5

Invariants

Must Preserve
  • Match dataset.schema exactly -- do not change file names, formats, or field names the schema prescribes.
  • Do not fabricate gold reference outputs when the correct output is uncertain -- omit references rather than include misleading ones.
  • Do not invent values for [EXTERNAL: System] fields -- emit TODO_<SYSTEM>_<FIELD> placeholders and surface them in the report.
  • Preserve provenance semantics: --count applies only to skill/from-traces paths and is ignored for synthetic (seed counts govern); do not override generation.strategy with a flag.
  • When augmenting, do not duplicate existing cases; continue case numbering from the highest existing case.
  • Generate realistic, varied content rather than lorem ipsum or obviously templated placeholders.
Fixed Context
toolsRead, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion
clipython3
knowledgerepository_contentpublic, task_inputtask_private, tool_outputtask_private, external_referenceorganization_private

Traceability

Skill
skills/eval-dataset/SKILL.md
Supporting

Diagram

eval-dataset diagram

Arguments

/eval-dataset [--config <path>] [--count <N>] [--strategy <type>] [--run-id <id>]
Argument Required Default Description
--config auto-discover Path to eval config.
--count 5 Number of test cases to generate.
--strategy bootstrap Generation strategy. bootstrap: from scratch. expand: fill gaps in existing dataset. from-traces: extract from MLflow traces (falls back to expand if none found).
--run-id - Previous eval run to learn from when filling coverage gaps (used with the expand strategy to target empirical failure patterns).

Usage

/eval-dataset
/eval-dataset --count 10 --strategy expand
/eval-dataset --strategy from-traces