Cookbook

Worked, copy-pasteable recipes for common evaluation scenarios. Each one is a full eval.yaml plus the CLI commands to run it. Pick the card that matches what you're testing.

Same config, any backend

Every recipe is an ordinary eval.yaml. The execution backend (Local, Harbor, EvalHub) is always the --runner flag — never a config key — so any recipe runs unchanged across all three. See backends.

Recipes

• Evaluate a skill (case mode)

  ---

  When you have a skill and want one invocation per test case — the default flow.

• Evaluate a skill (batch mode)

  ---

  When your skill loops over all cases internally in a single batch.yaml invocation.

• Agentic documentation testing

  ---

  When you want to test whether agents can navigate and correctly use your docs.

• Cross-runner: OpenCode

  ---

  When the agent under test isn't Claude Code — drive it through the opaque cli runner.

• Reward shaping for RL

  ---

  When you need to collapse judges into a single [0, 1] scalar for GRPO-style training.

• Comparing runs with ANOVA

  ---

  When you're sweeping models or configs and want to know which score differences are statistically real.

• Write your own analysis recipe

  ---

  When you want /eval-analyze --prompt to bootstrap configs for a new domain.

• Write custom judges

  ---

  When builtin judges aren't enough — author check, module, and LLM judges.

Choosing a recipe

flowchart TD
    A[What are you evaluating?] --> B{Testing a skill<br/>or a capability?}
    B -->|A predefined skill| C{One invocation<br/>per case?}
    B -->|Agent capability| D[Prompt mode]
    C -->|Yes| E[skill-case]
    C -->|No, loops internally| F[skill-batch]
    D --> G{Testing docs?}
    G -->|Yes| H[agentic-docs]
    G -->|No, custom domain| I[custom-analysis-recipe]
    A --> J{Non-Claude agent?}
    J -->|Yes| K[cross-runner-opencode]
    A --> L{Need a reward scalar?}
    L -->|Yes| M[reward-rl]
    A --> N{Scoring gaps?}
    N -->|Yes| O[custom-judges]
    A --> P{Comparing configs<br/>for significance?}
    P -->|Yes| Q[anova]

Start from the walkthrough

New to the harness? Do your first eval end to end first, then come back here for the variant that fits your case. The full field reference lives in the eval.yaml schema.