Concepts

The harness turns a single eval.yaml into a scored, traced evaluation run. These pages explain the moving parts behind that — how a config becomes an execution, how outputs get scored, and how the same file runs unchanged across backends.

New here?

If you just want to run something, start with your first eval and come back when you hit a concept you want to understand in depth.

The shape of a run

flowchart LR
    C[eval.yaml] --> E[Execution model<br/>case/batch · skill/prompt]
    E --> R[Runner<br/>agent runtime]
    R --> B[Backend<br/>Local · Harbor · EvalHub]
    D[Dataset] --> R
    B --> O[Outputs + traces]
    O --> J[Judges]
    J --> T[Thresholds]
    J --> RW[Reward scalar]
    J --> REP[HTML report]

In this section

• Architecture overview — The moving parts: config, runner, backend, judges, MLflow.
• The execution model — case/batch × skill/prompt: how many invocations, and what to run.
• Runners — The EvalRunner abstraction: claude-code, cli, responses-api.
• Execution backends — One eval.yaml, three execution paths: Local, Harbor, EvalHub.
• Datasets & provenance — Case anatomy and the skill / synthetic / from-traces strategies.
• Judges & scoring — The five judge types and the outputs record they see.
• Pairwise & sampling — A/B run comparison and statistical judge stability.
• Analysis of variance — Full-factorial DoE, repeated-measures / mixed-effects ANOVA, and the cost/quality Pareto frontier.
• Regression thresholds — How min_mean / min_pass_rate / min_win_rate / max_error_rate gate a run.
• The Reward API — Collapsing judges into a single [0, 1] scalar for GRPO.
• Tool interception — The PreToolUse hook and 3-tier AskUserQuestion answering.
• Lifecycle hooks — before/after all/each pipeline hooks (distinct from tool interception).
• MLflow tracing — Building hierarchical traces from stream-json.
• The HTML report — The generated report and its rendering features.

Where things are documented

Concepts explain how and why. For the exhaustive list of keys and their valid values, see the eval.yaml reference.