Skip to content

Judges & scoring

A judge turns a case's collected outputs into a score. Each case is loaded into a single outputs record, then every judge in judges: runs against it. Judges are deterministic (Python) or stochastic (LLM), boolean or numeric — and the harness aggregates them into per-judge pass rates and means for the report and thresholds.

The five judge types

The type is inferred from which field you set — there is no type: key. When more than one could apply, the harness resolves in this priority order (see load_judges):

Priority Type Field(s) set Runs Value
1 builtin builtin Registered judge from agent_eval/judges/ Python judge: whatever it returns · LLM judge (.md): boolean
2 inline check check A Python snippet, in-process (bool \| number, rationale)
3 agent agent (+ prompt / prompt_file / llm_rubric) An agent run through the runner abstraction (reads a staged workspace) numeric or boolean via output/score.json
4 LLM prompt / prompt_file / llm_rubric An Anthropic model call numeric on score_range (told 1–5 when undeclared) or boolean
5 external code module + function An imported Python callable whatever it returns

builtin is mutually exclusive

Setting builtin alongside check, prompt, prompt_file, agent, module, or function raises a load-time error. The other types are distinguished purely by which field is present, so don't set more than one.

agent: upgrades an LLM judge

agent: is not mutually exclusive — it coexists with prompt/prompt_file/llm_rubric. Adding an agent: block upgrades that otherwise-LLM judge from a single model call into a tool-using agent run (it's checked before the LLM branch).

judges:
  - name: budget_check
    builtin: cost_budget          # or "category/name", e.g. docs/consulted_docs
    arguments:
      max_cost_usd: 5.0

Builtins ship with the harness and need no inline code. Reference them by flat name or category/name. See builtin judges.

judges:
  - name: has_content
    check: |
      content = outputs["main_content"]
      if len(content.strip()) < 100:
          return False, f"Output too short ({len(content.strip())} chars)"
      return True, f"Output has {len(content.strip())} chars"

The snippet is compiled into a function receiving outputs and arguments. Return a (value, rationale) tuple — value may be a bool or a number.

judges:
  - name: output_quality
    score_range: [1, 5]   # declare the scale — omitting it warns at config load
    prompt: |
      Compare the generated output against the reference. Consider
      completeness, clarity, accuracy. Score 1-5 where 5 is excellent.

prompt is a full Jinja2 template; prompt_file loads one from disk; llm_rubric is sugar for a one-line criterion (it auto-appends {{ conversation }} if you don't). Priority when several are set: llm_rubric > prompt > prompt_file.

judges:
  - name: schema_valid
    module: eval.judges.schema_checks
    function: check_schema

Imports function from module (resolved against the project root) and calls it with outputs= (plus any arguments as kwargs). Use for validation too complex for an inline check.

judges:
  - name: architecture_score
    prompt_file: eval/prompts/architecture-agent-judge.md
    model: claude-opus-4-8
    feedback_type: int
    score_range: [0, 2]
    samples: 3
    agent:
      runner: {type: claude-code}               # optional; defaults to claude-code
      allowed_tools: [Read, Grep, Glob]         # read-only default
      context: [.context/architecture-context]  # staged read-only under ./.context/
      inputs: [strat-tasks]                      # output dirs to stage (default: all)

An agent: block runs the judge as a tool-using agent through the runner abstraction, against an isolated, staged workspace: the case's output files (filtered by agent.inputs) plus each agent.context dir/file are symlinked in read-only, and a writable ./output/ receives the verdict. The judge writes ./output/score.json — {"score": <integer in [0, 2]>, "rationale": "…"} or {"passed": <bool>, "rationale": "…"}. feedback_type selects which, and the numeric spec states the judge's effective scale ([1, 5] when it declares no score_range); the harness appends the contract + an untrusted-data guard to the prompt automatically. It still takes its instructions from prompt/prompt_file/llm_rubric and reuses model, samples, score_range, feedback_type, if, and thresholds like an LLM judge.

Use it for grading that must look something up rather than guess from prompt text: verify component/CRD/API claims against real docs, cross-reference a spec, or run the touched tests (add Bash to allowed_tools). See the judges config reference for every agent: field.

Bash needs a sandboxed runner

The default allowed_tools is read-only ([Read, Grep, Glob]). Enable Bash only on a runner with OS-level sandboxing (no network, clean credentials, path confinement) — the judge reads untrusted case artifacts. runner.workspace_mode: repo is rejected for agent judges.

What judges receive: the outputs record

Every judge is called with a single outputs dict, assembled per case by load_case_record. It reads all files from your configured output dirs — no schema parsing, so key names come straight from disk. Commonly available keys:

Key Contents
files {relative_path: text} for every artifact file (binary files become a {_binary, path, name} marker)
<dir>_content / <dir>_file First file's text / path for each outputs[].path dir (e.g. artifacts_content)
tool_calls Captured tool calls for each outputs[].tool pattern ({name, input})
annotations Parsed annotations.yaml from the case's dataset dir
inputs Formatted input.yaml fields
conversation Root-level assistant text extracted from events
events Parsed structured event stream (when traces.events)
modified_files In-place file edits collected during the run
exit_code, duration_s, token_usage, cost_usd, num_turns Execution metrics (when traces.metrics)
stdout, stderr Logs (when traces.stdout / traces.stderr)
hook_outputs Values emitted by before_each hooks

LLM judge template variables

Inside a prompt/prompt_file/llm_rubric template you also get convenience variables rendered from the record: {{ outputs }} (formatted file listing), {{ conversation }} (visible text only), {{ reasoning }} (visible text plus the model's extended-thinking / chain-of-thought, for reasoning-quality judges), {{ tool_trace }} (chronological tool calls), {{ inputs }}, {{ annotations }}, {{ evidence }} (verifiable tool-call summary), and {{ arguments }}. Use {{ tool_trace }} to judge behaviour (navigation, tool usage) and {{ conversation }} to judge the response.

Boolean vs numeric values

A judge's value is either a boolean (pass/fail) or a number (a score). This drives both how the LLM is prompted and how results aggregate.

flowchart TD
    J[Judge value] --> B{bool?}
    B -->|yes| PR["pass_rate = passes / cases<br/>(mean mirrors pass_rate)"]
    B -->|no| N{numeric?}
    N -->|yes| M["mean = avg(values)<br/>(pass_rate = None)"]
    N -->|no| X[not aggregated]

For LLM judges the shape is set by feedback_type:

feedback_type Tool the judge is forced to call Value
(omitted), whole bounds submit_score integer on score_range (default 1–5)
(omitted), fractional bounds e.g. [0, 2.5] submit_score number, the fraction preserved
int submit_score integer; a fractional score_range is rejected at load
float submit_score number on score_range, never rounded
bool submit_evaluation passed (boolean)

Builtin .md LLM judges are always boolean. Inline check and external judges decide their own return type — the aggregator infers boolean vs numeric from the values it actually sees across cases.

Numeric range: declared vs. assumed

score_range: [min, max] is the judge's scale. When declared, it is stated in the LLM judge's system prompt and tool schema, colours report cells, normalizes the judge in every reward composition that normalizes it, and is enforced on every judge type — a value off the scale is recorded as an error sample rather than counted. When omitted, LLM and agent judges are told [1, 5] but nothing is enforced (an inline check returning a raw count keeps returning it), and the report has no scale to band against, so those cells render neutral (uncoloured). Declare the range on every numeric judge — omitting it on an LLM or agent judge warns at config load. This range wins over reward.score_range, which is only a deprecated fallback for judges declaring none; judges listed in reward.raw, and a single reward.judge without normalize, are clamped to [0, 1] and consult no range at all — see the reward API.

Upgrading: scores move

A numeric LLM judge used to be asked for a 1–5 score whatever its score_range said, and an agent judge's off-scale verdict was silently clamped into range. The declared scale now reaches the model, and a value off it becomes an error sample excluded from the mean — so per-judge means shift and thresholds.min_mean needs re-baselining. Composites move twice over, with or without a reward: block: every composition now normalizes each numeric judge over its own score_range — instead of a flat [1, 5], or of reward.score_range where a block set one (judges listed in reward.raw, and a single-judge reward without normalize: true, are clamped as before). Don't compare runs across the upgrade.

Aggregation: pass_rate vs mean

For each judge the harness collects the values across all scored cases and computes:

  • Boolean judges → pass_rate = fraction of True values (and mean mirrors it).
  • Numeric judges → mean = average of the values (pass_rate is None).

These feed regression thresholds: min_pass_rate gates boolean judges, min_mean gates numeric ones (and min_win_rate gates the pairwise judge). Cases skipped by a condition, or that errored, contribute no value and are excluded from both aggregates.

Judge arguments

arguments: is a mapping that parameterizes a judge, passed differently per type:

  • Python judges (builtin-python, check, external) — spread as **kwargs into the function.
  • LLM judges — exposed as {{ arguments }} in the template.
judges:
  - name: budget_check
    builtin: cost_budget
    arguments:
      max_cost_usd: 5.0        # → cost_budget(outputs, max_cost_usd=5.0)

Conditional judges (if:)

if: skips a judge for cases where its Python expression is false. The expression is evaluated with annotations and outputs in scope (no builtins). Skipped cases are recorded as skipped and excluded from pass_rate/mean — they don't count as failures.

judges:
  - name: output_quality
    if: "not annotations.get('skip_quality', False)"
    score_range: [1, 5]
    prompt: "Score the output 1-5 for completeness, clarity, and accuracy."

Annotations come from the dataset

annotations is the per-case annotations.yaml in the dataset directory. See datasets for how cases carry metadata.

A condition that raises is an error, not a skip

annotations['tier'] on a case that has no tier key raises, and the expression runs without builtins, so a helper like len(...) raises too. A blown-up condition records an error rather than a skip: the judge's summary status flips from SKIP to ERROR, and a case where nothing else scored composes a reward of 0.0 instead of the 1.0 a genuinely skipped judge would leave. Prefer annotations.get('tier').

Judge model resolution

LLM and pairwise judges resolve their model with this precedence (first non-empty wins):

per-judge  model:   →   models.judge   →   EVAL_JUDGE_MODEL env

If none resolves, an LLM judge fails loudly at load rather than defaulting silently.

models:
  judge: claude-opus-4-6      # default for all LLM/pairwise judges

judges:
  - name: strict_review
    prompt: "..."
    score_range: [1, 5]
    model: claude-sonnet-4-6  # overrides models.judge for this judge only

Sampling stochastic judges

Only LLM judges are stochastic. Set samples: N (or --samples N on the CLI) to run a judge N times per case and reduce — median for numeric, majority vote for boolean — surfacing stability in the report. See pairwise & sampling.

See also