Writing custom judges¶

Judges turn a case's collected outputs into a score. This page shows how to write each of the five judge types by hand, which variables they receive, and how to parameterize and conditionally skip them. For the conceptual overview see Judges; for every field, the judges reference.

The five types¶

A judge's type is inferred from which field you set — you never declare it.

flowchart TD
    J[judge entry] --> B{which field?}
    B -->|builtin| BI[library judge]
    B -->|check| CK[inline Python]
    B -->|prompt / prompt_file / llm_rubric<br/>+ agent| AG[agent judge]
    B -->|prompt / prompt_file / llm_rubric| LLM[LLM judge]
    B -->|module + function| EXT[external code]

Field(s)	Type	Runs	Returns
`builtin`	Library judge	Deterministic Python	`(bool\\|number, str)`
`check`	Inline Python	Deterministic	`(bool\\|number, str)`
`prompt` / `prompt_file` / `llm_rubric`	LLM judge	An API call to `models.judge`	pass/fail, or a score on the judge's `score_range` (`1–5` when undeclared)
the above + `agent`	Agent judge	A tool-using agent run via the runner	pass/fail or a numeric score
`module` + `function`	External code	Your imported callable	`(bool\\|number, str)`

One type per judge

The type fields are mutually exclusive. Setting builtin alongside any of check / prompt / prompt_file / module / function fails at config load.

Every judge — regardless of type — receives the same outputs record for the case: the files it produced, its trace, metrics, annotations, and convenience keys. Numeric (score) judges aggregate to a mean; boolean judges to a pass rate.

Inline `check` judges¶

The most direct type: a snippet of Python whose body becomes a function. Two names are in scope — outputs (the case record) and arguments (this judge's arguments dict, {} if unset). Return a (value, rationale) tuple where value is a bool or a number and rationale is a human-readable string shown in the report.

eval.yaml

judges:
  - name: has_content
    description: Output is non-empty and substantial.
    check: |
      content = outputs.get("main_content", "")
      if len(content.strip()) < 100:
          return False, f"Output too short ({len(content.strip())} chars)"
      return True, f"Output has {len(content.strip())} chars"

Use outputs.get(...) with a default

A case that produced no artifacts won't have every convenience key. Reaching for a missing key with outputs["main_content"] raises KeyError, which the runner records as a judge error (no value) rather than a fail. outputs.get("main_content", "") degrades gracefully.

The record exposes far more than files. Common keys for check judges:

Key	Contents
`outputs["files"]`	`{relative_path: text}` for every collected artifact
`outputs["<dir>_content"]` / `outputs["<dir>_file"]`	First file's text / path per `outputs[].path` dir
`outputs["tool_calls"]`	Tool calls matching an `outputs[].tool` pattern (`{"name", "input"}`)
`outputs["cost_usd"]`, `["num_turns"]`, `["duration_s"]`, `["token_usage"]`, `["exit_code"]`	Trace metrics (require `traces.metrics: true`)
`outputs["events"]`	Parsed event stream (require `traces.events: true`)
`outputs["conversation"]`	Root-level assistant text
`outputs["stdout"]`, `["stderr"]`	Captured logs (require `traces.stdout` / `stderr`)
`outputs["annotations"]`	Per-case metadata from `annotations.yaml`

A metrics-driven check reads straight from the trace:

judges:
  - name: cost_reasonable
    description: Cost per case stays under budget.
    check: |
      cost = outputs.get("cost_usd", 0) or 0
      if cost > arguments.get("max_usd", 0.50):
          return False, f"Cost ${cost:.2f} exceeds limit"
      return True, f"Cost ${cost:.2f}"
    arguments:
      max_usd: 0.50

External `module` / `function` judges¶

When a check outgrows an inline snippet — it needs helpers, imports, or its own tests — move it into a Python module in your project and point module + function at it. The module is imported with your project root on sys.path. The function is called with the record as the outputs keyword argument, and any arguments are spread in as **kwargs, so mirror the builtin-judge signature:

eval/judges/schema_checks.py

import json

def check_schema(outputs, **kwargs):
    required = kwargs.get("required_fields", [])
    raw = outputs.get("main_content", "")
    try:
        doc = json.loads(raw)
    except json.JSONDecodeError as e:
        return False, f"Invalid JSON: {e}"
    missing = [f for f in required if f not in doc]
    if missing:
        return False, f"Missing fields: {', '.join(missing)}"
    return True, "Schema valid"

eval.yaml

judges:
  - name: schema_valid
    description: Output parses and has the required fields.
    module: eval.judges.schema_checks
    function: check_schema
    arguments:
      required_fields: [title, priority, labels]

Return contract is shared

Builtin, check, and module judges all return the same shape: a (value, rationale) tuple (value = bool or number). Returning a bare value with no rationale works too, but the report shows an empty rationale. The library judge cost_budget is a minimal example of the same signature.

A declared score_range is enforced on these judges too

score_range is not just report colouring. A builtin, check, or module judge whose returned number falls outside its declared range — or is non-finite — is recorded as an error sample (value: null plus an error, and a WARNING line on stderr) and left out of the judge's mean. The value is never rounded or clamped: rounding only happens where a model's answer is parsed. So declare score_range: [0, 8] only if the judge genuinely cannot return 9; leave the range off for an unbounded count and the judge keeps emitting whatever it computes.

LLM judges¶

For qualitative criteria, hand the case to a model. Choose one of three fields — they all compile to the same Jinja2 template, then render against the case record:

llm_rubricpromptprompt_file

Shortest form for a single criterion. If your text doesn't already reference {{ conversation }}, the agent's response is appended automatically.

judges:
  - name: cited_sources
    feedback_type: bool     # one yes/no criterion — no scale to declare
    llm_rubric: "The agent cited relevant documentation sources."

Full control over the template and placeholders.

judges:
  - name: output_quality
    description: Compare the output to the reference.
    score_range: [1, 5]     # declare the scale — omitting it warns at config load
    prompt: |
      Compare the generated output against the reference for
      completeness, clarity, and accuracy.

      {{ inputs }}

      # Output
      {{ outputs }}

Reuse a prompt across judges/configs. Path is relative to the project root. context files are appended to the prompt for shared rubrics.

judges:
  - name: detailed_quality
    score_range: [1, 5]
    prompt_file: eval/prompts/quality-judge.md
    context:
      - eval/prompts/scoring-rubric.md

Available Jinja variables¶

The prompt is rendered with these variables:

Variable	Value
`{{ outputs }}`	The record. Bare, it renders every artifact file as text; use `{{ outputs.files }}`, `{{ outputs.cost_usd }}`, etc. for structured access
`{{ conversation }}`	Root-level assistant visible text from the event stream (excludes extended-thinking)
`{{ reasoning }}`	`{{ conversation }}` plus the model's extended-thinking (chain-of-thought), for reasoning-quality judges
`{{ inputs }}`	The case's `input.yaml`, formatted as `key: value`
`{{ tool_trace }}`	Chronological trace of tool calls (Read, Bash, …)
`{{ evidence }}`	Summary of verifiable tool evidence (turns, cost, files read/written) — derived only if referenced
`{{ annotations }}`	Case annotations as formatted text; also `{{ annotations.get('category') }}` for values
`{{ annotations_text }}`	The annotation text alone
`{{ arguments }}`	This judge's `arguments` dict

Score scale vs. pass/fail

With no score_range an LLM judge returns an integer score on the unenforced 1–5 default (aggregated as a mean) and warns at config load — declare the scale. Set feedback_type: bool to make it a pass/fail judge (aggregated as a pass rate). {{ tool_trace }}, {{ evidence }}, and {{ reasoning }} need traces.events: true; {{ conversation }} falls back to stdout.log when no events were captured.

Grading on a scale other than 1–5

Declare it: score_range: [0, 2]. The range is stated in the judge's system prompt, set as minimum/maximum on the submit_score tool schema, and used to normalize the judge in every reward composition that normalizes it (not reward.raw, nor a clamped single reward.judge). Because a tool schema is advisory — the model is not constrained by minimum/maximum — a returned value off the scale is recorded as an error sample and left out of the judge's mean rather than clamped onto the scale, where a 4 from a 0-2 judge would land as a perfect 2. With samples: 3 the case still reduces over the surviving samples, so one bad sample costs a stability flag. Keep stating the scale in the rubric text as well; a prompt that says "score 0, 1, or 2" and a config that says nothing is the combination that produced silently out-of-range scores (#182). An LLM judge's answer is rounded to an integer unless the scale says otherwise — declare feedback_type: float, or fractional bounds such as [0, 2.5], for a continuous scale.

LLM judges — including the agent judges below — are the only types that can be sampled. Set samples: 3 to call the model several times per case and reduce the noise (median for scores, majority vote for booleans); the report flags cases where samples disagreed. samples on any other type is ignored with a warning.

Agent judges¶

Of the types above, only LLM judges (prompt/prompt_file/llm_rubric, and LLM builtins) make a model call — a single, stateless one from the prompt text they're handed; check, external module/function, and Python builtin judges run deterministic code with no model call. An agent judge upgrades an LLM judge into a tool-using run: add an agent: block and the judge executes through the runner abstraction in a staged, isolated workspace, where it may take multiple tool-using turns — Read/Grep/Glob over the case's artifacts and reference docs — before it decides. Reach for it when a verdict must look something up rather than guess.

The judge still takes its instructions from prompt / prompt_file / llm_rubric, and reuses model, feedback_type, score_range, samples, if, and thresholds exactly like an LLM judge — the agent: block is the only thing that changes how it runs. Its inputs are the case's collected output files (filtered by agent.inputs, default all) plus each agent.context dir staged read-only under ./.context/; its one writable spot is ./output/, for the verdict file.

Say you grade whether a strategy's architecture claims are real. A text-only LLM judge has no way to check component names, CRDs, or API fields against the platform docs, so it trusts the output and inflates the score to full marks. An agent judge greps the actual architecture-context docs and marks down invented components:

eval.yaml

judges:
  - name: architecture_score
    description: Component/CRD/API claims check out against the real arch-context docs.
    prompt_file: eval/prompts/architecture-agent-judge.md
    model: claude-opus-4-8
    feedback_type: int
    score_range: [0, 2]
    samples: 3
    agent:
      runner: { type: claude-code }             # per-judge runner (default claude-code)
      allowed_tools: [Read, Grep, Glob]         # read-only default
      context: [.context/architecture-context]  # staged read-only under ./.context/
      inputs: [strat-tasks]                      # which output dirs to stage (default: all)
      # timeout: 420        # default: execution.timeout or 600s
      # max_budget_usd: 2.0 # per-judge-run cap (default 2.0)

The prompt file holds only the criteria — the harness appends the output contract and an untrusted-data guard for you:

eval/prompts/architecture-agent-judge.md

Grade the strategy in ./strat-tasks/. For every component, CRD, and API it names,
Grep ./.context/architecture-context/ to confirm it exists. Score 0 (claims are
invented) to 2 (every claim verified).

The agent ends by writing its verdict to ./output/score.json — numeric here because feedback_type: int:

output/score.json

{"score": 1, "rationale": "TrainingRuntime CRD and the Kueue integration verified against arch-context; the referenced ModelMeshRouter component does not exist."}

Output contract, and the stdout fallback

Write {"score": <number>, "rationale": "…"} for numeric judges or {"passed": <bool>, "rationale": "…"} when feedback_type: bool. If score.json is missing, the harness parses the last such JSON object from the run's stdout; if neither yields a value the case records an error sample — an agent judge never silently passes.

Bash needs a sandboxed runner

The default allowed_tools is read-only ([Read, Grep, Glob]). Only add Bash (e.g. to run the tests a change touched) on a runner with OS-level sandboxing — no network, cleaned credentials, path confinement — because the judge reads untrusted case artifacts. ..-bearing artifact paths are containment-checked, and runner.workspace_mode: repo is rejected for agent judges: they must run in the isolated staged workspace.

`arguments`: parameterizing a judge¶

arguments is a plain dict, reused by every type — but consumed differently:

Python judges (builtin, check, module) — spread in as **kwargs (check judges also see it as the arguments variable).
LLM judges — exposed as {{ arguments }} inside the template.

This lets one judge implementation serve many thresholds without duplicating code — see the max_usd and required_fields examples above.

`if`: skipping a judge per case¶

Give a judge an if: expression to run it only on the cases where it's meaningful. The expression is evaluated against annotations and outputs; when it's false the judge is skipped — the case isn't counted in that judge's mean or pass rate.

eval.yaml

judges:
  - name: dedup_correct
    if: "annotations.get('is_duplicate', False)"   # only duplicate cases
    feedback_type: bool
    prompt: "Did the agent correctly flag the input as a duplicate?"

  - name: output_quality
    if: "not annotations.get('skip_quality', False)"
    feedback_type: int
    score_range: [1, 5]
    prompt: "Score the output 1-5 for completeness, clarity, and accuracy."

if runs in a restricted sandbox

The if expression is evaluated with no builtins — only annotations and outputs are in scope. Keep it to simple attribute/.get() lookups and boolean logic. Put anything heavier inside the judge body (check/module), which runs with full builtins.

Where to go next¶

Judge concepts

Judge types, aggregation, and how scoring fits the pipeline.

Judges
Full field reference

Every judge field, precedence rule, and validation.

judges config
Builtin judges

Library judges you can reference by name before writing your own.

Builtin judges
Sampling & pairwise

Reduce judge noise and run A/B comparisons between runs.

Pairwise & sampling

Writing custom judges¶

The five types¶

Inline check judges¶

External module / function judges¶

LLM judges¶

Available Jinja variables¶

Agent judges¶

arguments: parameterizing a judge¶

if: skipping a judge per case¶

Where to go next¶

Inline `check` judges¶

External `module` / `function` judges¶

`arguments`: parameterizing a judge¶

`if`: skipping a judge per case¶