Skip to content

Reference

Look-up material for every config key, built-in catalog, CLI command, and API symbol. For the concepts behind these knobs, see Concepts; for task-oriented walkthroughs, see Guides.

The one file to know

Almost everything is driven by a single eval.yaml. Start with the eval.yaml schema — it links out to a per-key page for every block below.

Config

  • eval.yaml schema — Every top-level key, with two minimal configs and a fully annotated example
  • execution — mode (case/batch), skill/prompt, arguments, timeout, max_budget_usd, parallelism, env
  • runner — type, effort, settings, plugin_dirs, env, system_prompt, workspace_mode
  • models — skill, subagent, judge, hook roles and CLI precedence
  • mlflow — experiment, tracking_uri, tags
  • dataset — path, schema, workspace.files
  • generation — strategy (skill/synthetic/from-traces), context, seeds
  • inputs.tools — Tool interception match / prompt handlers
  • outputs — path vs tool artifacts, schema, batch_pattern
  • traces — stdout, stderr, events, metrics
  • permissions — allow / deny tool patterns
  • hooks — Lifecycle shell hooks
  • judges — The five judge types and every field
  • thresholds — min_mean, min_pass_rate, min_win_rate, max_error_rate
  • reward — Collapse judges into an RL reward scalar

Catalogs & tooling

  • Built-in judges

    Reusable judges auto-discovered by category (e.g. consulted_docs), with their arguments.

    Built-in judges

  • Built-in generation prompts

    Shipped seed prompts for synthetic datasets (docs/navigation, docs/anti-pattern, and more).

    Built-in prompts

  • CLI & entry points

    Slash commands (/eval-*), claude-trace, and python -m entry points.

    CLI reference

  • Python API

    The agent_eval package: EvalConfig, EvalRunner, and the MLflow/Harbor modules.

    Python API

Environment & artifacts

  • Runs directory

    The per-run and per-case layout under $AGENT_EVAL_RUNS_DIR (default eval/runs).

    Runs directory

  • Container images

    The agent-eval-harness base image and the agent-eval-hub provider image.

    Container images

  • Environment variables

    Every AGENT_EVAL_*, ANTHROPIC_*, and MLFLOW_* variable the harness reads.

    Environment variables

  • Glossary

    Definitions of core terms used throughout the docs.

    Glossary