Skip to content

Guides

Task-oriented how-tos for the harness. Start with the pipeline overview, then dive into the skill that matches your step. Every guide uses the same eval.yaml — only the CLI flags change between local, containerized, and platform runs.

New here?

Read Skill mode vs prompt mode first — it decides which eval.yaml shape (and which guides) apply to you.

Orientation

  • The eval pipeline

    How setup, analyze, dataset, run, review, optimize, and mlflow fit together.

    Pipeline at a glance

  • Skill vs prompt mode

    Test a predefined skill (execution.skill) or an agent capability directly (execution.prompt).

    Choose a mode

The eval-* skills

The /eval-* slash commands drive the workflow in order. Each maps to a skill under skills/.

  • /eval-analyze

    Understand a skill or docs and generate an eval.yaml (+ eval.md).

    Generate a config

  • /eval-dataset

    Build test cases by skill authoring, synthetic generation, or from traces.

    Build a dataset

  • /eval-run

    Execute the suite, collect artifacts, score with judges, build the HTML report.

    Run an eval

  • /eval-compare

    Compare models or runs side-by-side into one self-contained HTML report.

    Compare models/runs

  • /eval-anova

    Sweep a matrix of models/configs and test whether score differences are statistically real.

    Analyze variance

  • /eval-review

    Present results, collect human feedback, and propose targeted changes.

    Review results

  • /eval-optimize

    Run the automated refinement loop (composes with /eval-run) until judges pass.

    Optimize a skill

  • /eval-mlflow

    Sync datasets, log run results, and push/pull trace feedback.

    Log to MLflow

  • /eval-check

    Scan the whole harness for skill/command overlap and configuration issues.

    Health-check

Running headless & at scale

The same eval.yaml runs unchanged across execution backends — the backend is a --runner CLI flag, never a config key.

  • Running headless

    Tool interception auto-answers AskUserQuestion and gates external services so skills run unattended.

    Headless execution

  • Harbor (containers)

    Run in containers via Podman (local) or Kubernetes/OpenShift.

    Run on Harbor

  • EvalHub

    Run the eval in-process inside an EvalHub Job pod.

    Run on EvalHub

Continuous integration

Looking for field-by-field details?

These guides are how-tos. For the exhaustive config reference, see the eval.yaml schema; for the underlying ideas, see Concepts.