Skip to content

eval-setup

Optional, non-destructive environment configurator for the evaluation harness. Installs dependencies into the isolated venv (as a fallback for mid-session installs or troubleshooting), configures MLflow tracking (local server, local file store, or remote/Databricks), verifies API keys (Anthropic API or Vertex AI), sets up the runs directory, checks skill-specific environment variables referenced in eval.yaml's execution.env, and creates the MLflow experiment. Runs check_env.py preflight checks (with --fix) and re-verifies at the end. Most users can skip this -- dependencies auto-install via the plugin's SessionStart hook and agent_eval is available via symlinks.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Verify and optionally configure the agent-eval-harness environment: check Python/dependencies/API keys/MLflow, set up MLflow tracking and the runs directory, and suggest available evaluation modes based on repository contents.

Identity

Functions
execute
Success
  • check_env.py preflight passes: Python >= 3.11, mlflow/pyyaml/agent_eval importable, and API key or Vertex project set.
  • MLflow tracking is configured (or explicitly skipped) and the runs directory exists.
  • If eval.yaml exists, it validates and any configured MLflow experiment is created.
  • Reports final environment status and suggests correct next steps along the analyze -> dataset -> run -> review/optimize -> mlflow pipeline.

Optimization Targets

task_success verifier_backed check_env.py @ 1559af5

Invariants

Must Preserve
  • Non-destructive: skip steps already done and do not overwrite existing configuration or env vars.
  • Treat MLflow as optional -- never fail setup when MLflow cannot be configured; fall back to local file store.
  • Do not leak credential values: mask literal env values and report API keys only as set/not-set.
  • Honor --skip-mlflow, --tracking-uri, --runs-dir, and --harbor argument semantics.
  • Report each check clearly with pass/fail status and a concrete fix for every failure.
Fixed Context
toolsRead, Bash, Glob, AskUserQuestion
clipython3
knowledgerepository_contentpublic, tool_outputtask_private, task_inputtask_private

Traceability

Skill
skills/eval-setup/SKILL.md
Supporting

Diagram

eval-setup diagram

Arguments

/eval-setup [--tracking-uri <uri>] [--skip-mlflow] [--runs-dir <path>]
Argument Required Default Description
--tracking-uri auto-detect MLflow tracking URI (skips interactive setup). Accepts local or remote URIs.
--skip-mlflow false Skip MLflow setup entirely. The harness works without MLflow.
--runs-dir eval/runs Directory where eval runs are stored. Configured via AGENT_EVAL_RUNS_DIR env var.

Usage

/eval-setup
/eval-setup --tracking-uri http://127.0.0.1:5000
/eval-setup --skip-mlflow