Skip to content

eval-mlflow

Bidirectional MLflow integration for evaluation results, datasets, and feedback. Syncs test cases to the MLflow dataset registry using a two-phase flow where you produce a schema_mapping.json (inputs vs expectations, mapping record fields to source files/field paths) and a script syncs deterministically; logs run params, metrics, artifacts, per-case results tables, and traces to MLflow experiments; pushes judge scores (source_type=CODE) and human feedback (source_type=HUMAN) to execution traces; and pulls annotations added via the MLflow UI back into review.yaml (under mlflow_feedback) for /eval-optimize to consume. Resolves tracking URI from mlflow.tracking_uri in eval.yaml, then the MLFLOW_TRACKING_URI env var, then defaults to http://127.0.0.1:5000. Degrades gracefully -- if MLflow is unavailable, scripts exit cleanly and the skill reports that it was skipped.

Plugin: agent-eval-harness | User-invocable

Contract

Skill Contract canonical-skill-v1

Bidirectionally integrate a file-based evaluation run with an external MLflow tracking server: sync case directories into an MLflow dataset via an agent-produced schema mapping, log a run's params, metrics, artifacts, per-case table, and execution traces, and push/pull judge and human feedback to and from traces so downstream eval-optimize can consume it.

Identity

Functions
execute
Success
  • sync-dataset builds MLflow records whose inputs vs expectations follow dataset.schema (not hardcoded field names) and reports RECORDS synced with STATUS synced.
  • log-results logs params, per-judge and execution metrics, summary artifact, per-case results table, and one trace per case/step (or run) to the resolved experiment for the given --run-id.
  • push-feedback attaches judge feedback (source_type CODE) and human review feedback (source_type HUMAN) to matching traces; pull-feedback writes external MLflow-UI annotations into review.yaml under mlflow_feedback.
  • Tracking URI is resolved from eval.yaml mlflow.tracking_uri, then MLFLOW_TRACKING_URI, then the local default, and a reachability check is reported before acting.

Optimization Targets

task_success deterministic

Invariants

Must Preserve
  • Graceful degradation: when MLflow is unavailable or unreachable the scripts exit 0 and the skill reports skipping, never failing the pipeline.
  • Idempotent operations: re-running is safe (merge_records deduplicates, log_feedback overwrites); do not create duplicate datasets, runs, or feedback.
  • Determine inputs vs expectations by reading dataset.schema, never by assuming or hardcoding field names.
  • Trace feedback is optional: if no traces are found, report 0 and succeed rather than blocking.
  • Preserve path-traversal guards (validate --run-id, reject absolute/.. paths and symlinks) when resolving run and harbor job directories.
  • Do not modify the underlying eval case data or source skill being evaluated.
Fixed Context
toolsRead, Write, Edit, Bash, Glob, Grep, AskUserQuestion
clipython3
knowledgerepository_contentpublic, task_inputtask_private, tool_outputtask_private, external_referenceorganization_private

Traceability

Skill
skills/eval-mlflow/SKILL.md
Supporting

Diagram

eval-mlflow diagram

Arguments

/eval-mlflow [--action <action>] [--run-id <id>] [--config <path>]
Argument Required Default Description
--action all Which sync action to perform.
--run-id - Which eval run to log or attach feedback to. Required for log-results, push-feedback, and pull-feedback.
--config auto-discover Path to eval config.

Usage

/eval-mlflow --run-id 2026-05-01-opus
/eval-mlflow --action sync-dataset
/eval-mlflow --run-id 2026-05-01-opus --action push-feedback