Running Evaluations

Run an Evaluation

agentv eval evals/my-eval.yaml

Results are written to .agentv/results/eval_<timestamp>.jsonl.

Common Options

Override Target

Run against a different target than specified in the eval file:

agentv eval --target azure_base evals/**/*.yaml

Run Specific Eval Case

Run a single eval case by ID:

agentv eval --eval-id case-123 evals/my-eval.yaml

Dry Run

Test the harness flow with mock responses (does not call real providers):

agentv eval --dry-run evals/my-eval.yaml

Note

Dry-run returns mock responses that don’t match evaluator output schemas. Use it only for testing harness flow, not evaluator logic.

Output to Specific File

agentv eval evals/my-eval.yaml --out results/baseline.jsonl

Validate Before Running

Check eval files for schema errors without executing:

agentv validate evals/my-eval.yaml

All Options

Run agentv eval --help for the full list of options including workers, timeouts, output formats, and trace dumping.