Skip to main content
canonic eval is an operator command; it makes live model calls, so it is not run in CI (only the deterministic harness internals are unit-tested).

​
eval baseline

Run candidate models through a labeled dataset over the real draft (or reconcile) path and publish a per-release baseline doc. 
canonic eval baseline --candidates candidates.yaml
canonic eval baseline --candidates candidates.yaml --task reconcile --out reports/reconcile-baseline.md
FlagDescription
--candidates, -cYAML list of openai_compatible candidate models.
--dataset, -dLabeled JSONL set (defaults to the shipped set).
--out, -oWhere to write the markdown baseline doc (default reports/baseline-models.md).
--taskTask to evaluate; only draft has a live call site in v1; reconcile is also supported.
--adherence-floorMinimum structured-output adherence required to recommend a candidate.
For each candidate, the report includes accuracy, structured-output adherence, and p50 latency, plus which candidate (if any) is recommended. Re-run this before tagging a release so the baseline tracks reality as models churn.