canonic eval answers “which model is actually good enough for that job.” It
is not a correctness test for canonic itself (that’s canonic assert,
which gates CI). Point it at one or more candidate local/hosted models, and it drives the real
production drafter over a labeled dataset for each one, then reports accuracy against the known
answers, whether the model actually returns parseable structured output, and latency, so you can
pick or swap the model your draft/reconcile pipeline runs on with numbers instead of a guess.
It’s an operator command: it makes live model calls (so results depend on whatever’s running at
base_url when you invoke it), and is therefore not run in CI. Only the deterministic harness
internals are unit-tested. Re-run it before tagging a release, or whenever you change the
candidate model, so the recommendation tracks reality instead of being asserted once. See
Instrumentation & evaluation
for how this fits alongside the event log and the accuracy harness.
eval baseline
Run candidate models through a labeled dataset over the real draft (or reconcile) path and publish a per-release baseline doc.
--candidates points at a YAML file listing the models to compare, each shaped like a
canonic.yaml llm block (openai_compatible covers local runtimes such as Ollama, vLLM, LM Studio,
llama.cpp, and TGI, as well as hosted endpoints. They differ only by base_url and whether an API
key is needed):
For each candidate, the report includes accuracy, structured-output adherence, and p50 latency, plus which candidate (if any) is recommended (the most accurate one clearing the adherence floor, or none if none clears it):