Skip to main content
canonic ingest is the engine that turns normalized connector evidence into reviewable context. It never edits committed files in place: every change reaches semantics/ or knowledge/ only through a diff you (or a policy) approve.

The four-stage pipeline

Nothing in stages 2–4 mutates an accepted file. A change reaches a committed file only through a reviewed (or threshold-approved) diff.

Context builder

Turns evidence into proposals, not files:
  • Deterministic core (no LLM): a RelationSchema maps directly to a semantic-source draft: table, typed columns, a grain candidate from the primary key, joins from discovered foreign keys. This path is reproducible and is the only builder path in headless mode.
  • Modeling definitions: definitions from a definition connector (dbt, Ossie) are folded into the draft of the relation they describe. Their measures replace the inferred ones, their dimensions replace inferred dimensions of the same name or column, and their joins replace FK joins with the same predicate and suppress LLM join guesses for the columns they cover. A dimension expression gets its type inferred from the live column types. Two sources that define the same measure or dimension identically merge, differing ones keep the first and send the draft to review. A definition flagged for review (an AVG measure, a join whose target columns are not a key) is listed in meta.review_flags and caps the draft’s confidence at 0.5, so it never auto-applies. Anything that cannot be placed is listed under skipped in the ingest output.
  • Knowledge pages: doc evidence from an evidence connector (Notion, URL, Ossie ai_context) becomes a proposed page under knowledge/global/<slug>.md at inferred, never auto-applied. Its topic_refs are resolved against the semantic state the run leaves behind: a fully-qualified entity name or an exact measure, dimension or alias match becomes an sl_ref, anything else stays unresolved and is listed on the proposal. The page records meta.source_fingerprint, so an unchanged page is a no-op on the next run and a page whose evidence disappeared is proposed for prune. Pages not drafted by ingest are never touched.
  • LLM-assisted drafting: the fuzzy parts only: naming a measure, drafting a knowledge page’s prose, proposing a grain when no primary key is declared. Every LLM-drafted proposal is labelled drafted_by: llm and carries a lower default confidence than a deterministic one.
Each proposal tracks two independent axes: provenance (where it came from: board_approved / human_curated / inferred) and confidence (how sure the builder is, 0–1). Provenance governs overwrite priority. Confidence governs propose-vs-auto-apply and review ordering. New evidence always enters at inferred, the lowest tier.

Reconciliation

Merges each proposal against the currently accepted file with a deterministic decision table: Provenance tiers: board_approved > human_curated > inferred. Higher always wins: since new evidence is always inferred, it can never silently displace a curated or approved fact. A frozen fact is never edited by reconciliation regardless of the incoming tier or confidence. Conflicting evidence is flagged, not applied. Provenance only ever rises through a deliberate human action: canonic review curate sets meta.provenance: human_curated on the resulting fact. Accepting a proposal never does this on its own: an accepted fact stays inferred until explicitly curated. (canonic apply, the batch path, doesn’t expose curate yet, same as it doesn’t expose freeze.) The answer_outcome row is a deliberate exception to that tier logic: it always reconciles to a contradiction, never to an add or edit, no matter how low-confidence or high-tier the target binding is. This is an attribution safeguard, feedback should never silently rewrite a definition. A human always reviews it. See Feedback loop (E11) below. A conflict is never resolved by silent overwrite. The worst case is a flagged contradiction a human resolves: contradictions ride into the review surface (a PR comment in headless mode) and don’t fail a run by default.

Example: adding a definition connector after an initial apply

Say a connection starts out queryable-only. canonic ingest runs live introspection and proposes semantic sources at provenance inferred, the tier every new proposal starts at. canonic apply accepts them into semantics/*.yaml. Accepting a file this way commits it, but doesn’t curate it: it’s still inferred. Promoting it to human_curated takes an explicit canonic review curate. Now a dbt connector is added, its own connection entry in canonic.yaml (its own id, type: dbt), with params.target_connection naming the physical connection above (see Connectors). That’s what makes this example work: target_connection stamps the dbt connector’s evidence with the physical connection’s id rather than its own, so it lands on the same semantics/<connection>/<name>.yaml target instead of proposing an independent source under its own connection id (which would collide on the project-wide unique source name instead of reconciling). With that link in place, canonic ingest runs again and extract_definitions produces definition evidence from the compiled manifest, an acquisition tier above raw introspection, but it still enters reconciliation at provenance inferred, same as the existing fact. Provenance and acquisition tier are independent: the two proposals disagreeing on the same relation is not a “existing tier higher” case and not a silent win either way. Instead, the modeling-code evidence is preferred as the tie-break within the inferred band, so the run resolves to the “conflicts, tier ≤ proposal” row above: propose an edit. That diff lands under .canonic/pending-diffs/<run-id>/ and goes through canonic review / canonic apply exactly like the first round. If the edit touches grain, joins, or measures, it’s ineligible for auto-apply no matter how confident the proposal is, per the never list under Propose-only by default below. Had the file instead been freezed during the first review, the dbt proposal would only flag a contradiction: the existing fact is left untouched regardless of its acquisition tier or confidence, per the “existing frozen” row above.

Feedback loop (E11)

canonic outcome mark records a correct/incorrect ground-truth verdict against a served answer, with an attribution reason-code (wrong_definition, wrong_data, wrong_interpretation, or unspecified) when incorrect. Only wrong_definition ever implicates the canonical binding that served the answer. The other reason codes are recorded but never feed back into reconciliation or trust. canonic ingest folds this history in alongside connector evidence: it builds a per-binding BindingOutcomeHistory from .canonic/events.jsonl and, once a binding crosses the feedback config’s pattern gate (pattern_min_count distinct-enough wrong_definition outcomes from pattern_min_markers distinct markers within pattern_window_days), mints answer_outcome evidence for it. That evidence always reconciles to a contradiction per the table above, never to an add or edit. E11 only flags, it never rewrites a binding in place. Independently, a recent confirmed wrong_definition outcome caps the binding’s E14 trust tier at caution for trust_cap_window_days, or until a later correct outcome on the same binding supersedes it. canonic audit lists every binding with wrong_definition history, gated or not, in its Feedback loop table. See canonic audit. The four thresholds are configured under feedback: in canonic.yaml. See config schema.

Propose-only by default

Auto-apply is opt-in, bounded by confidence, capped at the lowest provenance, and forbidden for structurally risky fields (grain, joins, measures always require review, regardless of confidence). strict_contradictions: true (or canonic ingest --strict) fails the run if any contradiction is flagged.

Headless mode

Headless mode (explicit --headless, or auto-detected via CI=true) pins the deterministic builder (no LLM in the loop) and, by default, opens an auto-PR carrying the diffs and contradiction notes. Because reconciliation’s decision is always deterministic (only the optional LLM drafting is not), identical evidence and accepted state produce byte-identical proposals across runs: what makes scheduled ingest in CI a safe, repeatable job.

Idempotent re-runs

Re-running canonic ingest refreshes from all configured sources, but a source whose source_fingerprint hasn’t changed is a no-op: only last_validated_at refreshes. A run with no upstream change proposes no diffs at all. Drift is detected purely as a fingerprint change, which becomes a normal edit/prune proposal through reconciliation. Definitions from a definition connector (dbt, Ossie) have their own definition_fingerprint. A changed measure, dimension or join on an unchanged table is therefore a change too, and against a human_curated source it is a contradiction that --strict fails on. The definition fingerprint is only compared when both the accepted file and the new proposal carry one. A run without the definition connector is not a change, and a file accepted before definitions were fingerprinted gets the current one as its baseline on the next unchanged run instead of being flagged.

After a run

Diffs land under .canonic/pending-diffs/<run-id>/. Walk through them interactively with canonic review, accept, reject, skip, freeze, or curate each one, or batch-apply everything still pending with canonic apply.