Data contracts before models

AI programmes rarely die at the model. They die eighteen months in, when a finance director asks why the churn number in the board deck is 11% and the number in the product dashboard is 7%, and four teams spend three weeks discovering that one of them treats a cancelled-then-reinstated subscription as churn and the other does not. Nobody was wrong. Nobody had written it down.

That is a data contract problem, and it is the least glamorous work in the entire stack. It has no demo. It does not photograph well. It is also the single strongest predictor of whether the models downstream will still be trusted a year after launch.

What a data contract actually contains#

A contract is not a schema file. A schema tells you a column is a string. A contract tells you who guarantees it, what it means, how often it arrives, what happens when it changes, and who to page when it stops. Six parts, all of them load-bearing:

• Structure — field names, types, nullability, and the encodings that matter. Currency in minor units as an integer, never a float. Timestamps as RFC 3339 with an explicit offset, never a naive local string.
• Semantics — the prose definition of each field, and of the entity itself. What does occurred_at mean when an event is replayed from a backfill? This is where most contracts are thin and most incidents originate.
• Ownership — a named team, resolvable in CODEOWNERS and in the on-call rota. A contract without an owner is documentation.
• Service levels — freshness (p99 lag, not average), completeness, and a volume floor. The volume floor is the one people forget; it is what catches a producer that has died silently rather than one that is emitting garbage.
• Classification — PII, pseudonymous identifier, internal, public. This drives masking policy, retention, and whether the field may cross a region boundary at all.
• Evolution policy — the declared compatibility mode, the deprecation notice period, and the version history. Without this, every change is a negotiation.

// contracts/orders/order_placed.v3.ts
import { z } from "zod";

export const OrderPlacedV3 = z.object({
  event_id: z.string().uuid(),
  occurred_at: z.string().datetime({ offset: true }),
  order_id: z.string().regex(/^ord_[0-9A-HJKMNP-TV-Z]{26}$/),
  buyer_id: z.string().uuid(),
  currency: z.string().length(3),              // ISO 4217
  gross_amount_minor: z.number().int().nonnegative(), // minor units, never a float
  channel: z.enum(["web", "ios", "android", "partner_api"]),
  promo_code: z.string().max(32).nullable().default(null), // added in v3
});

export type OrderPlaced = z.infer<typeof OrderPlacedV3>;

export const contract = {
  subject: "orders.order_placed",
  version: 3,
  owner: "team-checkout",
  compatibility: "FULL_TRANSITIVE",
  semantics: {
    occurred_at:
      "Wall-clock time the buyer confirmed payment intent, as recorded by " +
      "checkout-api. NOT the settlement time and NOT the ingest time. " +
      "Replayed events keep their original value.",
    gross_amount_minor:
      "Before discount, before tax, after currency normalisation. " +
      "A refund is a separate event, never a negative order.",
  },
  classification: { buyer_id: "pseudonymous-id", promo_code: "internal" },
  sla: {
    freshness_p99_seconds: 120,
    completeness: "gross_amount_minor non-null on 100% of rows",
    volume_floor_per_hour: 500, // alerts on silent producer death
  },
  deprecations: [{ field: "coupon_code", removedIn: 3, noticeDays: 90 }],
} as const;

Enforce at the producer, not the consumer#

Almost every warehouse contains the fossil record of consumer-side defence. COALESCE on a field that should never be null. A CASE statement mapping seven spellings of the same channel. A WHERE clause quietly dropping rows where the amount is negative, added during an incident in 2023 by someone who has since left.

Each of those is a private, undocumented patch over a broken promise. They accumulate, they disagree with each other, and they are invisible to the team that caused the breakage. The producer ships a change, the pipeline stays green, and the damage surfaces six weeks later as a metric that drifted.

The fix is structural: make the contract the only sanctioned way to emit the data. Validate inside the producer’s own transaction, and fail the write rather than the read.

import { OrderPlacedV3, contract } from "@/contracts/orders/order_placed.v3";
import { ContractViolation } from "@/platform/errors";

/** The only sanctioned emitter. Nothing else writes to this subject. */
export async function emitOrderPlaced(tx: Transaction, payload: unknown) {
  const parsed = OrderPlacedV3.safeParse(payload);
  if (!parsed.success) {
    // Reject at the boundary. A failed order is a visible, ownable incident;
    // a malformed event is fourteen dashboards quietly averaging nonsense.
    throw new ContractViolation(contract.subject, contract.version, parsed.error.issues);
  }

  // Transactional outbox: the event and the state change commit together,
  // or neither does. This is what removes the dual-write failure mode where
  // the order exists but the event never happened.
  await tx.insert("outbox", {
    subject: contract.subject,
    schema_version: contract.version,
    partition_key: parsed.data.order_id,
    payload: parsed.data,
    created_at: new Date(),
  });
}

Compatibility modes, and the one most teams choose wrongly#

Schema evolution is where contracts stop being philosophy and start being a CI check. A registry — Confluent Schema Registry, Apicurio, Buf for Protobuf — will reject an incompatible schema at merge time if you have told it which direction of compatibility you need. The default is usually BACKWARD, and for anything with a long retention window that default is wrong.

The transitive variants matter more than their adoption suggests. Non-transitive modes check the new schema against the latest registered version only. If a topic is replayed from the beginning — to rebuild a feature store, to backfill a model, to recover from a bad deploy — the reader meets v1 through v9, not just v8. A chain of individually valid changes can be collectively unreadable.

Two rules make the rest cheap. Never reuse a Protobuf field number or an Avro field name after removal; reserve them. And never change the meaning of a field while keeping its name — that is the one breakage no compatibility checker can catch, because the bytes still parse.

# ci/check_compatibility.py - runs on every PR touching contracts/
import json, pathlib, sys, urllib.request

REGISTRY = "http://schema-registry.internal:8081"
CT = "application/vnd.schemaregistry.v1+json"

def check(subject: str, schema: dict) -> tuple[bool, str]:
    # POSTing to .../versions (not /versions/latest) evaluates against every
    # registered version when the subject's level is *_TRANSITIVE.
    url = f"{REGISTRY}/compatibility/subjects/{subject}/versions?verbose=true"
    body = json.dumps({"schemaType": "JSON", "schema": json.dumps(schema)}).encode()
    req = urllib.request.Request(url, data=body, method="POST",
                                 headers={"Content-Type": CT})
    with urllib.request.urlopen(req, timeout=10) as res:
        out = json.load(res)
    return out["is_compatible"], "; ".join(out.get("messages", []))

failed = False
for path in sorted(pathlib.Path("contracts").rglob("*.schema.json")):
    subject = path.stem.removesuffix(".schema")
    ok, why = check(subject, json.loads(path.read_text()))
    print(("PASS " if ok else "FAIL ") + subject + ("" if ok else " -> " + why))
    failed |= not ok

sys.exit(1 if failed else 0)

What breaks without lineage#

Contracts tell you what a dataset promises. Lineage tells you who is standing behind you when the promise breaks. Without column-level lineage — OpenLineage events emitted from the orchestrator, or a warehouse-native graph — three specific things go wrong.

• Blast radius is unknown. A producer proposing a change cannot see the eleven models and two regulatory reports that read the column, so the change is either blocked forever or shipped blind.
• Backfills are silently partial. When a bug is found in a source table, the correct recovery is to recompute every derived artefact in dependency order. Without lineage, teams recompute the three they remember and leave the rest stale for months.
• Feature staleness reaches inference undetected. A training set is built from a table whose upstream stopped refreshing on a Friday. The model trains happily on frozen data. Accuracy degrades in a way no model-monitoring dashboard attributes correctly, because the model is behaving exactly as trained.

Lineage is also the only honest way to prioritise contract work. Fan-out multiplied by business criticality produces a ranked list. In most warehouses that list has around forty entries that genuinely matter and several thousand that do not.

The semantic layer: one definition per business entity#

Structural contracts stop schemas from breaking. They do not stop meaning from diverging. A pipeline can be perfectly typed and still produce four different answers to a simple question, because the definition lives in SQL scattered across dashboards rather than in one place with an owner.

A semantic layer — dbt’s MetricFlow, Cube, or a governed metrics repository of your own — makes the definition a versioned artefact with tests and an owner. Dashboards, notebooks and the feature store all resolve the same entity through the same code path. It is also what makes text-to-SQL and analytic agents viable: an LLM given raw tables invents joins, while an LLM given a semantic layer selects from a bounded, validated set of measures and dimensions.

Retrofitting contracts onto a warehouse that is already running#

The advice to start with contracts is useless to anyone who did not. Nobody gets to stop the business for a quarter. The staged plan below is designed so every step delivers value alone, and so no step requires a producer team to accept a blocking change before they have seen the evidence.

1. Instrument before you legislate. Deploy column-level lineage and audit thirty days of reads. Rank by downstream fan-out times business criticality. Do not write a single contract in this phase.
2. Write the top twenty contracts as observed, not as desired. Infer structure from ninety days of real data, including the nulls you dislike and the enum values nobody documented. A contract describing today’s mess is enforceable this week; one describing the ideal stays aspirational forever.
3. Run enforcement in shadow mode. Validate in the pipeline, emit a violation-rate metric per contract, page nobody. Two to four weeks of this tells you which contracts are wrong and which producers are — and you will be surprised how often it is the contract.
4. Move the check upstream, one producer at a time. The first producer is the expensive one: outbox table, emitter function, registry subject, CI gate. The fifth costs a day because the platform pieces already exist.
5. Assign owners in code. CODEOWNERS on the contract file, the owning team in the on-call rota, and the contract’s SLA wired to the same alerting path as the service itself.
6. Build the semantic layer for the ten metrics that reach the board deck. Ten, not a hundred. Migrate dashboards to it and delete the duplicate SQL as you go, or the old definitions will outlive the new ones.
7. Flip tier-one contracts from shadow to blocking, with a documented, logged override for genuine emergencies. An override that requires a pull request is used rarely; one that requires a Slack message is used daily.

Why this comes before the model#

A model trained on contested data produces contested predictions. The failure is not loud. Nobody files a bug saying the definitions diverged. Instead, adoption erodes: an analyst reconciles the output by hand once, then twice, then stops opening the dashboard. The programme is cancelled a year later for want of business impact, and the post-mortem blames the model.

Contracts, lineage and a semantic layer are what convert a data estate from an archaeological site into an interface. They are unglamorous, they take a quarter, and they are the reason the second and third AI system cost a fraction of the first.