Streaming Data & ML Serving
Real-Time Decision Platform
A platform that turns database changes into fresh features within seconds, serves a model decision inside a live request, and writes an immutable record of every decision — the inputs, the model version and the policy that applied — so it can be explained, replayed and rolled back months later.
At a glance
- Type
- Reference architecture
- Domain
- Streaming Data & ML Serving
- Architecture
- 6 layers
- First production cut
- 9–14 weeks, phases in sequence
A reference architecture. Figures on this page are design targets, not measured client results.
The challenge
A decision that arrives tomorrow morning is worth nothing. The fraudulent transaction cleared, the customer churned, the truck left the yard, the limit was approved. Real-time decisioning means a model reads state that is seconds old and answers inside a request somebody is waiting on — for every event, not a sampled batch. The hard part is rarely the model. It is that the features at serving time must be provably the same features the model was trained on, and that every decision must still be defensible when a regulator, a customer or an incident review asks about it eleven months later.
Failure mode
The obvious approach, and why it breaks.
Almost every team reaches the same first design, and it is not a bad instinct — it is the shortest route to something that demos. Here is that design, and the point at which real volume takes it apart.
Discarded
The obvious approach
A nightly batch job reads the warehouse, scores every customer with a Python script and writes the results into a scores table. The application looks the score up by primary key at request time. Training features are computed in a notebook against warehouse tables; serving features are whatever that notebook’s SQL happened to say the last time somebody ran it.
Why it breaks
What production does to it
Four failures, and they compound. Staleness: a score computed at 02:00 knows nothing about the eighteen hours since, and in fraud, churn and pricing the most recent events carry most of the signal. Coverage: entities created after the batch ran have no row, so the fallback branch quietly becomes the real product for exactly the traffic that matters most. Skew: two implementations of the same feature drift apart the first time someone fixes a bug in one of them, and the model’s offline AUC stops predicting anything about production behaviour. Leakage: joining training features to labels on entity id without an as-of timestamp pulls in values computed after the event being predicted — offline metrics look superb, production collapses, and nobody can explain the gap. Then a decision is disputed. All that exists is a number in a table, with no record of which model produced it, which feature values it saw, or which policy version was in force.
Architecture
How the system is put together.
6 layers, from the edge where work arrives to the operations that keep it honest. Each one names the components it owns and why it exists as a separate concern.
Layer 01
Change capture & event ingest
Reading the database log rather than the database means capture is complete and cheap — no missed writes, no polling load on the primary. Partitioning by entity key is the decision that makes everything downstream possible: all events for one customer land in order, on one partition, so stateful aggregation never has to reconcile across shards.
Components
- Debezium CDC on Postgres logical decoding and MySQL binlog
- Transactional outbox for events the application emits deliberately
- Kafka / Amazon MSK topics, partitioned by entity key, compacted where state-like
- Schema registry with Protobuf contracts and enforced compatibility checks
- Idempotent producers, exactly-once sinks and per-key ordering guarantees
- Dead-letter topics with a replay tool that preserves original offsets
Layer 02
Stream processing & feature computation
This layer holds the aggregates that make a decision worth making — count of declines in the last ten minutes, rolling basket value, time since last login. It is stateful, which means the state is the asset: a job that loses it has to rebuild from the retained log, and that rebuild time is the real recovery objective.
Components
- Apache Flink jobs with event-time windows and explicit watermark strategies
- RocksDB state backend with incremental checkpoints to S3
- Feature definitions declared once, compiled to a streaming plan and a batch plan
- Bounded lateness with a side output for events that arrive past the window
- Savepoint-based deploys so state survives a job upgrade
- Enrichment joins against slowly changing dimensions held as broadcast state
Layer 03
Feature store — online and offline
One definition, two materialisations. The online store answers “what is true now”; the offline store answers “what was true at 14:32:07 on the day of this label”. If those two answers come from different code, the model is being trained on a distribution it will never meet in production — and the parity job exists so that claim is measured, not asserted.
Components
- Online store in Redis or DynamoDB, keyed by entity, sized for single-digit-millisecond reads
- Offline store as Apache Iceberg tables on S3, partitioned by event day
- Point-in-time correct as-of joins for training set assembly
- Feature registry: owner, definition hash, freshness SLA, TTL, on-call contact
- Backfill jobs that run the same definition over history and write both stores
- Parity job that replays logged serving values through the offline definition
Layer 04
Decision service
The service owes an answer inside its budget, always. A missing feature, a slow store or an unhealthy model must degrade to a defined fallback rather than an exception, because an unhandled error in this path is an outage in the product that calls it. Hard constraints live in the policy layer, never inside model weights — a model cannot be instructed to obey a regulation.
Components
- gRPC serving endpoint behind Envoy with connection pooling and per-call deadlines
- Gradient-boosted models via ONNX Runtime in-process; neural models on Triton
- A policy layer for hard constraints — regulatory limits, blocklists, kill switches
- Strict request budget with a deterministic, documented fallback decision on timeout
- Shadow scoring of challenger models on the live feature vector
- Model registry aliases resolved at request time, not baked into the container image
Layer 05
Decision log & explainability
This is what separates an automated decision from an unaccountable one. Storing the vector that was actually served — not the one recomputed later — is the only way to answer “why was this application declined in March” without guessing. It also makes regression testing honest: a candidate model is scored against real historical inputs, not a synthetic sample.
Components
- Append-only decision record: decision id, timestamp, entity, full feature vector
- Model version, definition hashes, policy version and the code commit in force
- Reason codes and per-feature attributions computed at serve time or reconstructible
- Immutable Iceberg storage partitioned by day, with retention and access controls
- Replay harness that re-executes any historical decision from the log alone
- Lineage from a decision back to the source events that produced its features
Layer 06
Monitoring, drift & evaluation
Models fail quietly. Accuracy cannot be measured until labels arrive — often weeks later for credit or churn — so the early-warning signals have to be upstream: a feature that went null, an input distribution that shifted, an approval rate that moved four points overnight. Those are the alerts that page someone.
Components
- Feature freshness, null-rate and cardinality monitors with per-feature thresholds
- Population stability index and KS tests on input distributions, per segment
- Delayed-label joins that score real outcomes once labels mature
- Champion versus challenger comparison on identical traffic and identical features
- Consumer lag, checkpoint duration and state size as first-class SLIs
- Alerting on decision-rate shifts, which move before accuracy metrics do
Engineering decisions
Six decisions, and what each one costs.
A decision without a stated cost is marketing. These are the choices that shape the system, the reasoning behind them, and what is given up in exchange.
- 01
One feature definition compiles to the streaming job, the backfill and the point-in-time training join.
Why
Train/serve skew is not a bug to be found, it is a structural consequence of writing the same logic twice. Making the definition the deployable artefact — versioned, hashed, owned — removes the class of failure entirely rather than monitoring for it after the fact.
What it costs
Feature expressiveness is capped at what the shared execution layer supports. A data scientist who wants an arbitrary Python transform with a library dependency cannot simply ship it, and the iteration loop is slower than a notebook. Some genuinely useful features will be rejected because they cannot be computed identically in both paths.
- 02
Capture change from the database log with CDC rather than dual-writes or periodic polling.
Why
Dual-writes lose events whenever the second write fails after the first succeeded, and polling trades primary-database load against staleness with no good setting. The write-ahead log is the authoritative record of what happened, in commit order, at no cost to the transaction path.
What it costs
The platform becomes coupled to database internals and to the DBA team. Postgres replication slots accumulate WAL and can fill the disk if a consumer stalls, an unchanged TOASTed column arrives as a placeholder rather than a value unless replica identity is set to full, and every schema migration is now a coordination event with the streaming layer. Ordering is guaranteed per key, not globally.
- 03
Process on event time with watermarks and bounded lateness, not on arrival time.
Why
Mobile clients buffer offline, gateways retry and partitions lag. Counting events by when they were processed produces aggregates that change depending on infrastructure weather, which makes a model trained on history unreproducible and an incident review impossible.
What it costs
A watermark delay is a latency floor: results wait for the lateness allowance before they are final. Events arriving past the allowance land in a side output and the online value is briefly wrong until corrected. Operationally this is the hardest part of the system to debug — a stuck watermark on one idle partition stalls an entire job, and the symptom looks like a system that is simply not producing output.
- 04
Serve precomputed features from a key-value online store instead of computing them per request.
Why
A sub-50ms budget does not survive a warehouse query, a join or a window aggregation. Moving computation off the request path and into the stream turns serving into a handful of key lookups, and makes latency a function of store performance rather than query complexity.
What it costs
Write amplification — features are materialised for every entity, including the long tail that is never scored — and storage cost that scales with entity count rather than traffic. Freshness is bounded by stream lag, so a burst of ingest lag degrades decision quality silently. Every new feature needs a backfill before it can serve, which makes shipping one a scheduled operation rather than a deploy.
- 05
Challengers run in shadow on live traffic first; promotion is a registry alias flip, not a deployment.
Why
Shadow scoring on the identical feature vector reveals latency, error and distribution differences under real load without exposing a single customer. Decoupling the model version from the container image means promotion and rollback are the same operation in opposite directions, executed in seconds by whoever is on call.
What it costs
Every request pays for two inferences, so serving compute roughly doubles during evaluation windows, and challenger failures must be isolated so they cannot affect the served decision. Shadow results are also observational — they show what the challenger would have scored, not what would have happened, because no downstream action was taken. Genuine causal comparison still requires a real traffic split.
- 06
Write the decision log on the request path, synchronously, before the response is returned.
Why
A decision that is not recorded did not happen, as far as any later audit is concerned. Capturing the exact feature vector served, with model and policy versions, is what makes a decision explainable, a dispute answerable and a candidate model testable against real history.
What it costs
Milliseconds and bytes on the hot path, and storage that grows linearly with traffic forever. The feature vector frequently contains personal data, which pulls retention schedules, access control and deletion obligations into what started as an engineering concern. A durable local buffer can recover most of the latency, at the cost of a small window where a crash loses records.
Design targets
The operating point we build toward.
Targets, with the basis for each one written underneath it. They describe what this architecture is designed to hold — not a result measured on somebody else’s system.
- Decision latency at the serving boundary, feature lookup through policy check
- p99 < 50 msDecision latency at the serving boundary, feature lookup through policy checkDesign target for precomputed online features plus a gradient-boosted model on CPU; deep models or on-demand features require a different budget
- Feature freshness — source commit to value readable in the online store
- p95 < 5 sFeature freshness — source commit to value readable in the online storeDesign target for log-based CDC and a single stateful stream stage; multi-stage joins and rekeying add measurably to this
- Served feature values that disagree with the offline recomputation
- < 0.1%Served feature values that disagree with the offline recomputationDesign target, measured continuously by replaying logged serving vectors through the offline definition — not an assumption
- Time to revert to the previous champion model under a rehearsed rollback
- < 5 minTime to revert to the previous champion model under a rehearsed rollbackDesign target for a registry alias flip with pre-warmed replicas; validated in game days, not inferred from architecture
Stack
What this is built with
Named, current technology. Substitutions are normal — the shape of the system matters more than the vendor behind any one box.
- Debezium (Postgres logical decoding, MySQL binlog)
- Apache Kafka / Amazon MSK
- Confluent Schema Registry with Protobuf contracts
- Apache Flink with RocksDB state and S3 checkpoints
- Apache Iceberg on S3 as the offline store
- Trino and Spark for backfills and point-in-time joins
- Feast for the feature registry and materialisation
- Redis and Amazon DynamoDB as online stores
- XGBoost and LightGBM for tabular decisions
- NVIDIA Triton and ONNX Runtime for model serving
- MLflow model registry with staged aliases
- gRPC behind Envoy at the serving edge
- Evidently for drift and data-quality checks
- OpenTelemetry, Prometheus and Grafana
Timeline
To a first production cut.
Four phases, run in sequence. The first one is not engineering — it is deciding precisely what the system owes, because that is what every later phase is measured against.
1–2 weeks
Decision definition and offline baseline
The decision written down precisely: its operating point, the relative cost of a false positive and a false negative, the label and how long it takes to mature, and the fallback when the system cannot answer. Plus a point-in-time correct training set and a batch baseline that establishes what the model is actually worth before any streaming work begins.
3–5 weeks
Streaming spine and feature store
CDC connectors, topic layout and schema contracts, the first Flink feature jobs with event-time semantics, both stores materialised from a single definition, and a parity job proving the backfilled values match the served values within tolerance.
3–4 weeks
Serving path and decision log
The decision service with its latency budget, fallback behaviour and policy layer, the immutable decision log with reason codes, a replay harness that reconstructs any past decision, and the first model running in shadow on production traffic.
2–3 weeks
Promotion and operational hardening
Freshness, drift and decision-rate monitors wired to alerts, load testing at projected peak with lag recovery measured, a rehearsed rollback, runbooks for a stalled watermark and a lagging consumer, and a staged traffic ramp with champion–challenger comparison at each step.
Keep reading
Related blueprints.
Systems that share a spine with this one — the same evaluation, budget and rollback discipline, applied to a different problem.
Next step
Walk this architecture against your constraints.
A 45-minute session on the Real-Time Decision Platform: which layers you already have, which ones you do not, and the decisions on this page that would go the other way for you.