The Dataflow Model, 11 Years Later: What Got It Right and What It Got Wrong

Eleven years ago, the Dataflow Model paper made a bold argument: unbounded, out-of-order data was the new normal, and systems should stop waiting for data to become complete. It proposed a unified model—windowing, triggers, watermarks, and retractions—to trade off correctness, latency, and cost across batch and streaming engines. Now, on winning a VLDB Test of Time award, the authors have published a retrospective grading their own work.

The core foundations, they say, aged well. Event time primacy, the futility of waiting for completeness, and strong consistency remain sound. But the analytical interface got important parts wrong. Windowing and triggering, tangled with operational concerns, dominated the exposition beyond their due. Triggers were an over-engineered answer to a question users should never have faced. And the stream-centric worldview missed a deeper truth: streams and tables are two representations of the same object with different access semantics.

The mechanisms that actually delivered on the paper's analytical goals evolved out of the database playbook: SQL, incremental view maintenance, and materialized views with explicit freshness contracts. The authors admit they focused too much on the mechanics of streaming instead of finishing what the database community started—making analytical streaming complexity disappear almost entirely.

The retrospective also explores how the completeness principle split into two forms: watermarks, where streams stay visible, and snapshot-consistent refresh, where they do not. The latter reached far more users by asking far less of them. The authors generalize watermarks into declared constraints on change, find the batch-versus-streaming debate mostly semantic, and observe low-latency demand bifurcating along the old OLTP/OLAP line—leaving analytics at gentler freshness. They adopt the framing they wish they had started with: leave in, leave out, push harder. And they ponder the eventual disappearance of streaming beyond analytics.