Lambda Architecture

The Lambda Architecture's Batch and Speed Layers

Nathan Marz described the Lambda architecture in an October 2011 blog post, "How to beat the CAP theorem", built on Hadoop 129 and Storm. Every event is appended to an immutable master dataset. A batch layer periodically recomputes views over all of it, slowly but exactly. Because those views are hours behind, a speed layer processes only recent events in real time. A serving layer answers each query by merging the batch view with the real-time view, and every batch run replaces the speed layer's approximate results with exact ones.

The design was a sensible answer to the tools of its time. Stream processors of the early 2010s could lose or double-count events, so the batch layer acted as a safety net that corrected them. Immutable raw data meant any bug could be fixed by recomputing from scratch.

The price is two code bases computing the same thing. At BookNest, "revenue per genre" would exist once as a Spark 129 job and once as a streaming job, in different APIs, and the two must agree to the cent. Every business rule change is made twice, tested twice and deployed twice, and when the numbers differ someone must work out which layer is wrong. Teams that ran Lambda at scale often found that keeping the layers consistent cost more than either layer alone.