The stack follows the lifecycle from The Data Engineering Lifecycle, with a product category at each stage and a few that span all of them.
Each major cloud offers its own managed service for most layers. You will meet these names in job descriptions and architecture diagrams, so the table maps them, using each vendor's current product name. Several were renamed in 2024-2026: AWS 24 now calls its machine-learning service SageMaker AI, and Google Cloud 1 renamed Dataproc 1 , Cloud Composer 1 and Vertex AI.
| Layer | AWS | Azure | Google Cloud | Taught in |
|---|---|---|---|---|
| Streaming | Kinesis Data Streams; MSK (managed Kafka 129 ) | Event Hubs, with a Kafka endpoint | Pub/Sub; Managed Service for Apache Kafka | 6.17 |
| Lake storage | S3 | Data Lake Storage | Cloud Storage | 8.2, 8.15 |
| ETL and Spark 129 | Glue; EMR | Data Factory (now also in Fabric) | Dataflow; Managed Service for Apache Spark (formerly Dataproc) | 5.14, 7.14 |
| Warehouse | Redshift 24 | Synapse Analytics; Fabric Data Warehouse | BigQuery 1 | 4.13 |
| Orchestration | Managed Workflows for Apache Airflow 129 | Fabric Apache Airflow job; Data Factory | Managed Service for Apache Airflow (formerly Cloud Composer) | 7.16 |
| Catalog | Glue Data Catalog | Purview Unified Catalog | Knowledge Catalog (formerly Dataplex) | 8.13 |
| Serverless code | Lambda | Functions | Cloud Run 1 | Not covered |
| ML and AI | SageMaker AI | Machine Learning | Gemini Enterprise Agent Platform (formerly Vertex AI) | 1.2.5 |
Read across a row and you get the same reference architecture on every cloud: data sources feed a stream (Kafka, Kinesis or Event Hubs), events and files land in object storage (S3, Data Lake Storage or Cloud Storage), batch or stream jobs (Glue, Dataflow, Spark or Databricks 2,717 ) clean and model them, a warehouse (Redshift, Synapse or Fabric, BigQuery) serves analysts, and the curated tables feed machine-learning models and LLM applications. The open-source stack this book runs locally has the same shape, which is why the skills carry over between clouds.
Equivalent does not mean identical. Kinesis is an AWS-native stream with its own API, while MSK runs real Apache Kafka, and Event Hubs speaks the Kafka protocol on its Standard tier and above, so existing Kafka clients can often switch by configuration alone. None of these is free to run: this book describes them from documentation, marked "not run here", and runs the open-source equivalents on your workstation.