Every report, dashboard and machine-learning model depends on data that someone moved, cleaned, checked and stored before anyone looked at it. Data engineering is that work: building and running the systems that take data from where it is born, such as an order form, a payment service or a supplier's catalog feed, and deliver it, trustworthy and on time, to the people and programs that use it.
This chapter is the map for the rest of the book. It names the ideas that come up again and again and gives each a page or two and a pointer to the chapter that teaches it properly.
You also meet BookNest, the small online bookshop that runs through all eight chapters. Its six-book catalog becomes XML in XML and Its Toolchain, columnar files in JSON, Columnar and Binary Formats, a sales mart in Analytical SQL and Data Warehouses, a Spark 129 job in Batch Processing with Apache Spark, a stream of order events in Apache Kafka and Managed Cloud Kafka, an orchestrated pipeline in Orchestration and Pipelines and a lakehouse in Lakehouses, Data Quality and Governance. Here you set up the book's workstation (WSL2 6 with Ubuntu 26.04 225 , Docker 514 , Python and DuckDB 61,228 ) and push the catalog through a first, tiny pipeline from CSV to Parquet 129 to SQL to a chart.
What you will learn
What data engineers do, and how the role differs from data science, analytics and platform engineering.
The data engineering lifecycle and the undercurrents that run beneath it.
When to use OLTP or OLAP, batch or streaming, and the Lambda or Kappa architecture.
The main data modeling paradigms, and schema-on-write versus schema-on-read.
The layers of the modern data stack and their AWS 24 , Azure 6 and Google Cloud 1 equivalents.
How warehouses, lakes, lakehouses, data mesh and data fabric differ, and what tools really cost.
How to set up the workstation and run a first CSV-to-Parquet-to-SQL-to-chart pipeline.
Which roles and skills the field rewards, and how to handle personal data safely.
Sections
- What Data Engineering Is
- The Data Engineering Lifecycle
- The Lifecycle's Undercurrents
- OLTP and OLAP at a Glance
- Batch and Streaming Processing
- Lambda and Kappa Architectures
- Data Modeling Paradigms
- The Modern Data Stack
- Warehouses to Fabric
- Choosing Tools and Cost
- Workstation Setup
- A Tiny End-to-End Pipeline
- Careers and Skills
- Security and Privacy Basics
- Test Yourself!