What Data Engineering Is

What Data Engineering Is and Is Not

A data engineer designs, builds and runs the systems that collect data, store it, reshape it and hand it to the people and programs that need it. The output is not a chart or a model but a dependable supply of data: tables that are complete, correct, documented and refreshed on schedule. The title is newer than the work, which database administrators and "ETL developers" (extract, transform, load) did for decades; what changed in the 2010s was the volume and variety of data, more than one database and a nightly script could handle.

Where the four data roles sit along the path from source systems to decisions
Where the four data roles sit along the path from source systems to decisions

In small teams one person may do all four jobs, but the boundaries still tell you where a broken number should be fixed.

Subsections