Security and Data Management

Data platforms concentrate an organization's most sensitive information in one place, which makes them attractive targets. Security for a data engineer starts with three habits. Grant least privilege: each person and each job gets only the access it needs, so the nightly sales job can read orders but not customers' passwords (Least Privilege as a Default). Encrypt data in transit (TLS between every service) and at rest (encrypted disks and buckets). Keep secrets such as database passwords and API keys out of code and out of Git 1,932 , in a secrets manager or environment variables, as Connections and Testing shows for Airflow 129 . Most breaches are mundane: a bucket made public by mistake or a key committed to a repository.

Data management is the discipline of knowing what data you have and keeping it fit for use. It covers: