A pipeline can fail loudly (a task errors) or quietly (it succeeds and publishes wrong or late numbers). Tests catch the first kind before deployment, checks inside the pipeline catch the second kind at run time, and alerts make sure a person hears about either. Data Quality covers data-quality tools (GX Core, Soda Core 2,433 ) in depth; this section stays with what the orchestrator does.
MENU
Pipeline Testing and Alerts
Testing and Alerting for Data Pipelines