Marquez as a Lineage Backend

Marquez (https://github.com/MarquezProject/marquez 2,283 ) (Apache-2.0), started at WeWork and the reference implementation of OpenLineage, receives run events on POST /api/v1/lineage, stores jobs, runs and dataset versions in PostgreSQL 1,289 , and serves a lineage graph through an API and a web UI. It moves far more slowly than the specification: the last GitHub 29 release is 0.50.0 (24 October 2024), and Docker Hub 514 's newest images are 0.51.1 (March 2025), which ran here.

Its packaged Compose file fails because the API's entrypoint calls a wait-for-it.sh missing from the 0.51.1 image. governance/marquez.yaml instead waits for PostgreSQL with a health check (depends_on: condition: service_healthy) and sets SEARCH_ENABLED=false so the API does not need OpenSearch: l1-marquez-db (postgres:16), l1-marquez (API on port 31500; 91 schema migrations on first start) and l1-marquez-web (UI on 31300). After Capturing Lineage reported BookNest's jobs, the UI showed the Spark 129 job's lineage:

Marquez 0.51.1: lineage of the spark.daily_sales job
Marquez 0.51.1: lineage of the spark.daily_sales job

DataHub 559,249 and OpenMetadata 627,669 (Governance) also accept OpenLineage events, beside ownership and access policies.