Master directory with all DAGs developed. Airflow, as a primary method for orchestrating data-pipelines, will give us;
- Monitoring
- Alerting
- Observability
- Upon failure, we are able to cathcup/backfill
Out of the box we get all of the above. Well written DAGs will allow a very small data-science team to own all production data pipelines and prioritize support time of engineering, should we have any issues with any of the pipelines.