Apache Airflow

Apache Airflow is a community-created platform for programmatically authoring, scheduling, and monitoring workflows. It’s much easier to do all these things when workloads are defined as code. They become more versionable, testable, maintainable and collaborative.

With Airflow you can author workflows as directed acyclic graphs (DAGs) of tasks. This means that your tasks are executed on an array of workers while following the specified dependencies. And with rich command line utilities in place, performing complex surgeries on DAGs is simple and straightforward.

Apache Airflow has a rich and useful UI that easily visualizes pipelines running in production, monitors progress, and troubleshoots issues when necessary.