Audience
Anyone who wants to to programmatically author, schedule and monitor workflows
About Apache Airflow
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity. Airflow pipelines are defined in Python, allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. Easily define your own operators and extend libraries to fit the level of abstraction that suits your environment. Airflow pipelines are lean and explicit. Parametrization is built into its core using the powerful Jinja templating engine. No more command-line or XML black-magic! Use standard Python features to create your workflows, including date time formats for scheduling and loops to dynamically generate tasks. This allows you to maintain full flexibility when building your workflows.
Other Popular Alternatives & Related Software
ActiveBatch Workload Automation
ActiveBatch by Redwood makes setting up and launching automation easy with no custom scripting required. With a low-code Super REST API adapter, over 100 pre-built job steps and a user-friendly drag-and-drop workflow designer, you can integrate across any system, application and data source, on-prem, in the cloud or in hybrid environments.
Maintain complete control and visibility and meet SLAs with monitoring of all automation from a single pane of glass and get custom alerts via emails or SMS. Managed Smart Queues dynamically scale resources for high-volume workloads, reducing process times while the self-service portal enables business users to run and monitor workflows independently.
ActiveBatch meets security and compliance standards, with ISO 27001 and SOC 2, Type II certifications, encrypted connections and regular third-party tests, always keeping security at the forefront. Along with ongoing product advancements, get the added benefit of 24x7 support and on-site training.
Learn more
Union Cloud
Union.ai is an award-winning, Flyte-based data and ML orchestrator for scalable, reproducible ML pipelines. With Union.ai, you can write your code locally and easily deploy pipelines to remote Kubernetes clusters.
“Flyte’s scalability, data lineage, and caching capabilities enable us to train hundreds of models on petabytes of geospatial data, giving us an edge in our business.”
— Arno, CTO at Blackshark.ai
“With Flyte, we want to give the power back to biologists. We want to stand up something that they can play around with different parameters for their models because not every … parameter is fixed. We want to make sure we are giving them the power to run the analyses.”
— Krishna Yeramsetty, Principal Data Scientist at Infinome
“Flyte plays a vital role as a key component of Gojek's ML Platform by providing exactly that."
— Pradithya Aria Pura, Principal Engineer at Goj
Learn more
AWS Glue
AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue provides all the capabilities needed for data integration so that you can start analyzing your data and putting it to use in minutes instead of months. Data integration is the process of preparing and combining data for analytics, machine learning, and application development. It involves multiple tasks, such as discovering and extracting data from various sources; enriching, cleaning, normalizing, and combining data; and loading and organizing data in databases, data warehouses, and data lakes. These tasks are often handled by different types of users that each use different products. AWS Glue runs in a serverless environment. There is no infrastructure to manage, and AWS Glue provisions, configures, and scales the resources required to run your data integration jobs.
Learn more
Moxo
Moxo’s service orchestration platform transforms complex B2B relationships into seamless experiences.
Business processes often fragment across departments, clients, vendors, and partners, creating inefficiency and risks. Our platform streamlines these workflows—turning disjointed experiences into smooth, efficient operations that reduce costs and enhance client satisfaction.
Moxo accelerates critical processes including client onboarding, document collection, and exception handling. The results: faster completion times, reduced compliance risks, and superior client experiences.
Leading institutions across financial services, consulting, legal, healthcare, and real estate—including Citibank and BNP Paribas —trust Moxo to orchestrate their mission-critical business relationships.
Learn more
Integrations
Company Information
The Apache Software Foundation
Founded: 1999
United States
airflow.apache.org
Videos and Screen Captures
Other Useful Business Software
Cut Cloud Costs with Google Compute Engine
Save on compute costs with Compute Engine. Reduce your batch jobs and workload bill 60-91% with Spot VMs. Compute Engine's committed use offers customers up to 70% savings through sustained use discounts. Plus, you get one free e2-micro VM monthly and $300 credit to start.
Product Details
Platforms Supported
Cloud
