Your Cart is empty. Keep Shopping to find a course!
Browse CoursesMore Learnfly
Business Solution Become an InstructorYour Cart is empty. Keep shopping to find a course!
Browse CoursesApache Airflow is an open-source platform for orchestrating complex workflows. It allows the scheduling and monitoring of data pipelines, offering flexibility and extensibility for managing tasks in a distributed and scalable manner.
Learn more topics in various categories at one place. Explore unlimited courses in other categories and up-skill yourself today.
4.2 770751 Beginner Level
4.1 568668 All Level
4.1 346363 All Level
4.2 100821 All Level
4.6 100564 All Level
4.8 100390 All Level
4.9 99647 All Level
4.8 99615 Beginner Level
4.8 99437 All Level
12 Lectures
13 Lectures
5 Lectures
16 Lectures
273 Lectures
62 Lectures
59 Lectures
19 Lectures
28 Lectures
27 Lectures
87 Lectures
17 Lectures
16 Lectures
140 Lectures
71 Lectures
25 Lectures
14 Lectures
31 Lectures
22 Lectures
103 Lectures
29 Lectures
70 Lectures
23 Lectures
47 Lectures
19 Lectures
21 Lectures
26 Lectures
31 Lectures
15 Lectures
6 Lectures
Apache Airflow is an open-source platform for orchestrating complex workflows and data pipelines. It allows users to schedule, monitor, and manage workflows as directed acyclic graphs (DAGs). Airflow is widely used for automating and orchestrating tasks in data engineering, machine learning, and other domains.
Workflows in Apache Airflow are defined as directed acyclic graphs (DAGs), where nodes represent tasks and edges define the sequence of task execution. Each task is a unit of work, and dependencies between tasks are explicitly defined, allowing for flexible and dynamic workflow execution.
Key components include the Airflow Scheduler (for managing task scheduling and execution), the Airflow Web Server (providing a user interface for DAG management), the Metadata Database (storing metadata and configuration), and Executors (defining how tasks are executed).
Apache Airflow supports dynamic workflows by allowing the definition of dynamic DAGs and tasks at runtime. Workflows can also be parameterized using Jinja templating, enabling the dynamic configuration of tasks based on parameters or external variables.
Unlike traditional schedulers, Apache Airflow provides a more expressive and dynamic way to define, schedule, and monitor workflows. It allows for complex dependencies between tasks, retries on failure, dynamic workflow generation, and a centralized web interface for monitoring and managing workflows.