Skip to main content

Home Data Engineering

Course

Introduction to Apache Airflow in Python

Advanced

4.8+

Updated 03/2025

Learn how to implement and schedule data engineering workflows.

Start Course for Free

Included withPremium or Teams

AirflowData Engineering4 hours16 videos55 Exercises4,050 XP45,738Statement of Accomplishment

Create Your Free Account

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.

Training 2 or more people?

Try DataCamp for Business

Loved by learners at thousands of companies

Course Description

Now Updated to Apache Airflow 2.7 - Delivering data on a schedule can be a manual process. You write scripts, add complex cron tasks, and try various ways to meet an ever-changing set of requirements—and it's even trickier to manage everything when working with teammates. Apache Airflow can remove this headache by adding scheduling, error handling, and reporting to your workflows. In this course, you'll master the basics of Apache Airflow and learn how to implement complex data engineering pipelines in production. You'll also learn how to use Directed Acyclic Graphs (DAGs), automate data engineering workflows, and implement data engineering tasks in an easy and repeatable fashion—helping you to maintain your sanity.

Prerequisites

Intermediate Python Introduction to Shell

1

Intro to Airflow

Introduction to Apache Airflow

Testing a task in Airflow

Examining Airflow commands

Airflow DAGs

Defining a simple DAG

Working with DAGs and the Airflow shell

Troubleshooting DAG creation

Airflow web interface

Starting the Airflow webserver

Navigating the Airflow UI

Examining DAGs with the Airflow UI

2

Implementing Airflow DAGs

Airflow operators

Defining a BashOperator task

Multiple BashOperators

Airflow tasks

Define order of BashOperators

Determining the order of tasks

Troubleshooting DAG dependencies

Additional operators

Using the PythonOperator

More PythonOperators

EmailOperator and dependencies

Airflow scheduling

Schedule a DAG via Python

Deciphering Airflow schedules

Troubleshooting DAG runs

3

Maintaining and monitoring Airflow workflows

Airflow sensors

Sensors vs operators

Sensory deprivation

Airflow executors

Determining the executor

Executor implications

Debugging and troubleshooting in Airflow

DAGs in the bag

Missing DAG

SLAs and reporting in Airflow

Defining an SLA

Defining a task SLA

Generate and email a report

Adding status emails

4

Building production pipelines in Airflow

Working with templates

Creating a templated BashOperator

Templates with multiple arguments

More templates

Using lists with templates

Understanding parameter options

Sending templated emails

Define a BranchPythonOperator

Branch troubleshooting

Creating a production pipeline

Creating a production pipeline #1

Creating a production pipeline #2

Adding the final changes to your pipeline

Congratulations!

Introduction to Apache Airflow in Python

Course
Complete

Earn Statement of Accomplishment

Add this credential to your LinkedIn profile, resume, or CV
Share it on social media and in your performance review

Included withPremium or Teams

Don’t just take our word for it

*4.8

from 242 reviews

84%

15%

1%

0%

0%

Highest to Lowest
Lowest to Highest
Most recent
Top reviews

Jay

about 2 hours

julien

about 7 hours

Emily

about 16 hours

Emma

about 19 hours

Hizkia

about 20 hours

Akanksha

about 21 hours

Jay

julien

Emily

Join over 16 million learners and start Introduction to Apache Airflow in Python today!

Create Your Free Account

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.