Introduction to Airflow in Python

Learn how to to implement and schedule data engineering workflows.

Start Course for Free
4 Hours16 Videos55 Exercises13,822 Learners
4050 XP

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).

Loved by learners at thousands of companies

Course Description

Delivering data on a schedule can be a manual process. You write scripts, add complex cron tasks, and try various ways to meet an ever-changing set of requirements—and it’s even trickier to manage everything when working with teammates. Airflow can remove this headache by adding scheduling, error handling, and reporting to your workflows. In this course, you’ll master the basics of Airflow and learn how to implement complex data engineering pipelines in production. You'll also learn how to use Directed Acyclic Graphs (DAGs), automate data engineering workflows, and implement data engineering tasks in an easy and repeatable fashion—helping you to maintain your sanity.

  1. 1

    Intro to Airflow

    Free

    In this chapter, you’ll gain a complete introduction to the components of Apache Airflow and learn how and why you should use them.

    Play Chapter Now
    Introduction to Airflow
    50 xp
    Running a task in Airflow
    50 xp
    Examining Airflow commands
    50 xp
    Airflow DAGs
    50 xp
    Defining a simple DAG
    100 xp
    Working with DAGs and the Airflow shell
    50 xp
    Troubleshooting DAG creation
    100 xp
    Airflow web interface
    50 xp
    Starting the Airflow webserver
    50 xp
    Navigating the Airflow UI
    50 xp
    Examining DAGs with the Airflow UI
    50 xp
  2. 2

    Implementing Airflow DAGs

    What’s up DAG? Now it’s time to learn the basics of implementing Airflow DAGs. Through hands-on activities, you’ll learn how to set up and deploy operators, tasks, and scheduling.

    Play Chapter Now
    Airflow operators
    50 xp
    Defining a BashOperator task
    100 xp
    Multiple BashOperators
    100 xp
    Airflow tasks
    50 xp
    Define order of BashOperators
    100 xp
    Determining the order of tasks
    100 xp
    Troubleshooting DAG dependencies
    50 xp
    Additional operators
    50 xp
    Using the PythonOperator
    100 xp
    More PythonOperators
    100 xp
    EmailOperator and dependencies
    100 xp
    Airflow scheduling
    50 xp
    Schedule a DAG via Python
    100 xp
    Deciphering Airflow schedules
    100 xp
    Troubleshooting DAG runs
    50 xp
  3. 3

    Maintaining and monitoring Airflow workflows

    In this chapter, you’ll learn how to save yourself time using Airflow components such as sensors and executors while monitoring and troubleshooting Airflow workflows.

    Play Chapter Now
    Airflow sensors
    50 xp
    Sensors vs operators
    100 xp
    Sensory deprivation
    50 xp
    Airflow executors
    50 xp
    Determining the executor
    50 xp
    Executor implications
    100 xp
    Debugging and troubleshooting in Airflow
    50 xp
    DAGs in the bag
    50 xp
    Missing DAG
    100 xp
    SLAs and reporting in Airflow
    50 xp
    Defining an SLA
    100 xp
    Defining a task SLA
    100 xp
    Generate and email a report
    100 xp
    Adding status emails
    100 xp
  4. 4

    Building production pipelines in Airflow

    Put it all together. In this final chapter, you’ll apply everything you've learned to build a production-quality workflow in Airflow.

    Play Chapter Now
    Working with templates
    50 xp
    Creating a templated BashOperator
    100 xp
    Templates with multiple arguments
    100 xp
    More templates
    50 xp
    Using lists with templates
    100 xp
    Understanding parameter options
    50 xp
    Sending templated emails
    100 xp
    Branching
    50 xp
    Define a BranchPythonOperator
    100 xp
    Branch troubleshooting
    50 xp
    Creating a production pipeline
    50 xp
    Creating a production pipeline #1
    100 xp
    Creating a production pipeline #2
    100 xp
    Adding the final changes to your pipeline
    100 xp
    Congratulations!
    50 xp

In the following tracks

Data Engineer

Collaborators

Hadrien LacroixLis Sulmont

Prerequisites

Intermediate PythonIntroduction to Shell
Mike Metzger Headshot

Mike Metzger

Data Engineer Consultant @ Flexible Creations

Mike is a consultant focusing on data engineering and analysis using SQL, Python, and Apache Spark among other technologies. He has a 20+ year history of working with various technologies in the data, networking, and security space.
See More

What do other learners have to say?

“I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.”

Devon Edwards Joseph
Lloyds Banking Group

“DataCamp is the top resource I recommend for learning data science.”

Louis Maiden
Harvard Business School

“DataCamp is by far my favorite website to learn from.”

Ronald Bowers
Decision Science Analytics, USAA