Skip to main content
HomeData Engineering

Course

Building Data Pipelines with Airflow

AdvancedSkill Level
Updated 06/2026
Author Dags with the TaskFlow API, asset-based scheduling, and deferrable sensors, and run an end-to-end SQL ETL pipeline with quality checks.
Start Course for Free
AirflowData Engineering
4 hr
16 videos
60 Exercises
4,500 XP
Statement of Accomplishment

Create Your Free Account

Continue with GoogleShow more options

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.

Loved by learners at thousands of companies

Group

Training a Team?

Try for Business

Course Description

Want to take your Airflow skills further? This course uses Airflow 3.2, so you'll learn the latest way of doing things. You'll author Dags with the TaskFlow API, schedule them based on data using Assets and the new Asset Partitions, and make them reliable with retries, callbacks, and tests.In the final chapter, you'll build an end-to-end SQL ETL pipeline on DuckDB and add data quality checks directly in Airflow, without any third-party libraries, so the data your pipeline produces stays trustworthy. By the end, you'll know how to take a pipeline from a prototype to something you can actually run in production.

What you'll learn

  • Author Dags with the TaskFlow API (@dag, @task) and pass data between tasks using XCom.
  • Schedule pipelines on data instead of time with Assets and Asset Partitions, and scale them with dynamic task mapping.
  • Harden Dags for production using retries, callbacks, deferrable sensors, and tests.
  • Build an end-to-end SQL ETL pipeline on DuckDB with embedded data quality checks.

Feels like what you want to learn?

Start Course for Free

Prerequisites

Introduction to Apache Airflow in Python
1

Authoring Dags with TaskFlow and XCom

You'll start by meeting the Airflow components, writing your first Dags with the TaskFlow API, and passing data between tasks with XCom.
Start Chapter
2

Dynamic and Data-Aware Pipelines

From there, you'll run tasks in parallel with dynamic task mapping, schedule Dags by data with Assets, and add human approval steps.
Start Chapter
3

Preparing Dags for Production

In this chapter, you'll handle failures with retries and callbacks, save resources with deferrable sensors, and test your Dags at three levels.
Start Chapter
4

Building a Production SQL ETL Pipeline

In this final chapter, you'll build a SQL ETL pipeline on DuckDB, add partition-aware scheduling with Asset Partitions, and embed data quality checks.
Start Chapter
Building Data Pipelines with Airflow
Course
Complete

Earn Statement of Accomplishment

Add this credential to your LinkedIn profile, resume, or CV
Share it on social media and in your performance reviewEnroll Now

FAQs

Is this course suitable for beginners?

No. This is an intermediate course that assumes you have completed Introduction to Apache Airflow in Python. You should already know the basics of Dags, tasks, operators, and XCom before starting.

What kinds of exercises does this course include?

This course is fully interactive. You will write Dags in hands-on coding exercises to work with Airflow programmatically, and use explorable exercises in a live Airflow UI to trigger runs, respond to required actions, and watch scheduling happen in real time.

What will I be able to build after this course?

You will build a complete production SQL ETL pipeline on a DuckDB warehouse, with date-aware scheduling, idempotent re-runs, data quality checks, and full testing, then operate it from the Airflow CLI.

Which Airflow features and tools does this course cover?

You will work with the TaskFlow API, dynamic task mapping with expand and partial, asset-based and partition-aware scheduling, deferrable sensors, SQLExecuteQueryOperator, and SQL quality check operators, all in Python.

Who will benefit most from this course?

Data engineers and pipeline developers who already use Airflow and want production-ready skills like retries, failure callbacks, testing with dag.test, Task Groups, and human-in-the-loop workflows.

    Join over 19 million learners and start Building Data Pipelines with Airflow today!

    Create Your Free Account

    Continue with GoogleShow more options

    or

    By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.

    Grow your data skills with DataCamp for Mobile

    Make progress on the go with our mobile courses and daily 5-minute coding challenges.