After doing these courses, I feel confident creating professional visualizations and dashboards
Shoebur RahmanProcurement Specialist, Giz Bangladesh
Course Description
Want to take your Airflow skills further? This course uses Airflow 3.2, so you'll learn the latest way of doing things. You'll author Dags with the TaskFlow API, schedule them based on data using Assets and the new Asset Partitions, and make them reliable with retries, callbacks, and tests.In the final chapter, you'll build an end-to-end SQL ETL pipeline on DuckDB and add data quality checks directly in Airflow, without any third-party libraries, so the data your pipeline produces stays trustworthy. By the end, you'll know how to take a pipeline from a prototype to something you can actually run in production.
Feels like what you want to learn?
Start Course for FreeWhat you'll learn
- Author Dags with the TaskFlow API (@dag, @task) and pass data between tasks using XCom.
- Schedule pipelines on data instead of time with Assets and Asset Partitions, and scale them with dynamic task mapping.
- Harden Dags for production using retries, callbacks, deferrable sensors, and tests.
- Build an end-to-end SQL ETL pipeline on DuckDB with embedded data quality checks.
Prerequisites
Curriculum
Course outline
1
Authoring Dags with TaskFlow and XCom
You'll start by meeting the Airflow components, writing your first Dags with the TaskFlow API, and passing data between tasks with XCom.
- Airflow core concepts50 XP
- Writing a minimal TaskFlow Dag100 XP
- Matching Airflow components to their roles100 XP
- The TaskFlow API50 XP
- Converting a classic Dag to TaskFlow100 XP
- Adding a task and wiring dependencies100 XP
- Exploring a pipeline in the Airflow UI50 XP
- Sharing data between tasks50 XP
- Returning data from a TaskFlow function100 XP
- Connecting a classic operator to a TaskFlow task100 XP
- Understanding XCom limitations50 XP
2
Dynamic and Data-Aware Pipelines
From there, you'll run tasks in parallel with dynamic task mapping, schedule Dags by data with Assets, and add human approval steps.
3
Preparing Dags for Production
In this chapter, you'll handle failures with retries and callbacks, save resources with deferrable sensors, and test your Dags at three levels.
4
Building a Production SQL ETL Pipeline
In this final chapter, you'll build a SQL ETL pipeline on DuckDB, add partition-aware scheduling with Asset Partitions, and embed data quality checks.
Building Data Pipelines with Airflow
Course
Complete
