Praewpun Chaiyapornruangdet has completed
Introduction to Apache Airflow in Python
Start Course for Free4 hr
4,000 XP
Loved by learners at thousands of companies
Course Description
Now Updated to Apache Airflow 3.1.6 - Delivering data on a schedule can be a manual process. You write scripts, add complex cron tasks, and try various ways to meet an ever-changing set of requirements—and it's even trickier to manage everything when working with teammates. Apache Airflow can remove this headache by adding scheduling, error handling, and reporting to your workflows. In this course, you'll master the basics of Apache Airflow and learn how to implement complex data engineering pipelines in production. You'll also learn how to use Directed Acyclic Graphs (Dags), automate data engineering workflows, and implement data engineering tasks in an easy and repeatable fashion—helping you to maintain your sanity.
Training 2 or more people?
Get your team access to the full DataCamp platform, including all the features.- 1
Intro to Airflow
FreeIn this chapter, you’ll gain a complete introduction to the components of Apache Airflow and learn how and why you should use them.
Introduction to Apache Airflow50 xpDag counter50 xpRunning your first Dag50 xpAirflow Dags50 xpCreating a simple Dag100 xpWorking with Dags and the Airflow shell50 xpExamining Airflow commands50 xpTesting a task in Airflow50 xpAirflow web interface50 xpNavigating the Airflow UI50 xpExamining Dags with the Airflow UI50 xpDag errors50 xp - 2
Building Dags in Airflow
What’s up Dag? Now it’s time to learn the basics of implementing Airflow Dags. Through hands-on activities, you’ll learn how to set up and deploy operators, tasks, and scheduling.
Airflow operators50 xpDecorating a task100 xpDefining a BashOperator with @task.bash100 xpDefine order of Tasks100 xpMore @tasks100 xpScheduling Dags50 xpSchedule a Dag via Python100 xpDeciphering Airflow schedules100 xpTroubleshooting Dag runs50 xpPassing data between tasks with XCom50 xpPassing data with XCom100 xpXCom variables50 xpAirflow Sensors50 xpDetermining the order of tasks100 xpCreating a FileSensor100 xpSensory Deprivation50 xp - 3
Maintaining and monitoring Airflow workflows
In this chapter, you’ll learn how to save yourself time using Airflow components such as sensors and executors while monitoring and troubleshooting Airflow workflows.
Monitoring, Alerting, and Callbacks50 xpImplementing a callback function100 xpImplementing an SmtpNotifier100 xpWhat's in the logs?50 xpTemplating with Jinja50 xpWriting with Jinja100 xpSubbing out50 xpAirflow Variables50 xpThe answer is Variable50 xpReading a variable in Python100 xpVariables not quite right50 xpDebugging and troubleshooting in Airflow50 xpDag troubleshooting50 xpMissing Dag100 xp - 4
Controlling Dag Logic
Put it all together. In this final chapter, you’ll apply everything you've learned to build a production-quality workflow in Airflow.
Triggers and Fault Tolerance50 xpImplementing a Trigger Rule100 xpAdding retries100 xpTriggering a child Dag100 xpBranching50 xpDefine a @task.branch100 xpBranch troubleshooting50 xpThe HITL Operator50 xpAdding Human approval100 xpApprovalOperator Task50 xpCreating a production pipeline50 xpSetting up the sales ETL pipeline100 xpAdding monthly branching100 xpAdding a human approval gate100 xpCongratulations!50 xp
Training 2 or more people?
Get your team access to the full DataCamp platform, including all the features.collaborators
Mike MetzgerData Engineer Consultant @ Flexible Creations
Mike is a consultant focusing on data engineering and analysis using SQL, Python, and Apache Spark among other technologies. He has a 20+ year history of working with various technologies in the data, networking, and security space.
Join over 19 million learners and start Introduction to Apache Airflow in Python today!
Grow your data skills with DataCamp for Mobile
Make progress on the go with our mobile courses and daily 5-minute coding challenges.