मुख्य सामग्री पर जाएं

होम Data Engineering

कोर्स

Building Data Pipelines with Airflow

उन्नतकौशल स्तर

अपडेट किया गया 06/2026

Author Dags with the TaskFlow API, asset-based scheduling, and deferrable sensors, and run an end-to-end SQL ETL pipeline with quality checks.

मुफ़्त में पाठ्यक्रम शुरू करें

AirflowData Engineering

4 घंटे

16 वीडियो

60 अभ्यास

4,500 XP

उपलब्धि का प्रमाण पत्र

अपना मुफ़्त खाता बनाएं

Google के साथ जारी रखें अधिक विकल्प दिखाएँ

या

जारी रखने पर, आप हमारी उपयोग की शर्तें, हमारी गोपनीयता नीति को स्वीकार करते हैं और यह भी कि आपका डेटा संयुक्त राज्य अमेरिका में संग्रहीत किया जाता है।

हजारों कंपनियों के शिक्षार्थियों द्वारा पसंद किया गया

टीम को ट्रेनिंग देना चाहते हैं?

व्यवसाय के लिए आज़माएँ

पाठ्यक्रम विवरण

Want to take your Airflow skills further? This course uses Airflow 3.2, so you'll learn the latest way of doing things. You'll author Dags with the TaskFlow API, schedule them based on data using Assets and the new Asset Partitions, and make them reliable with retries, callbacks, and tests.In the final chapter, you'll build an end-to-end SQL ETL pipeline on DuckDB and add data quality checks directly in Airflow, without any third-party libraries, so the data your pipeline produces stays trustworthy. By the end, you'll know how to take a pipeline from a prototype to something you can actually run in production.

पूर्व आवश्यकताएं

Introduction to Apache Airflow in Python

1

Authoring Dags with TaskFlow and XCom

You'll start by meeting the Airflow components, writing your first Dags with the TaskFlow API, and passing data between tasks with XCom.

Airflow core concepts

Writing a minimal TaskFlow Dag

Matching Airflow components to their roles

The TaskFlow API

Converting a classic Dag to TaskFlow

Adding a task and wiring dependencies

Exploring a pipeline in the Airflow UI

Sharing data between tasks

Returning data from a TaskFlow function

Connecting a classic operator to a TaskFlow task

Understanding XCom limitations

अध्याय शुरू करें

2

Dynamic and Data-Aware Pipelines

From there, you'll run tasks in parallel with dynamic task mapping, schedule Dags by data with Assets, and add human approval steps.

Dynamic Task Mapping

Processing multiple endpoints with expand

Fixing a shared parameter with partial

Pairing inputs with zip

Data-aware scheduling with Assets

Signaling data availability with an Asset outlet

Scheduling a consumer Dag on an Asset

Seeing asset-based scheduling in action

Templating, idempotency, and backfilling

Adding date-aware templating

Making a pipeline idempotent

Triggering a backfill

Human-in-the-loop workflows

Collecting form input with HITLEntryOperator

Responding to a Required Action in the UI

Branching with HITLBranchOperator

अध्याय शुरू करें

3

Preparing Dags for Production

In this chapter, you'll handle failures with retries and callbacks, save resources with deferrable sensors, and test your Dags at three levels.

Robust Dags with failure handling and retries

Configuring retries with exponential backoff

Auto-pausing a failing Dag

Choosing the right failure handling strategy

Deferrable operators

Configuring a sensor for production

Making a sensor deferrable

When to use poke, reschedule, or deferrable

Testing Airflow code

Writing a Dag integrity test

Unit testing a task function

Integration testing with dag.test()

Organizing complex Dags with Task Groups

Grouping related tasks

Nesting task groups with display names

Organizing tasks by domain

अध्याय शुरू करें

4

Building a Production SQL ETL Pipeline

In this final chapter, you'll build a SQL ETL pipeline on DuckDB, add partition-aware scheduling with Asset Partitions, and embed data quality checks.

Running SQL workloads

Aggregating sales data with SQL

Parameterizing SQL and extracting it into files

Choosing the safe approach for user input

Partition-aware scheduling with Asset Partitions

Creating a partitioned pipeline

Scheduling on partitioned assets

Watching partitioned assets trigger a downstream Dag

Data quality checks

Adding column-level quality checks

Adding table-level quality checks

Catching data issues with the right operator

From development to production

Triggering a pipeline from the CLI

Backfilling historical data

Production challenges with Airflow

Congratulations!

अध्याय शुरू करें

Building Data Pipelines with Airflow

पाठ्यक्रम
पूर्ण

उपलब्धि का प्रमाण पत्र अर्जित करें

इस प्रमाण पत्र को अपनी LinkedIn प्रोफ़ाइल, रिज्यूमे या CV में जोड़ें
इसे सोशल मीडिया पर और अपनी प्रदर्शन समीक्षा में साझा करेंअभी नामांकन करें

19 मिलियन से अधिक शिक्षार्थियों के साथ जुड़ें और आज ही Building Data Pipelines with Airflow शुरू करें!

अपना मुफ़्त खाता बनाएं

Google के साथ जारी रखें अधिक विकल्प दिखाएँ

या

जारी रखने पर, आप हमारी उपयोग की शर्तें, हमारी गोपनीयता नीति को स्वीकार करते हैं और यह भी कि आपका डेटा संयुक्त राज्य अमेरिका में संग्रहीत किया जाता है।

मोबाइल के लिए DataCamp के साथ अपने डेटा कौशल को बढ़ाएं

हमारे मोबाइल कोर्स और दैनिक 5 मिनट की कोडिंग चुनौतियों के साथ चलते-फिरते प्रगति करें।