Skip to main content

Building Data Engineering Pipelines in Python

Learn how to build and test data engineering pipelines in Python using PySpark and Apache Airflow.

Start Course for Free
4 Hours14 Videos52 Exercises20,060 Learners3950 XP

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.

Loved by learners at thousands of companies


Course Description

Build a Data Pipeline in Python

Learn how to use Python to build data engineering pipelines with this 4-hour course.

In any data-driven company, you will undoubtedly cross paths with data engineers. Among other things, they facilitate work by making data readily available to everyone within the organization and may also bring machine learning models into production.

One way to speed up this process is through building an understanding of what it means to bring processes into production and what features are of high-grade code. In this course, we’ll be looking at various data pipelines that data engineers build and how some of the tools they use can help you get your models into production or run repetitive tasks consistently and efficiently.

Use PySpark to Create a Data Transformation Pipeline

In this course, we illustrate common elements of data engineering pipelines. In Chapter 1, you will learn what a data platform is and how to ingest data.

Chapter 2 will go one step further with cleaning and transforming data, using PySpark to create a data transformation pipeline.

In Chapter 3, you will learn how to safely deploy code, looking at the different forms of testing. Finally, in Chapter 4, you will schedule complex dependencies between applications, using the basics of Apache Airflow to trigger the various components of an ETL pipeline on a certain time schedule and execute tasks in a specific order.

Learn How to Manage and Orchestrate Workflows

By the end of this course, you’ll have an understanding of building data pipelines in Python for data engineering. You’ll also have a knowledge of how to orchestrate and manage your workflows using DAG schedules and Apache Airflow for automated testing.
  1. 1

    Ingesting Data

    Free

    After seeing this chapter, you will be able to explain what a data platform is, how data ends up in it, and how data engineers structure its foundations. You will be able to ingest data from a RESTful API into the data platform’s data lake using a self-written ingestion pipeline, made using Singer’s taps and targets.

    Play Chapter Now
    Components of a data platform
    50 xp
    Dashboards providing business value
    50 xp
    Snapshots in a data lake
    50 xp
    The data catalog
    50 xp
    Introduction to data ingestion with Singer
    50 xp
    Working with JSON
    100 xp
    Specifying the schema of the data
    100 xp
    Running an ingestion pipeline with Singer
    50 xp
    Properly propagating state
    50 xp
    Communicating with an API
    100 xp
    Streaming records
    100 xp
    Chain taps and targets
    100 xp
  2. 3

    Testing your data pipeline

    Stating “it works on my machine” is not a guarantee it will work reliably elsewhere and in the future. Requirements for your project will change. In this chapter, we explore different forms of testing and learn how to write unit tests for our PySpark data transformation pipeline, so that we make robust and reusable parts.

    Play Chapter Now
  3. 4

    Managing and orchestrating a workflow

    We will explore the basics of Apache Airflow, a popular piece of software that allows you to trigger the various components of an ETL pipeline on a certain time schedule and execute tasks in a specific order. Here too, we illustrate how a deployment of Apache Airflow can be tested automatically.

    Play Chapter Now

Datasets

prices.csvpurchased.csvratings.csvratings_with_incomplete_rows.csvratings_with_invalid_rows.csv

Collaborators

hadrien-d4e73b49-bc29-46b7-a485-2f598f38e3b9
Hadrien Lacroix
hillary-green-lerman
Hillary Green-Lerman
Kai  Zhang Headshot

Kai Zhang

Data Engineer at Data Minded

Kai is a data engineer, data scientist and solutions architect who is passionate about delivering business value and actionable insights through well architected data products. Kai holds a Master's degree in Electrical Engineering from KU Leuven.
See More
Oliver Willekens Headshot

Oliver Willekens

Data Engineer at Data Minded

Oliver is a Data Engineer and Data Scientist. He’s also an educator in these fields. Enthusiastic life-long learner and automator, he has a PhD in photonics (think lasers here). Experienced in Python and Scala, he is often found helping on StackOverflow, usually in the Python and NumPy tags.
See More

What do other learners have to say?

Join over 10 million learners and start Building Data Engineering Pipelines in Python today!

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.