Building Data Engineering Pipelines in Python

Learn how to build data engineering pipelines in Python.
Start Course for Free
4 Hours14 Videos52 Exercises14,402 Learners
3950 XP

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).

Loved by learners at thousands of companies

Course Description

In any data-driven company, you will undoubtedly cross paths with data engineers. Among other things, they facilitate some of your work by making data readily available to everyone within the organization, and possibly in bringing machine learning models into production. One way to speed up this process is through building an understanding of what it means to bring processes into production and what features are of high-grade code. In this course, we’ll be looking at various data pipelines the data engineer is building, and how some of the tools he or she is using can help you in getting your models into production or run repetitive tasks consistently and efficiently. <br /> <br /> In this course, we illustrate common elements of data engineering pipelines. In Chapter 1, you will learn how to ingest data. Chapter 2 will go one step further with cleaning and transforming data. In Chapter 3, you will learn how to safely deploy code. Finally, in Chapter 4 you will schedule complex dependencies between applications. <br /> <br /> <i>Building Data Engineering Pipelines covers new technologies and material, so we recommend that you have a strong understanding of the prerequisites to get the most out of this course.</i>

  1. 1

    Ingesting Data

    After seeing this chapter, you will be able to explain what a data platform is, how data ends up in it, and how data engineers structure its foundations. You will be able to ingest data from a RESTful API into the data platform’s data lake using a self-written ingestion pipeline, made using Singer’s taps and targets.
    Play Chapter Now
  2. 2

    Creating a data transformation pipeline with PySpark

    You will learn how to process data in the data lake in a structured way using PySpark. Of course, you must first understand when PySpark is the right choice for the job.
    Play Chapter Now
  3. 3

    Testing your data pipeline

    Stating “it works on my machine” is not a guarantee it will work reliably elsewhere and in the future. Requirements for your project will change. In this chapter, we explore different forms of testing and learn how to write unit tests for our PySpark data transformation pipeline, so that we make robust and reusable parts.
    Play Chapter Now
  4. 4

    Managing and orchestrating a workflow

    We will explore the basics of Apache Airflow, a popular piece of software that allows you to trigger the various components of an ETL pipeline on a certain time schedule and execute tasks in a specific order. Here too, we illustrate how a deployment of Apache Airflow can be tested automatically.
    Play Chapter Now
In the following tracks
Data Engineer
Hadrien LacroixHillary Green-Lerman
Kai  Zhang Headshot

Kai Zhang

Data Engineer at Data Minded
Kai is a data engineer, data scientist and solutions architect who is passionate about delivering business value and actionable insights through well architected data products. Kai holds a Master's degree in Electrical Engineering from KU Leuven.
See More
Oliver Willekens Headshot

Oliver Willekens

Data Engineer at Data Minded
Oliver is a Data Engineer and Data Scientist. He’s also an educator in these fields. Enthusiastic life-long learner and automator, he has a PhD in photonics (think lasers here). Experienced in Python and Scala, he is often found helping on StackOverflow, usually in the Python and NumPy tags.
See More

What do other learners have to say?

I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.

Devon Edwards Joseph
Lloyds Banking Group

DataCamp is the top resource I recommend for learning data science.

Louis Maiden
Harvard Business School

DataCamp is by far my favorite website to learn from.

Ronald Bowers
Decision Science Analytics, USAA