Building Data Engineering Pipelines in Python
Learn how to build data engineering pipelines in Python.
Start Course for Free4 Hours14 Videos52 Exercises17,882 Learners3950 XP
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).Loved by learners at thousands of companies
Course Description
In any data-driven company, you will undoubtedly cross paths with data engineers. Among other things, they facilitate some of your work by making data readily available to everyone within the organization, and possibly in bringing machine learning models into production. One way to speed up this process is through building an understanding of what it means to bring processes into production and what features are of high-grade code. In this course, we’ll be looking at various data pipelines the data engineer is building, and how some of the tools he or she is using can help you in getting your models into production or run repetitive tasks consistently and efficiently.
In this course, we illustrate common elements of data engineering pipelines. In Chapter 1, you will learn how to ingest data. Chapter 2 will go one step further with cleaning and transforming data. In Chapter 3, you will learn how to safely deploy code. Finally, in Chapter 4 you will schedule complex dependencies between applications.
Building Data Engineering Pipelines covers new technologies and material, so we recommend that you have a strong understanding of the prerequisites to get the most out of this course.
- 1
Ingesting Data
FreeAfter seeing this chapter, you will be able to explain what a data platform is, how data ends up in it, and how data engineers structure its foundations. You will be able to ingest data from a RESTful API into the data platform’s data lake using a self-written ingestion pipeline, made using Singer’s taps and targets.
Components of a data platform50 xpDashboards providing business value50 xpSnapshots in a data lake50 xpThe data catalog50 xpIntroduction to data ingestion with Singer50 xpWorking with JSON100 xpSpecifying the schema of the data100 xpRunning an ingestion pipeline with Singer50 xpProperly propagating state50 xpCommunicating with an API100 xpStreaming records100 xpChain taps and targets100 xp - 2
Creating a data transformation pipeline with PySpark
You will learn how to process data in the data lake in a structured way using PySpark. Of course, you must first understand when PySpark is the right choice for the job.
Basic introduction to PySpark50 xpReading a CSV file100 xpDefining a schema100 xpCleaning data50 xpSensible data types50 xpRemoving invalid rows100 xpFilling unknown data100 xpConditionally replacing values100 xpTransforming data with Spark50 xpSelecting and renaming columns100 xpGrouping and aggregating data100 xpPackaging your application50 xpCreating a deployable artifact100 xpSubmitting your Spark job100 xpDebugging simple errors50 xpVerifying your pipeline’s output50 xp - 3
Testing your data pipeline
Stating “it works on my machine” is not a guarantee it will work reliably elsewhere and in the future. Requirements for your project will change. In this chapter, we explore different forms of testing and learn how to write unit tests for our PySpark data transformation pipeline, so that we make robust and reusable parts.
On the importance of tests50 xpRegression errors50 xpCharacteristics of tests100 xpWriting unit tests for PySpark50 xpCreating in-memory DataFrames100 xpMaking a function more widely reusable100 xpContinuous testing50 xpA high-level view on CI/CD100 xpUnderstanding the output of pytest50 xpImproving style guide compliancy100 xp - 4
Managing and orchestrating a workflow
We will explore the basics of Apache Airflow, a popular piece of software that allows you to trigger the various components of an ETL pipeline on a certain time schedule and execute tasks in a specific order. Here too, we illustrate how a deployment of Apache Airflow can be tested automatically.
Modern day workflow management50 xpSpecifying the DAG schedule100 xpSetting up daily tasks50 xpSpecifying operator dependencies100 xpBuilding a data pipeline with Airflow50 xpPreparing a DAG for daily pipelines100 xpScheduling bash scripts with Airflow100 xpScheduling Spark jobs with Airflow100 xpScheduling the full data pipeline with Airflow100 xpDeploying Airflow50 xpAirflow’s executors50 xpRecovering from deployed but broken DAGs100 xpRunning tests on Airflow100 xpFinal thoughts50 xp
Hadrien Lacroix
Hillary Green-Lerman
Kai Zhang
Data Engineer at Data Minded
Kai is a data engineer, data scientist and solutions architect who is passionate about delivering business value and actionable insights through well architected data products. Kai holds a Master's degree in Electrical Engineering from KU Leuven.
Oliver Willekens
Data Engineer at Data Minded
Oliver is a Data Engineer and Data Scientist. He’s also an educator in these fields. Enthusiastic life-long learner and automator, he has a PhD in photonics (think lasers here). Experienced in Python and Scala, he is often found helping on StackOverflow, usually in the Python and NumPy tags.
What do other learners have to say?
I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.
Devon Edwards Joseph
Lloyds Banking Group
DataCamp is the top resource I recommend for learning data science.
Louis Maiden
Harvard Business School
DataCamp is by far my favorite website to learn from.
Ronald Bowers
Decision Science Analytics, USAA
Join over 9 million learners and start Building Data Engineering Pipelines in Python today!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).