Building Data Engineering Pipelines in Python

Build a Data Pipeline in Python

Learn how to use Python to build data engineering pipelines with this 4-hour course.

In any data-driven company, you will undoubtedly cross paths with data engineers. Among other things, they facilitate work by making data readily available to everyone within the organization and may also bring machine learning models into production.

One way to speed up this process is through building an understanding of what it means to bring processes into production and what features are of high-grade code. In this course, we’ll be looking at various data pipelines that data engineers build and how some of the tools they use can help you get your models into production or run repetitive tasks consistently and efficiently.

Use PySpark to Create a Data Transformation Pipeline

In this course, we illustrate common elements of data engineering pipelines. In Chapter 1, you will learn what a data platform is and how to ingest data.

Chapter 2 will go one step further with cleaning and transforming data, using PySpark to create a data transformation pipeline.

In Chapter 3, you will learn how to safely deploy code, looking at the different forms of testing. Finally, in Chapter 4, you will schedule complex dependencies between applications, using the basics of Apache Airflow to trigger the various components of an ETL pipeline on a certain time schedule and execute tasks in a specific order.

Learn How to Manage and Orchestrate Workflows

By the end of this course, you’ll have an understanding of building data pipelines in Python for data engineering. You’ll also have a knowledge of how to orchestrate and manage your workflows using DAG schedules and Apache Airflow for automated testing.