课程

Cleaning Data with PySpark

高级技能水平

更新时间 2026年2月

Learn how to clean data with Apache Spark in Python.

免费开始课程

SparkData Preparation4 小时16 视频53 练习4,150 经验值32,829成就声明

创建您的免费帐户

或

继续操作即表示您接受我们的《使用条款》和《隐私政策》，并同意您的数据存储在美国。

深受数千家公司学习者的喜爱

培训2人或更多？

试用DataCamp for Business

课程描述

Working with data is tricky - working with millions or even billions of rows is worse. Did you receive some data processing code written on a laptop with fairly pristine data? Chances are you’ve probably been put in charge of moving a basic data process from prototype to production. You may have worked with real world datasets, with missing fields, bizarre formatting, and orders of magnitude more data. Even if this is all new to you, this course helps you learn what’s needed to prepare data processes using Python with Apache Spark. You’ll learn terminology, methods, and some best practices to create a performant, maintainable, and understandable data processing platform.

先决条件

Intermediate Python Introduction to PySpark

1

DataFrame details

A review of DataFrame fundamentals and the importance of data cleaning.

Intro to data cleaning with Apache Spark

50 经验值

Data cleaning review

50 经验值

Defining a schema

100 经验值

Immutability and lazy processing

50 经验值

Immutability review

50 经验值

Using lazy processing

100 经验值

Understanding Parquet

50 经验值

Saving a DataFrame in Parquet format

100 经验值

SQL and Parquet

100 经验值

2

Manipulating DataFrames in the real world

A look at various techniques to modify the contents of DataFrames in Spark.

DataFrame column operations

50 经验值

Filtering column content with Python

100 经验值

Filtering Question #1

50 经验值

Filtering Question #2

50 经验值

Modifying DataFrame columns

100 经验值

Conditional DataFrame column operations

50 经验值

when() example

100 经验值

When / Otherwise

100 经验值

User defined functions

50 经验值

Understanding user defined functions

50 经验值

Using user defined functions in Spark

100 经验值

Partitioning and lazy processing

50 经验值

Adding an ID Field

100 经验值

IDs with different partitions

100 经验值

More ID tricks

100 经验值

3

Improving Performance

Improve data cleaning tasks by increasing performance or reducing resource requirements.

50 经验值

Caching a DataFrame

100 经验值

Removing a DataFrame from cache

100 经验值

Improve import performance

50 经验值

File size optimization

50 经验值

File import performance

100 经验值

Cluster configurations

50 经验值

Reading Spark configurations

100 经验值

Writing Spark configurations

100 经验值

Performance improvements

50 经验值

Normal joins

100 经验值

Using broadcasting on Spark joins

100 经验值

Comparing broadcast vs normal joins

100 经验值

4

Complex processing and data pipelines

Learn how to process complex real-world data using Spark and the basics of pipelines.

Introduction to data pipelines

50 经验值

Quick pipeline

100 经验值

Pipeline data issue

50 经验值

Data handling techniques

50 经验值

Removing commented lines

100 经验值

Removing invalid rows

100 经验值

Splitting into columns

100 经验值

Further parsing

100 经验值

Data validation

50 经验值

Validate rows via join

100 经验值

Examining invalid rows

100 经验值

Final analysis and delivery

50 经验值

Dog parsing

100 经验值

Per image count

100 经验值

Percentage dog pixels

100 经验值

Congratulations and next steps

50 经验值

Cleaning Data with PySpark

课程完成

获得成就证明

将此证书添加到你的 LinkedIn 档案、简历或履历中
在社交媒体和绩效评估中分享立即注册

加入超过19百万学习者，今天就开始Cleaning Data with PySpark！

创建您的免费帐户

或

继续操作即表示您接受我们的《使用条款》和《隐私政策》，并同意您的数据存储在美国。

通过 DataCamp for Mobile 提升您的数据技能

随时随地通过我们的移动课程和每日 5 分钟编程挑战提升技能。