课程

Big Data Fundamentals with PySpark

高级技能水平

更新时间 2025年2月

Learn the fundamentals of working with big data with PySpark.

免费开始课程

SparkData Engineering4 小时16 视频55 练习4,600 经验值64,506成就声明

创建您的免费帐户

或

继续操作即表示您接受我们的《使用条款》和《隐私政策》，并同意您的数据存储在美国。

深受数千家公司学习者的喜爱

培训2人或更多？

试用DataCamp for Business

课程描述

There's been a lot of buzz about Big Data over the past few years, and it's finally become mainstream for many companies. But what is this Big Data? This course covers the fundamentals of Big Data via PySpark. Spark is a "lightning fast cluster computing" framework for Big Data. It provides a general data processing platform engine and lets you run programs up to 100x faster in memory, or 10x faster on disk, than Hadoop. You’ll use PySpark, a Python package for Spark programming and its powerful, higher-level libraries such as SparkSQL, MLlib (for machine learning), etc. You will explore the works of William Shakespeare, analyze Fifa 2018 data and perform clustering on genomic datasets. At the end of this course, you will have gained an in-depth understanding of PySpark and its application to general Big Data analysis.

先决条件

Introduction to Python

1

Introduction to Big Data analysis with Spark

This chapter introduces the exciting world of Big Data, as well as the various concepts and different frameworks for processing Big Data. You will understand why Apache Spark is considered the best framework for BigData.

What is Big Data?

50 经验值

The 3 V's of Big Data

50 经验值

PySpark: Spark with Python

50 经验值

Understanding SparkContext

100 经验值

Interactive Use of PySpark

100 经验值

Loading data in PySpark shell

100 经验值

Review of functional programming in Python

50 经验值

Use of lambda() with map()

100 经验值

Use of lambda() with filter()

100 经验值

2

Programming in PySpark RDD’s

The main abstraction Spark provides is a resilient distributed dataset (RDD), which is the fundamental and backbone data type of this engine. This chapter introduces RDDs and shows how RDDs can be created and executed using RDD Transformations and Actions.

Abstracting Data with RDDs

50 经验值

RDDs from Parallelized collections

100 经验值

RDDs from External Datasets

100 经验值

Partitions in your data

100 经验值

Basic RDD Transformations and Actions

50 经验值

Map and Collect

100 经验值

Filter and Count

100 经验值

Pair RDDs in PySpark

50 经验值

ReduceBykey and Collect

100 经验值

SortByKey and Collect

100 经验值

Advanced RDD Actions

50 经验值

CountingBykeys

100 经验值

Create a base RDD and transform it

100 经验值

Remove stop words and reduce the dataset

100 经验值

Print word frequencies

100 经验值

3

PySpark SQL & DataFrames

In this chapter, you'll learn about Spark SQL which is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as a distributed SQL query engine. This chapter shows how Spark SQL allows you to use DataFrames in Python.

Abstracting Data with DataFrames

50 经验值

RDD to DataFrame

100 经验值

Loading CSV into DataFrame

100 经验值

Operating on DataFrames in PySpark

50 经验值

Inspecting data in PySpark DataFrame

100 经验值

PySpark DataFrame subsetting and cleaning

100 经验值

Filtering your DataFrame

100 经验值

Interacting with DataFrames using PySpark SQL

50 经验值

Running SQL Queries Programmatically

100 经验值

SQL queries for filtering Table

100 经验值

Data Visualization in PySpark using DataFrames

50 经验值

PySpark DataFrame visualization

100 经验值

Part 1: Create a DataFrame from CSV file

100 经验值

Part 2: SQL Queries on DataFrame

100 经验值

Part 3: Data visualization

100 经验值

4

Machine Learning with PySpark MLlib

PySpark MLlib is the Apache Spark scalable machine learning library in Python consisting of common learning algorithms and utilities. Throughout this last chapter, you'll learn important Machine Learning algorithms. You will build a movie recommendation engine and a spam filter, and use k-means clustering.

Overview of PySpark MLlib

50 经验值

PySpark ML libraries

50 经验值

PySpark MLlib algorithms

100 经验值

Collaborative filtering

50 经验值

Loading Movie Lens dataset into RDDs

100 经验值

Model training and predictions

100 经验值

Model evaluation using MSE

100 经验值

Classification

50 经验值

Loading spam and non-spam data

100 经验值

Feature hashing and LabelPoint

100 经验值

Logistic Regression model training

100 经验值

50 经验值

Loading and parsing the 5000 points data

100 经验值

K-means training

100 经验值

Visualizing clusters

100 经验值

Congratulations!

50 经验值

Big Data Fundamentals with PySpark

课程完成

获得成就证明

将此证书添加到你的 LinkedIn 档案、简历或履历中
在社交媒体和绩效评估中分享立即注册

加入超过19百万学习者，今天就开始Big Data Fundamentals with PySpark！

创建您的免费帐户

或

继续操作即表示您接受我们的《使用条款》和《隐私政策》，并同意您的数据存储在美国。

通过 DataCamp for Mobile 提升您的数据技能

随时随地通过我们的移动课程和每日 5 分钟编程挑战提升技能。