Skip to main content

Machine Learning with PySpark

Learn how to make predictions with Apache Spark.

Start Course for Free
4 Hours16 Videos56 Exercises15,943 Learners4550 XPBig Data with PySpark TrackMachine Learning Scientist Track

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).

Loved by learners at thousands of companies

Course Description

Spark is a powerful, general purpose tool for working with Big Data. Spark transparently handles the distribution of compute tasks across a cluster. This means that operations are fast, but it also allows you to focus on the analysis rather than worry about technical details. In this course you'll learn how to get data into Spark and then delve into the three fundamental Spark Machine Learning algorithms: Linear Regression, Logistic Regression/Classifiers, and creating pipelines. Along the way you'll analyse a large dataset of flight delays and spam text messages. With this background you'll be ready to harness the power of Spark and apply it on your own Machine Learning projects!

  1. 1

    Introduction

    Free

    Spark is a framework for working with Big Data. In this chapter you'll cover some background about Spark and Machine Learning. You'll then find out how to connect to Spark using Python and load CSV data.

    Play Chapter Now
    Machine Learning & Spark
    50 xp
    Characteristics of Spark
    50 xp
    Components in a Spark Cluster
    50 xp
    Connecting to Spark
    50 xp
    Location of Spark master
    50 xp
    Creating a SparkSession
    100 xp
    Loading Data
    50 xp
    Loading flights data
    100 xp
    Loading SMS spam data
    100 xp
  2. 2

    Classification

    Now that you are familiar with getting data into Spark, you'll move onto building two types of classification model: Decision Trees and Logistic Regression. You'll also find out about a few approaches to data preparation.

    Play Chapter Now
    Data Preparation
    50 xp
    Removing columns and rows
    100 xp
    Column manipulation
    100 xp
    Categorical columns
    100 xp
    Assembling columns
    100 xp
    Decision Tree
    50 xp
    Train/test split
    100 xp
    Build a Decision Tree
    100 xp
    Evaluate the Decision Tree
    100 xp
    Logistic Regression
    50 xp
    Build a Logistic Regression model
    100 xp
    Evaluate the Logistic Regression model
    100 xp
    Turning Text into Tables
    50 xp
    Punctuation, numbers and tokens
    100 xp
    Stop words and hashing
    100 xp
    Training a spam classifier
    100 xp
  3. 3

    Regression

    Next you'll learn to create Linear Regression models. You'll also find out how to augment your data by engineering new predictors as well as a robust approach to selecting only the most relevant predictors.

    Play Chapter Now
    One-Hot Encoding
    50 xp
    Encoding flight origin
    100 xp
    Encoding shirt sizes
    50 xp
    Regression
    50 xp
    Flight duration model: Just distance
    100 xp
    Interpreting the coefficients
    100 xp
    Flight duration model: Adding origin airport
    100 xp
    Interpreting the coefficients
    100 xp
    Bucketing & Engineering
    50 xp
    Bucketing departure time
    100 xp
    Flight duration model: Adding departure time
    100 xp
    Regularization
    50 xp
    Flight duration model: More features!
    100 xp
    Flight duration model: Regularisation!
    100 xp
  4. 4

    Ensembles & Pipelines

    Finally you'll learn how to make your models more efficient. You'll find out how to use pipelines to make your code clearer and easier to maintain. Then you'll use cross-validation to better test your models and select good model parameters. Finally you'll dabble in two types of ensemble model.

    Play Chapter Now
    Pipeline
    50 xp
    Flight duration model: Pipeline stages
    100 xp
    Flight duration model: Pipeline model
    100 xp
    SMS spam pipeline
    100 xp
    Cross-Validation
    50 xp
    Cross validating simple flight duration model
    100 xp
    Cross validating flight duration model pipeline
    100 xp
    Grid Search
    50 xp
    Optimizing flights linear regression
    100 xp
    Dissecting the best flight duration model
    100 xp
    SMS spam optimised
    100 xp
    How many models for grid search?
    50 xp
    Ensemble
    50 xp
    Delayed flights with Gradient-Boosted Trees
    100 xp
    Delayed flights with a Random Forest
    100 xp
    Evaluating Random Forest
    100 xp
    Closing thoughts
    50 xp

In the following tracks

Big Data with PySparkMachine Learning Scientist

Datasets

FlightsSMS

Collaborators

hadrien-d4e73b49-bc29-46b7-a485-2f598f38e3b9Hadrien Lacroixmona-kayMona Khalil

Prerequisites

Introduction to PySparkSupervised Learning with scikit-learn
Andrew Collier Headshot

Andrew Collier

Data Scientist @ Exegetic Analytics

Andrew Collier is a Data Scientist, working mostly in R and Python but also dabbling in a wide range of other technologies. When not in front of a computer he spends time with his family and runs obsessively.
See More

What do other learners have to say?

I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.

Devon Edwards Joseph
Lloyds Banking Group

DataCamp is the top resource I recommend for learning data science.

Louis Maiden
Harvard Business School

DataCamp is by far my favorite website to learn from.

Ronald Bowers
Decision Science Analytics, USAA

Join over 9 million learners and start Machine Learning with PySpark today!

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).