Machine Learning with PySpark
Learn how to make predictions with Apache Spark.
Start Course for Free4 Hours16 Videos56 Exercises15,943 Learners4550 XPBig Data with PySpark TrackMachine Learning Scientist Track
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).Loved by learners at thousands of companies
Course Description
Spark is a powerful, general purpose tool for working with Big Data. Spark transparently handles the distribution of compute tasks across a cluster. This means that operations are fast, but it also allows you to focus on the analysis rather than worry about technical details. In this course you'll learn how to get data into Spark and then delve into the three fundamental Spark Machine Learning algorithms: Linear Regression, Logistic Regression/Classifiers, and creating pipelines. Along the way you'll analyse a large dataset of flight delays and spam text messages. With this background you'll be ready to harness the power of Spark and apply it on your own Machine Learning projects!
- 1
Introduction
FreeSpark is a framework for working with Big Data. In this chapter you'll cover some background about Spark and Machine Learning. You'll then find out how to connect to Spark using Python and load CSV data.
- 2
Classification
Now that you are familiar with getting data into Spark, you'll move onto building two types of classification model: Decision Trees and Logistic Regression. You'll also find out about a few approaches to data preparation.
Data Preparation50 xpRemoving columns and rows100 xpColumn manipulation100 xpCategorical columns100 xpAssembling columns100 xpDecision Tree50 xpTrain/test split100 xpBuild a Decision Tree100 xpEvaluate the Decision Tree100 xpLogistic Regression50 xpBuild a Logistic Regression model100 xpEvaluate the Logistic Regression model100 xpTurning Text into Tables50 xpPunctuation, numbers and tokens100 xpStop words and hashing100 xpTraining a spam classifier100 xp - 3
Regression
Next you'll learn to create Linear Regression models. You'll also find out how to augment your data by engineering new predictors as well as a robust approach to selecting only the most relevant predictors.
One-Hot Encoding50 xpEncoding flight origin100 xpEncoding shirt sizes50 xpRegression50 xpFlight duration model: Just distance100 xpInterpreting the coefficients100 xpFlight duration model: Adding origin airport100 xpInterpreting the coefficients100 xpBucketing & Engineering50 xpBucketing departure time100 xpFlight duration model: Adding departure time100 xpRegularization50 xpFlight duration model: More features!100 xpFlight duration model: Regularisation!100 xp - 4
Ensembles & Pipelines
Finally you'll learn how to make your models more efficient. You'll find out how to use pipelines to make your code clearer and easier to maintain. Then you'll use cross-validation to better test your models and select good model parameters. Finally you'll dabble in two types of ensemble model.
Pipeline50 xpFlight duration model: Pipeline stages100 xpFlight duration model: Pipeline model100 xpSMS spam pipeline100 xpCross-Validation50 xpCross validating simple flight duration model100 xpCross validating flight duration model pipeline100 xpGrid Search50 xpOptimizing flights linear regression100 xpDissecting the best flight duration model100 xpSMS spam optimised100 xpHow many models for grid search?50 xpEnsemble50 xpDelayed flights with Gradient-Boosted Trees100 xpDelayed flights with a Random Forest100 xpEvaluating Random Forest100 xpClosing thoughts50 xp

Andrew Collier
Data Scientist @ Exegetic Analytics
Andrew Collier is a Data Scientist, working mostly in R and Python but also dabbling in a wide range of other technologies. When not in front of a computer he spends time with his family and runs obsessively.
What do other learners have to say?
I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.
Devon Edwards Joseph
Lloyds Banking Group
DataCamp is the top resource I recommend for learning data science.
Louis Maiden
Harvard Business School
DataCamp is by far my favorite website to learn from.
Ronald Bowers
Decision Science Analytics, USAA
Join over 9 million learners and start Machine Learning with PySpark today!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).