Introduction to Spark SQL in Python
Learn how to manipulate data and create machine learning feature sets in Spark using SQL in Python.
Start Course for Free4 Hours15 Videos52 Exercises13,252 Learners4200 XP
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.Loved by learners at thousands of companies
Course Description
Learn Spark SQL
If you’re familiar with SQL and have heard great things about Apache Spark, this course is for you. Apache Spark is a computing framework for processing big data, and Spark SQL is a component of Apache Spark. This four-hour course will show you how to take Spark to a new level of usefulness, using advanced SQL features, such as window functions.Over the course of four chapters, you’ll use Spark SQL to analyze time series data, extract the most common words from a text document, create feature sets from natural language text, and use them to predict the last word in a sentence using logistic regression.
Discover the Uses of Spark SQL
You’ll start by creating and querying an SQL table in Spark, as well as learning how to use SQL window functions to perform running sums, running differences, and other operations.Next, you’ll explore how to use the window function in Spark SQL for natural language processing, including using a moving window analysis to find common word sequences.
In chapter 3, you’ll learn how to use the SQL Spark UI to properly cache DataFrames and SQL tables before exploring the best practices for logging in Spark.
Finally, you use all of the skills learned so far to load and tokenize raw text before extracting word sequences. You’ll then use logistic regression to classify the text, using raw natural language data to train a text classifier.
Gain a Thorough Introduction to Spark SQL
By the end of the course, you’ll have a firm understanding of Spark SQL and will understand how Spark combines the power of distributed computing with the ease of use of Python and SQL.- 1
PySpark SQL
FreeIn this chapter you will learn how to create and query a SQL table in Spark. Spark SQL brings the expressiveness of SQL to Spark. You will also learn how to use SQL window functions in Spark. Window functions perform a calculation across rows that are related to the current row. They greatly simplify achieving results that are difficult to express using only joins and traditional aggregations. We'll use window functions to perform running sums, running differences, and other operations that are challenging to perform in basic SQL.
Creating and querying a SQL table in Spark50 xpCreate a SQL table from a dataframe100 xpDetermine the column names of a table100 xpWindow function SQL50 xpRunning sums using window function SQL100 xpFix the broken query100 xpDot notation and SQL50 xpAggregation, step by step100 xpAggregating the same column twice100 xpAggregate dot SQL100 xpConvert window function from dot notation to SQL100 xp - 2
Using Window Function SQL for Natural Language Processing
In this chapter, you will be loading natural language text. Then you will apply a moving window analysis to find frequent word sequences.
Loading natural language text50 xpLoading a dataframe from a parquet file100 xpSplit and explode a text column100 xpUsing monotonically_increasing_id()50 xpMoving window analysis50 xpCreating context window feature data100 xpRepartitioning the data100 xpCommon word sequences50 xpWhat type of data is this50 xpFinding common word sequences100 xpUnique 5-tuples in sorted order100 xpMost frequent 3-tuples per chapter100 xp - 3
Caching, Logging, and the Spark UI
In the previous chapters you learned how to use the expressiveness of window function SQL. However, this expressiveness now makes it important that you understand how to properly cache dataframes and cache SQL tables. It is also important to know how to evaluate your application. You learn how to do do this using the Spark UI. You'll also learn a best practice for logging in Spark. Spark SQL brings with it another useful tool for tuning query performance issues, the query execution plan. You will learn how to use the execution plan for evaluating the provenance of a dataframe.
Caching50 xpPractice query plans 225 xpPracticing caching: part 1100 xpPracticing caching: the SQL100 xpPracticing caching: putting it all together50 xpCaching and uncaching tables100 xpThe Spark UI50 xpSpark UI storage tab50 xpInspecting cache in the Spark UI50 xpLogging50 xpPractice logging100 xpPractice logging 2100 xpQuery plans50 xpPractice query plans100 xpPractice reading query plans 2100 xp - 4
Text Classification
Previous chapters provided you with the tools for loading raw text, tokenizing it, and extracting word sequences. This is already very useful for analysis, but it is also useful for machine learning. What you've learned now comes together by using logistic regression to classify text. By the conclusion of this chapter, you will have loaded raw natural language text data and used it to train a text classifier.
Extract Transform Select50 xpPracticing creating a UDF100 xpPracticing array column100 xpCreating feature data for classification50 xpCreating a UDF for vector data100 xpApplying a UDF to vector data100 xpTransforming text to vector format100 xpText Classification50 xpLabel the data100 xpSplit the data100 xpTrain the classifier100 xpPredicting and evaluating50 xpEvaluate the classifier100 xpPredict test data100 xpRecap50 xp
Collaborators
Mark Plutowski
Big Data Architect & Scientist @ Flipboard
Mark Plutowski received his PhD in Computer Science from UCSD with emphasis in machine learning and econometrics. He holds 29 published patents, has been working with cloud apis and hadoop ecosystem for over ten years, and has been involved with hundreds of interviews.
What do other learners have to say?
FAQs
Join over 11 million learners and start Introduction to Spark SQL in Python today!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.