メインコンテンツへスキップ

ホーム Spark

コース

Pythonで学ぶ Spark SQL 入門

上級スキルレベル

更新日 2026/03

PythonのSQLを使用して、Sparkにおけるデータの操作方法と機械学習の特徴量セットの作成方法を学びましょう。

コースを無料で開始

SparkData Manipulation

4時間

15 ビデオ

52 演習

4,200 XP

20,363

修了証明書

何千もの企業の従業員が支持

チームのトレーニングを担当していますか？

Businessをお試しください

コース説明

前提条件

Python Toolbox PostgreSQL Summary Stats and Window Functions Introduction to PySpark

1

PySpark SQL

In this chapter you will learn how to create and query a SQL table in Spark. Spark SQL brings the expressiveness of SQL to Spark. You will also learn how to use SQL window functions in Spark. Window functions perform a calculation across rows that are related to the current row. They greatly simplify achieving results that are difficult to express using only joins and traditional aggregations. We'll use window functions to perform running sums, running differences, and other operations that are challenging to perform in basic SQL.

Creating and querying a SQL table in Spark

Create a SQL table from a dataframe

Determine the column names of a table

Window function SQL

Running sums using window function SQL

Fix the broken query

Dot notation and SQL

Aggregation, step by step

Aggregating the same column twice

Aggregate dot SQL

Convert window function from dot notation to SQL

チャプターを開始

2

Using Window Function SQL for Natural Language Processing

In this chapter, you will be loading natural language text. Then you will apply a moving window analysis to find frequent word sequences.

Loading natural language text

Loading a dataframe from a parquet file

Split and explode a text column

Using monotonically_increasing_id()

Moving window analysis

Creating context window feature data

Repartitioning the data

Common word sequences

What type of data is this

Finding common word sequences

Unique 5-tuples in sorted order

Most frequent 3-tuples per chapter

チャプターを開始

3

Caching, Logging, and the Spark UI

In the previous chapters you learned how to use the expressiveness of window function SQL. However, this expressiveness now makes it important that you understand how to properly cache dataframes and cache SQL tables. It is also important to know how to evaluate your application. You learn how to do do this using the Spark UI. You'll also learn a best practice for logging in Spark. Spark SQL brings with it another useful tool for tuning query performance issues, the query execution plan. You will learn how to use the execution plan for evaluating the provenance of a dataframe.

Practice query plans 2

Practicing caching: part 1

Practicing caching: the SQL

Practicing caching: putting it all together

Caching and uncaching tables

The Spark UI

Spark UI storage tab

Inspecting cache in the Spark UI

Practice logging

Practice logging 2

Query plans

Practice query plans

Practice reading query plans 2

チャプターを開始

4

Text Classification

Previous chapters provided you with the tools for loading raw text, tokenizing it, and extracting word sequences. This is already very useful for analysis, but it is also useful for machine learning. What you've learned now comes together by using logistic regression to classify text. By the conclusion of this chapter, you will have loaded raw natural language text data and used it to train a text classifier.

Extract Transform Select

Practicing creating a UDF

Practicing array column

Creating feature data for classification

Creating a UDF for vector data

Applying a UDF to vector data

Transforming text to vector format

Text Classification

Label the data

Split the data

Train the classifier

Predicting and evaluating

Evaluate the classifier

Predict test data

チャプターを開始

Pythonで学ぶ Spark SQL 入門

コース完了

修了証明書を取得

この修了書をLinkedInや履歴書、CVに追加しましょう
ソーシャルメディアや人事評価で共有しましょう今すぐ登録

19百万人を超える学習者と共にPythonで学ぶ Spark SQL 入門を始めましょう！

DataCamp for Mobileでデータスキルを磨きましょう

モバイルコースと毎日の 5 分間のコーディングチャレンジで、外出先でも進歩できます。