Saltar al contenido principal
InicioData EngineeringBig Data Fundamentals with PySpark

Big Data Fundamentals with PySpark

Learn the fundamentals of working with big data with PySpark.

Comience El Curso Gratis
4 Horas16 Videos55 Ejercicios
46.474 AprendicesTrophyDeclaración de cumplimiento

Crea Tu Cuenta Gratuita

GoogleLinkedInFacebook

o

Al continuar, acepta nuestros Términos de uso, nuestra Política de privacidad y que sus datos se almacenan en los EE. UU.
Group¿Entrenar a 2 o más personas?Pruebe DataCamp para empresas

Preferido por estudiantes en miles de empresas

Descripción del curso

There's been a lot of buzz about Big Data over the past few years, and it's finally become mainstream for many companies. But what is this Big Data? This course covers the fundamentals of Big Data via PySpark. Spark is a "lightning fast cluster computing" framework for Big Data. It provides a general data processing platform engine and lets you run programs up to 100x faster in memory, or 10x faster on disk, than Hadoop. You’ll use PySpark, a Python package for Spark programming and its powerful, higher-level libraries such as SparkSQL, MLlib (for machine learning), etc. You will explore the works of William Shakespeare, analyze Fifa 2018 data and perform clustering on genomic datasets. At the end of this course, you will have gained an in-depth understanding of PySpark and its application to general Big Data analysis.
Empresas

Group¿Entrenar a 2 o más personas?

Obtenga acceso de su equipo a la biblioteca completa de DataCamp, con informes centralizados, tareas, proyectos y más
Pruebe DataCamp Para EmpresasPara obtener una solución a medida, solicite una demonstración.

En las siguientes pistas

Big Data con PySpark

Ir a la pista
  1. 1

    Introduction to Big Data analysis with Spark

    Gratuito

    This chapter introduces the exciting world of Big Data, as well as the various concepts and different frameworks for processing Big Data. You will understand why Apache Spark is considered the best framework for BigData.

    Reproducir Capítulo Ahora
    What is Big Data?
    50 xp
    The 3 V's of Big Data
    50 xp
    PySpark: Spark with Python
    50 xp
    Understanding SparkContext
    100 xp
    Interactive Use of PySpark
    100 xp
    Loading data in PySpark shell
    100 xp
    Review of functional programming in Python
    50 xp
    Use of lambda() with map()
    100 xp
    Use of lambda() with filter()
    100 xp
  2. 2

    Programming in PySpark RDD’s

    The main abstraction Spark provides is a resilient distributed dataset (RDD), which is the fundamental and backbone data type of this engine. This chapter introduces RDDs and shows how RDDs can be created and executed using RDD Transformations and Actions.

    Reproducir Capítulo Ahora
    Abstracting Data with RDDs
    50 xp
    RDDs from Parallelized collections
    100 xp
    RDDs from External Datasets
    100 xp
    Partitions in your data
    100 xp
    Basic RDD Transformations and Actions
    50 xp
    Map and Collect
    100 xp
    Filter and Count
    100 xp
    Pair RDDs in PySpark
    50 xp
    ReduceBykey and Collect
    100 xp
    SortByKey and Collect
    100 xp
    Advanced RDD Actions
    50 xp
    CountingBykeys
    100 xp
    Create a base RDD and transform it
    100 xp
    Remove stop words and reduce the dataset
    100 xp
    Print word frequencies
    100 xp
  3. 3

    PySpark SQL & DataFrames

    In this chapter, you'll learn about Spark SQL which is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as a distributed SQL query engine. This chapter shows how Spark SQL allows you to use DataFrames in Python.

    Reproducir Capítulo Ahora
    Abstracting Data with DataFrames
    50 xp
    RDD to DataFrame
    100 xp
    Loading CSV into DataFrame
    100 xp
    Operating on DataFrames in PySpark
    50 xp
    Inspecting data in PySpark DataFrame
    100 xp
    PySpark DataFrame subsetting and cleaning
    100 xp
    Filtering your DataFrame
    100 xp
    Interacting with DataFrames using PySpark SQL
    50 xp
    Running SQL Queries Programmatically
    100 xp
    SQL queries for filtering Table
    100 xp
    Data Visualization in PySpark using DataFrames
    50 xp
    PySpark DataFrame visualization
    100 xp
    Part 1: Create a DataFrame from CSV file
    100 xp
    Part 2: SQL Queries on DataFrame
    100 xp
    Part 3: Data visualization
    100 xp
  4. 4

    Machine Learning with PySpark MLlib

    PySpark MLlib is the Apache Spark scalable machine learning library in Python consisting of common learning algorithms and utilities. Throughout this last chapter, you'll learn important Machine Learning algorithms. You will build a movie recommendation engine and a spam filter, and use k-means clustering.

    Reproducir Capítulo Ahora
    Overview of PySpark MLlib
    50 xp
    PySpark ML libraries
    50 xp
    PySpark MLlib algorithms
    100 xp
    Collaborative filtering
    50 xp
    Loading Movie Lens dataset into RDDs
    100 xp
    Model training and predictions
    100 xp
    Model evaluation using MSE
    100 xp
    Classification
    50 xp
    Loading spam and non-spam data
    100 xp
    Feature hashing and LabelPoint
    100 xp
    Logistic Regression model training
    100 xp
    Clustering
    50 xp
    Loading and parsing the 5000 points data
    100 xp
    K-means training
    100 xp
    Visualizing clusters
    100 xp
    Congratulations!
    50 xp
Empresas

Group¿Entrenar a 2 o más personas?

Obtenga acceso de su equipo a la biblioteca completa de DataCamp, con informes centralizados, tareas, proyectos y más

En las siguientes pistas

Big Data con PySpark

Ir a la pista

Sets De Datos

Complete ShakespeareMovie ratings5000 pointsFIFA 2018PeopleSpamHam

Colaboradores

Collaborator's avatar
Hadrien Lacroix
Collaborator's avatar
Chester Ismay

Requisitos Previos

Introduction to Python
Upendra Kumar Devisetty HeadshotUpendra Kumar Devisetty

Science Analyst at CyVerse

Ver Mas

¿Qué tienen que decir otros alumnos?

    Únete a 13 millones de estudiantes y empeza Big Data Fundamentals with PySpark hoy!

    Crea Tu Cuenta Gratuita

    GoogleLinkedInFacebook

    o

    Al continuar, acepta nuestros Términos de uso, nuestra Política de privacidad y que sus datos se almacenan en los EE. UU.