Skip to main content

Parallel Programming with Dask in Python

Learn to upscale your Python workflows to efficiently handle big data with Dask.

Start Course for Free
4 Hours15 Videos51 Exercises4150 XP

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).

Loved by learners at thousands of companies

Course Description

When working with big data, you’ll face two common obstacles: using too much memory and long runtimes. The Dask library can lower your memory use by loading chunks of data only when needed. It can lower runtimes by using all your available computing cores in parallel. Best of all, it requires very few changes to your existing Python code. In this course, you use Dask to analyze Spotify song data, process images of sign language gestures, calculate trends in weather data, analyze audio recordings, and train machine learning models on big data.

  1. 1

    Lazy Evaluation and Parallel Computing

    Free

    This chapter will teach you the basics of Dask and lazy evaluation. At the end of this chapter, you'll be able to speed up almost any Python code by using parallel processing or multi-threading. You'll learn the difference between these two task scheduling methods and which one is better under which circumstances.

    Play Chapter Now
    Introduction to Dask
    50 xp
    Lazy evaluation
    50 xp
    Delaying functions
    100 xp
    Task graphs and scheduling methods
    50 xp
    What are the different schedulers?
    100 xp
    Plotting the task graph
    100 xp
    Building delayed pipelines
    50 xp
    Analyzing songs on Spotify
    100 xp
    How danceable are songs these days?
    100 xp
    Most popular songs
    100 xp
  2. 2

    Parallel Processing of Big, Structured Data

    Here you’ll learn how to analyze big structured data using Dask arrays and Dask DataFrames. You'll learn how everything you know about NumPy and pandas can easily be applied to data that is too large to fit in memory.

    Play Chapter Now
    Dask arrays
    50 xp
    Dask array chunksizes
    50 xp
    Loading and processing photos
    100 xp
    An image processing pipeline
    100 xp
    Dask DataFrames
    50 xp
    Creating Dask dataframes from CSVs
    100 xp
    Read Dask DataFrames from Parquet
    100 xp
    Summertime grooves
    100 xp
    Multidimensional arrays
    50 xp
    Exploring HDF5 files
    50 xp
    Dask arrays from HDF5 datasets
    100 xp
    Dask arrays from Zarr datasets
    100 xp
    Xarray
    50 xp
    Exploratory data analysis with xarray
    100 xp
    Monthly mean temperatures
    100 xp
    Calculating the trend in European temperatures
    100 xp
  3. 3

    Dask Bags for Unstructured Data

    Process any kind of data. You'll learn how Dask bags can be used to efficiently process unstructured text data, semi-structured JSON data, and even recorded audio.

    Play Chapter Now
    Introduction to Dask bags
    50 xp
    Creating a Dask bag
    100 xp
    Creating a bag from saved text
    100 xp
    String operations
    100 xp
    Dask bag operations
    50 xp
    Loading JSON data
    100 xp
    Filtering Dask bags
    100 xp
    Chaining operations
    100 xp
    Converting unstructured data to a DataFrame
    50 xp
    Restructuring a dictionary
    100 xp
    Converting to DataFrame
    100 xp
    Using any data in Dask bags
    50 xp
    Loading wav data
    100 xp
    Constructing custom Dask bags
    100 xp
    Processing unstructured audio
    100 xp
  4. 4

    Dask Machine Learning and Final Pieces

    Harness the power of Dask to train machine learning models. You'll learn how to train machine learning models on big data using the Dask-ML package, and how to split Dask calculations across a mixture of processes and threads for even greater computing speed.

    Play Chapter Now
    Using processes and threads
    50 xp
    Which scheduler will be used?
    50 xp
    Clusters and clients
    100 xp
    Training machine learning models on big datasets
    50 xp
    Using Dask to train a linear model
    100 xp
    Making lazy predictions
    100 xp
    Preprocessing big datasets
    50 xp
    Lazily transforming training data
    100 xp
    Lazy train-test split
    100 xp
    Wrap-up
    50 xp

Datasets

Spotify Songs - CSVSpotify Songs - ParquetEuropean Rainfall - HDF5European Rainfall - ZarrTripadvisor Hotel ReviewsPoliticians

Collaborators

james-datacampJames Chapmanamy-4121b590-cc52-442a-9779-03eb58089e08Amy Peterson

Prerequisites

Data Manipulation with pandasPython Data Science Toolbox (Part 2)
James Fulton Headshot

James Fulton

Climate Informatics Researcher

James is a PhD researcher at the University of Edinburgh, where he tutors computing, machine learning, data analysis, and statistical physics. His research involves using and developing machine learning algorithms to extract space-time patterns from climate records and climate models. He has held visiting researcher roles, working on planet-scale data analysis and modeling, at the University of Oxford and Queen's University Belfast and has a masters in physics where he specialized in quantum simulation. In a previous life, he was employed as a data scientist in the insurance sector. When not several indents deep in Python, he performs improvised comedy.
See More

What do other learners have to say?

I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.

Devon Edwards Joseph
Lloyds Banking Group

DataCamp is the top resource I recommend for learning data science.

Louis Maiden
Harvard Business School

DataCamp is by far my favorite website to learn from.

Ronald Bowers
Decision Science Analytics, USAA