R has great ways to handle working with big data including programming in parallel and interfacing with Spark. In this track, you'll learn how to write scalable and efficient R code and ways to visualize it too.

Work with big data in R via parallel programming, interfacing with Spark, writing scalable & efficient R code, and learn ways to visualize big data.

with R

Big Data

In order to make your code go faster, you need to know how long it takes to run. This chapter introduces the idea of benchmarking your code.

The Art of Benchmarking

R is flexible because you can often solve a single problem in many different ways. Some ways can be several orders of magnitude faster than the others. This chapter teaches you how to write fast base R code.

Fine Tuning: Efficient Base R

Profiling helps you locate the bottlenecks in your code. This chapter teaches you how to visualize the bottlenecks using the `profvis` package.

Diagnosing Problems: Code Profiling

Some problems can be solved faster using multiple cores on your machine. This chapter shows you how to write R code that runs in parallel.

Turbo Charged Code: Parallel Programming

Information on 45,000 movies

The beauty of R is that it is built for performing data analysis. The downside is that sometimes R can be slow, thereby obstructing our analysis. For this reason, it is essential to become familiar with the main techniques for speeding up your analysis, so you can reduce computational time and get insights as quickly as possible.

Intermediate R

Learn to write faster R code, discover benchmarking and profiling, and unlock the secrets of parallel programming.

Writing Efficient R Code

Programming

Learn different strategies for plotting big data using ggplot2, including calculating and plotting summary statistics, various techniques to deal with overplotting, and principles of small multiples with faceting, which leads into Trelliscope.

General strategies for visualizing big data

In the previous chapter you saw how faceting can be used as a powerful technique for visualizing a lot of data that can be naturally partitioned in some meaningful way. Now, using the trelliscopejs package with ggplot2, you will learn how to create faceted visualizations when the number of partitions in the data becomes too large to effectively view in a single screen.

ggplot2 + TrelliscopeJS

The ggplot2 + trelliscopejs interface is easy to use, but trelliscopejs also provides a faceted plotting mechanism that gives you much more flexibility in what plotting system you use and how to specify cognostics. You will learn all about that in this chapter!

Trelliscope in the Tidyverse

<a href="https://www.bixi.com" target="_blank">The Montreal BIXI bike network</a> provides open data for every bike ride, including the date, time, duration, and start and end stations of the ride. In this chapter, you will analyze data from over 4 million bike rides in 2017, going between 546 stations. There are many interesting exploratory questions to ask from this data and you will create exploratory visualizations ranging from summary statistics to detailed Trelliscope visualizations that will give you interesting insight into the data.

Case Study: Exploring Montreal BIXI Bike Data

Bikes

Pokémon

Top 100 routes

Stocks

Taxi rides

Having honed your visualization skills by learning ggplot2, it's now time to tackle larger datasets. In this course, you will learn several techniques for visualizing big data, with particular focus on the scalable visualization technique of faceting. You will learn how to put this technique into action using the Trelliscope approach as implemented in the trelliscopejs R package. Trelliscope plugs seamlessly into standard R workflows and produces interactive visualizations that allow you to visually explore your data in detail. By the end of this course, you will be able to easily create interactive exploratory displays of large datasets that will help you and your colleagues gain new insights into your data.

Introduction to the Tidyverse

Learn how to visualize big data in R using ggplot2 and trelliscopejs.

Visualizing Big Data with Trelliscope in R

Data Visualization

In this chapter, we cover the reasons you need to apply new techniques when data sets are larger than available RAM.  We show that importing and exporting data using the base R functions can be slow and some easy ways to remedy this. Finally, we introduce the bigmemory package. 

Working with increasingly large data sets

Now that you've got some experience using bigmemory, we're going to go through some simple data exploration and analysis techniques. In particular, we'll see how to create tables and implement the split-apply-combine approach.

Processing and Analyzing Data with bigmemory

We'll use the iotools package that can process both numeric and string data, and introduce the concept of chunk-wise processing.


Working with iotools

In the previous chapters, we've introduced the housing data and shown how to compute with data that is about as big, or bigger than, the amount of RAM on a single machine. In this chapter, we'll go through a preliminary analysis of the data, comparing various trends over time.

Case Study: A Preliminary Analysis of the Housing Data

Mortgage data (sample)

Datasets are often larger than available RAM, which causes problems for R programmers since by default all the variables are stored in memory. You’ll learn tools for processing, exploring, and analyzing data directly from disk. You’ll also implement the split-apply-combine approach and learn how to write scalable code using the bigmemory and iotools packages. In this course, you'll make use of the Federal Housing Finance Agency's data, a publicly available data set chronicling all mortgages that were held or securitized by both Federal National Mortgage Association (Fannie Mae) and Federal Home Loan Mortgage Corporation (Freddie Mac) from 2009-2015.

Learn how to write scalable code for working with big data in R using the bigmemory and iotools packages.

Scalable Data Processing in R

In which you learn how Spark and R complement each other, how to get data to and from Spark, and how to manipulate Spark data frames using dplyr syntax.

Light My Fire: Starting To Use Spark With dplyr Syntax

In which you learn more about using the <code>dplyr</code> interface to Spark, including advanced field selection, calculating groupwise statistics, and joining data frames.

Tools of the Trade: Advanced dplyr Usage

In which you learn about Spark's machine learning data transformation features, and functionality for manipulating native DataFrames.

Going Native: Use The Native Interface to Manipulate Spark DataFrames

A case study in which you learn to use <code>sparklyr</code>'s machine learning routines, by predicting the year in which a song was released.

Case Study: Learning to be a Machine: Running Machine Learning Models on Spark

Anti-join

Both-model-responses

Gbt-model-responses

Inner-join

Left-join

Predicted vs actual

Residual density

Semi-join

Timbre

Timbre parquet

Title text parquet

Track data parquet

Track data to model parquet

Track data to predict parquet

Track metadata

R is mostly optimized to help you write data analysis code quickly and readably. Apache Spark is designed to analyze huge datasets quickly. The <code>sparklyr</code> package lets you write <code>dplyr</code> R code that runs on a Spark cluster, giving you the best of both worlds. This course teaches you how to manipulate Spark DataFrames using both the <code>dplyr</code> interface and the native interface to Spark, as well as trying machine learning techniques. Throughout the course, you'll explore the Million Song Dataset.

Learn how to analyze huge datasets using Apache Spark and R using the sparklyr package.

Data Science Courses

Data Science Tracks

Our Learning Experience

Big Data with R

Create Your Free Account

Loved by learners at thousands of companies

Instructors