R has great ways to handle working with big data including programming in parallel and interfacing with Spark. In this track, you'll learn how to write scalable and efficient R code and ways to visualize it too.

Work with big data in R via parallel programming, interfacing with Spark, writing scalable & efficient R code, and learn ways to visualize big data.

with R

Big Data

In order to make your code go faster, you need to know how long it takes to run. This chapter introduces the idea of benchmarking your code.

The Art of Benchmarking

R is flexible because you can often solve a single problem in many different ways. Some ways can be several orders of magnitude faster than the others. This chapter teaches you how to write fast base R code.

Fine Tuning: Efficient Base R

Profiling helps you locate the bottlenecks in your code. This chapter teaches you how to visualize the bottlenecks using the `profvis` package.

Diagnosing Problems: Code Profiling

Some problems can be solved faster using multiple cores on your machine. This chapter shows you how to write R code that runs in parallel.

Turbo Charged Code: Parallel Programming

Information on 45,000 movies

The beauty of R is that it is built for performing data analysis. The downside is that sometimes R can be slow, thereby obstructing our analysis. For this reason, it is essential to become familiar with the main techniques for speeding up your analysis, so you can reduce computational time and get insights as quickly as possible.

Intermediate R

Learn to write faster R code, discover benchmarking and profiling, and unlock the secrets of parallel programming.

Writing Efficient R Code

Programming

Learn different strategies for plotting big data using ggplot2, including calculating and plotting summary statistics, various techniques to deal with overplotting, and principles of small multiples with faceting, which leads into Trelliscope.

General strategies for visualizing big data

In the previous chapter you saw how faceting can be used as a powerful technique for visualizing a lot of data that can be naturally partitioned in some meaningful way. Now, using the trelliscopejs package with ggplot2, you will learn how to create faceted visualizations when the number of partitions in the data becomes too large to effectively view in a single screen.

ggplot2 + TrelliscopeJS

The ggplot2 + trelliscopejs interface is easy to use, but trelliscopejs also provides a faceted plotting mechanism that gives you much more flexibility in what plotting system you use and how to specify cognostics. You will learn all about that in this chapter!

Trelliscope in the Tidyverse

<a href="https://www.bixi.com" target="_blank">The Montreal BIXI bike network</a> provides open data for every bike ride, including the date, time, duration, and start and end stations of the ride. In this chapter, you will analyze data from over 4 million bike rides in 2017, going between 546 stations. There are many interesting exploratory questions to ask from this data and you will create exploratory visualizations ranging from summary statistics to detailed Trelliscope visualizations that will give you interesting insight into the data.

Case Study: Exploring Montreal BIXI Bike Data

Bikes

Pokémon

Top 100 routes

Stocks

Taxi rides

Having honed your visualization skills by learning ggplot2, it's now time to tackle larger datasets. In this course, you will learn several techniques for visualizing big data, with particular focus on the scalable visualization technique of faceting. You will learn how to put this technique into action using the Trelliscope approach as implemented in the trelliscopejs R package. Trelliscope plugs seamlessly into standard R workflows and produces interactive visualizations that allow you to visually explore your data in detail. By the end of this course, you will be able to easily create interactive exploratory displays of large datasets that will help you and your colleagues gain new insights into your data.

Introduction to the Tidyverse

Learn how to visualize big data in R using ggplot2 and trelliscopejs.

Visualizing Big Data with Trelliscope in R

Data Visualization

In this chapter, we cover the reasons you need to apply new techniques when data sets are larger than available RAM.  We show that importing and exporting data using the base R functions can be slow and some easy ways to remedy this. Finally, we introduce the bigmemory package. 

Working with increasingly large data sets

Now that you've got some experience using bigmemory, we're going to go through some simple data exploration and analysis techniques. In particular, we'll see how to create tables and implement the split-apply-combine approach.

Processing and Analyzing Data with bigmemory

We'll use the iotools package that can process both numeric and string data, and introduce the concept of chunk-wise processing.


Working with iotools

In the previous chapters, we've introduced the housing data and shown how to compute with data that is about as big, or bigger than, the amount of RAM on a single machine. In this chapter, we'll go through a preliminary analysis of the data, comparing various trends over time.

Case Study: A Preliminary Analysis of the Housing Data

Mortgage data (sample)

Datasets are often larger than available RAM, which causes problems for R programmers since by default all the variables are stored in memory. You’ll learn tools for processing, exploring, and analyzing data directly from disk. You’ll also implement the split-apply-combine approach and learn how to write scalable code using the bigmemory and iotools packages. In this course, you'll make use of the Federal Housing Finance Agency's data, a publicly available data set chronicling all mortgages that were held or securitized by both Federal National Mortgage Association (Fannie Mae) and Federal Home Loan Mortgage Corporation (Freddie Mac) from 2009-2015.

Learn how to write scalable code for working with big data in R using the bigmemory and iotools packages.

Scalable Data Processing in R

In which you learn how Spark and R complement each other, how to get data to and from Spark, and how to manipulate Spark data frames using dplyr syntax.

Light My Fire: Starting To Use Spark With dplyr Syntax

In which you learn more about using the <code>dplyr</code> interface to Spark, including advanced field selection, calculating groupwise statistics, and joining data frames.

Tools of the Trade: Advanced dplyr Usage

In which you learn about Spark's machine learning data transformation features, and functionality for manipulating native DataFrames.

Going Native: Use The Native Interface to Manipulate Spark DataFrames

A case study in which you learn to use <code>sparklyr</code>'s machine learning routines, by predicting the year in which a song was released.

Case Study: Learning to be a Machine: Running Machine Learning Models on Spark

Anti-join

Both-model-responses

Gbt-model-responses

Inner-join

Left-join

Predicted vs actual

Residual density

Semi-join

Timbre

Timbre parquet

Title text parquet

Track data parquet

Track data to model parquet

Track data to predict parquet

Track metadata

R is mostly optimized to help you write data analysis code quickly and readably. Apache Spark is designed to analyze huge datasets quickly. The <code>sparklyr</code> package lets you write <code>dplyr</code> R code that runs on a Spark cluster, giving you the best of both worlds. This course teaches you how to manipulate Spark DataFrames using both the <code>dplyr</code> interface and the native interface to Spark, as well as trying machine learning techniques. Throughout the course, you'll explore the Million Song Dataset.

Learn how to analyze huge datasets using Apache Spark and R using the sparklyr package.

Introduction to Spark with sparklyr in R

In order to take advantage of parallel environment, the application needs to be split into pieces. In this introductory chapter, you will learn about different ways of partitioning and how it fits different hardware configurations. You will also be introduced to various R packages that support parallel programming.

Can I Run My Application in Parallel?

This chapter will dive deeper into the parallel package. You'll learn about the various backends and their differences and get a deep understanding about the workhorse of the package, namely the clusterApply() function. Strategies for task segmentation including their pitfalls will also be discussed.

The parallel Package

In this chapter, you will look at two user-contributed packages, namely foreach and future.apply, which make parallel programming in R even easier. They are built on top of the parallel and future packages. In the last lesson of this chapter, you will learn about the advantages and pitfalls of load balancing and scheduling.

foreach, future.apply and Load Balancing

Now you might ask, can I reproduce my results if the application uses random numbers? Can I generate the same results regardless of if the code runs sequentially or in parallel? This chapter will answer these questions. You will learn about a random number generator well suited to a parallel environment and how the various packages make use of it.

Random Numbers and Reproducibility

Words (Jane Austen's 6 books)

US migration

SOTU (2016)

With an increasing amount of data and more complex algorithms available to scientists and practitioners today, parallel processing is almost always a must, and in fact, is expected in packages implementing time-consuming methods. This course introduces you to concepts and tools available in R for parallel computing and provides solutions to a few important non-trivial issues in parallel processing like reproducibility, generating random numbers and load balancing.

This course covers in detail the tools available in R for parallel computing.

Learning Experience

Business Features

Academics

Big Data with R

Create Your Free Account

Loved by learners at thousands of companies

Instructors

Loved by learners at thousands of companies

Instructors

Join over .css-w0wpus-TrackShowPage{color:#03ef62;}7,940,000 learners and start Big Data with R today!

Create Your Free Account

Join over 7,940,000 learners and start Big Data with R today!