Skip to main content
HomeAbout RLearn R

Web Scraping with R and PhantomJS

Short tutorial on scraping Javascript generated data with R using PhantomJS.
Mar 2015  · 5 min read

When you need to do web scraping, you would normally make use of Hadley Wickham’s rvest package. This package provides an easy to use, out of the box solution to fetch the html code that generates a webpage. However, when the website or webpage makes use of JavaScript to display the data you're interested in, the rvest package misses the required functionality. One solution is to make use of PhantomJS.

(Want to practice importing more data into R? Try this tutorial.)

Load the Necessary Packages

As a first step, you'll have to load all packages that we need for this analysis into the workspace (if you haven't installed these packages on your local system yet, use install.packages() to make them available):

library(rvest)
library(stringr)
library(plyr)
library(dplyr)
library(ggvis)
library(knitr)
options(digits = 4)

Scraping Javascript Generated Data with R

The next step is the collection of the TechStars data using PhantomJS. Check out the following basic .js file:

// scrape_techstars.js

var webPage = require('webpage');
var page = webPage.create();

var fs = require('fs');
var path = 'techstars.html'

page.open('http://www.techstars.com/companies/stats/', function (status) {
  var content = page.content;
  fs.write(path,content,'w')
  phantom.exit();
});

The script basically renders the HTML page after the underlying javascript code has done its work, allowing you to fetch the HTML page, with all the tables in there. To stay in R for the rest of this analysis, we suggest you use the system() function to invoke PhantomJS (you'll have to download and install PhantomJS and put it in your working directory):

# Let phantomJS scrape techstars, output is written to techstars.html
system("./phantomjs scrape_techstars.js")

After this small detour, you finally have an HTML file, techstars.html, on our local system, that can be scrape with rvest. An inspection of the Techstars webpage reveals that the tables we're interested in are located in divs with the css class batch:

batches <- html("techstars.html") %>%
  html_nodes(".batch")

class(batches)
[1] "XMLNodeSet"

You now have a list of XMLNodeSet objects: each object contains the data for a single TechStars batch. In there, we can find information concerning the batch location, the year, the season, but also about the companies, their current headquarters, their current status and the amount of funding they raised in total. We will not go into detail on the data collection and cleaning steps below; you can execute the code yourself and inspect what they accomplish. You'll see that some custom cleaning is going on to make sure that each bit of information is nicely formatted:

batch_titles <- batches %>%
  html_nodes(".batch_class") %>%
  html_text()

batch_season <- str_extract(batch_titles, "(Fall|Spring|Winter|Summer)")
batch_year <- str_extract(batch_titles, "([[:digit:]]{4})")
# location info is everything in the batch title that is not year info or season info
batch_location <- sub("\\s+$", "",
                      sub("([[:digit:]]{4})", "",
                          sub("(Fall|Spring|Winter|Summer)","",batch_titles)))

# create data frame with batch info.
batch_info <- data.frame(location = batch_location,
                         year = batch_year,
                         season = batch_season)

breakdown <- lapply(batches, function(x) {
  company_info <- x %>% html_nodes(".parent")
  companies_single_batch <- lapply(company_info, function(y){
    as.list(gsub("\\[\\+\\]\\[\\-\\]\\s", "", y %>%
       html_nodes("td") %>%
       html_text()))
  })
  df <- data.frame(matrix(unlist(companies_single_batch),
                   nrow=length(companies_single_batch),
                   byrow=T,
                   dimnames = list(NULL, c("company","funding","status","hq"))))
  return(df)
})

# Add batch info to breakdown
batch_info_extended <- batch_info[rep(seq_len(nrow(batch_info)),
                                  sapply(breakdown, nrow)),]
breakdown_merged <- rbind.fill(breakdown)

# Merge all information
techstars <- tbl_df(cbind(breakdown_merged, batch_info_extended)) %>%
  mutate(funding = as.numeric(gsub(",","",gsub("\\$","",funding))))

With a combination of core R, rvest, plyr and dplyr functions, we now we have the techstars data frame; a data set of all TechStars company, with all publicly available information that is nicely formatted:

techstars
## Source: local data frame [535 x 7]
##
##          company funding   status                hq location year season
## 1    Accountable  110000   Active    Fort Worth, TX   Austin 2013   Fall
## 2          Atlas 1180000   Active        Austin, TX   Austin 2013   Fall
## 3        Embrace  110000   Failed        Austin, TX   Austin 2013   Fall
## 4  Filament Labs 1490000   Active        Austin, TX   Austin 2013   Fall
## 5        Fosbury  300000   Active        Austin, TX   Austin 2013   Fall
## 6          Gone!  840000   Active San Francisco, CA   Austin 2013   Fall
## 7     MarketVibe  110000 Acquired        Austin, TX   Austin 2013   Fall
## 8           Plum 1630000   Active        Austin, TX   Austin 2013   Fall
## 9  ProtoExchange  110000   Active        Austin, TX   Austin 2013   Fall
## 10       Testlio 1020000   Active        Austin, TX   Austin 2013   Fall
## ..           ...     ...      ...               ...      ...  ...    ...
names(techstars)
## [1] "company"  "funding"  "status"   "hq"       "location" "year"
## [7] "season"

Conclusion

Using the above combination of tools and code, we managed to scrape data from a website that uses a JavaScript script to generate its data. As one can see, this is a very structured process, that can be easily done once the initial code is available. Want to learn R, and looking for data science and R tutorials and courses? Check out DataCamp

Learn more about R

Introduction to R

Beginner
4 hr
2.5M
Master the basics of data analysis in R, including vectors, lists, and data frames, and practice R with real data sets.
See DetailsRight Arrow
Start Course
See MoreRight Arrow
Related

Building Your Data Science Portfolio with DataCamp Workspace (Part 1)

Learn how to build a comprehensive data science portfolio by exploring examples different examples, mastering tips to make your work stand out, and utilizing the DataCamp Workspace effectively to showcase your results.
Justin Saddlemyer's photo

Justin Saddlemyer

9 min

Google Bard for Data Science Projects

Learn how to leverage Google Bard for project planning, data preprocessing, exploratory data analysis, feature engineering, model selection, hyperparameter tuning, model validation, and building and deployment of a web application.
Abid Ali Awan's photo

Abid Ali Awan

13 min

Building a Safer Internet with Data Science

Learn the key drivers of a data strategy that helps ensure online safety and consumer protection with Richard Davis, the Chief Data Officer at Ofcom, the UK’s government-approved regulatory and competition authority. 
Adel Nehme's photo

Adel Nehme

43 min

How Data Scientists Can Thrive in the FMCG Industry

Find out how data science drives strategy in the FMCG industry.
Adel Nehme's photo

Adel Nehme

42 min

Conda Cheat Sheet

In this cheat sheet, learn all about the basics of working with Conda. From managing and installing packages, to working with channels & environments, learn the fundamentals of the conda package management tool suite.
Richie Cotton's photo

Richie Cotton

Git Rebase Tutorial for Beginners

Discover what Git Rebase is and how to use it in your data science workflows.
Javier Canales Luna's photo

Javier Canales Luna

8 min

See MoreSee More