Analyzing US flight data on Amazon S3 with sparklyr and Apache Spark 2.0

Categories: CDH Data Science Hadoop Spark Use Case

We posted several blog posts about sparklyr (introduction, automation), which enables you to analyze big data leveraging Apache Spark seamlessly with R. sparklyr, developed by RStudio, is an R interface to Spark that allows users to use Spark as the backend for dplyr, which is the popular data manipulation package for R.

If you are interested in sparklyr, you can learn how to use it with the official document, or you also can try it on a Spark cluster with Cloudera Director. In this blog post, we are going to build a Spark cluster on AWS with Cloudera Director. If you will try sparklyr, the official cheat sheet is very helpful.

In this post, we will show you the automated launch of sparklyr with Cloudera Director client and visualize and build a predictive model of US air flight.

Launch RStudio Server and sparklyr with Spark cluster by Cloudera Director

You can refer to the previous post about automated deployment with Cloudera Director, but this time we will use this configuration file for Cloudera Director client in my repository. This cluster.conf requires Cloudera Director 2.3 or higher and it assumes you will use the ap-northeast-1 region on AWS. You should change several settings such as the security group and subnet ID.

You can launch a cluster with Cloudera Director as follows:

If you have your own Cloudera Director server, you can deploy as follows:

If you don’t have an environment with a Cloudera Director client, you can use a Docker-based tool.

Analyzing US flight data with sparklyr

In this post, we will show you a visualization and build a predictive model of US flights with sparklyr. Flight visualization code is based on this article:

This post assumes you already have the following tables:

You should make these tables available through Apache Hive or Apache Impala (incubating) with Hue.

After installation of  sparklyr and instantiation of the Spark cluster with Cloudera Director configuration file, you can access the RStudio server on <sparklyr-gateway-hostname>:8787 with your browser and log in with rsuser/cloudera.

Connect to Spark with sparklyr

Let’s connect to your Spark cluster with sparklyr. In this post, we installed Spark 2.0 additionally. Before running the following code, you should install additional R packages as install.packages(c("ggplot2", "maps", "geosphere", "dplyr")).

Read the table from S3 and plot with ggplot

Summarize flight number of airlines_bi_pq table by year.



sparklyr’s table is evaluated lazily, so you should use collect to convert into a data.frame.

Plot summarized data with ggplot:

us flights

We found the decreasing of flight number in 2002. But why?

See flight number between 2001 and 2003

Next, to dig into the data in 2002, let’s plot the number of flights between 2001 and 2003.

US flights 2

It appears that the number of flights after Sept. 2001 significantly decreased. We can understand it is the effect of 9/11. In this way, sparklyr makes exploratory data analysis easier for large-scale data, so we can obtain new insight quickly.

Summarize flight data by year, carrier, origin and dest

Next, we will summarize the data by carrier, origin and destination.

Now we extract AA’s flight in 2007.


Plotting flights into map.

Let’s plot the flight number of AA in 2007 on a map. You can change the condition of a filter to plot other airlines.

US flights 3

Build a predictive model for delay with linear regression

We will build a predictive model with Spark MLlib. We use linear regression from MLlib.

First, we will prepare training data. In order to handle categorical data, you should use ft_string_indexer for converting to label indices.


Now, we can see the trained linear regression model and its coefficients.


Using sparklyr enables you to analyze big data on Amazon S3 with R smoothly. You can build a Spark cluster easily with Cloudera Director. sparklyr makes Spark as a backend database of dplyr. You can create tidy data from huge messy data, plot complex maps from this big data the same way as small data, and build a predictive model from big data with MLlib. I believe sparklyr helps all R users perform exploratory data analysis faster and easier on large-scale data. Let’s try!

You can see the Rmarkdown of this analysis on RPubs. With RStudio, you can share Rmarkdown easily on RPubs.
Learn more about sparklyr and Cloudera in this on-demand video.