https://www.datacamp.com/community/news/difference-between-data-science-and-big-data-5tm3qkq80bj

Difference Between Data Science and Big Data is really complicated. Data Science is a field that comprises of everything that related to data cleaning, preparation, and analysis. Big Data refers to humongous volumes of data that cannot be processed effectively with the traditional applications that 

Difference between Data Science and Big Data

https://www.datacamp.com/community/news/how-to-become-a-data-scientist-in-one-week-sr4leab48c

How to Become a Data Scientist in One Week?

https://www.datacamp.com/community/news/from-local-machine-to-dask-cluster-with-terraform-d9klqtbw6e6

A comprehensive  tutorial on how to start a dask cluster

From Local Machine to Dask Cluster with Terraform

<h2 id="the-state-of-data-driven-digital-transformation">The state of data-driven digital transformation</h2>
As organizations undergo accelerated digital transformations amid the COVID-19 pandemic, the case for leveraging data, machine learning and analytics has never been more important.
The 2020 McKinsey Global Survey on the <a href="https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/global-survey-the-state-of-ai-in-2020">State of Artificial Intelligence</a> reported that 50% of businesses have adopted artificial intelligence in at least one business function, and 22% of organizations attribute at least 5% of earnings before interest and tax to artificial intelligence initiatives.
However, in order to unlock the value of data and artificial intelligence at scale, organizations must democratize their data and unlock organization-wide data fluency. One of the ways to do so is by providing access to trusted and actionable data across teams and skill levels so that everyone can make data-informed business decisions.
<h2 id="data-infrastructure-at-the-core-of-data-democratization">Data infrastructure at the core of data democratization</h2>
Before achieving data fluency, organizations need to get the foundations right. This starts with understanding how data flows through an organization and then building toward an organized and unified data infrastructure.
In a <a href="https://www.datacamp.com/resources/webinars/data-trends-2021">webinar on 2021 data trends and predictions</a>, Ramnath Vaidyanathan, Vice President of Product Research at DataCamp, described how data typically flows through an organization. This flow requires data engineering tools to take in raw data, ingest and transform them, store them in a centralized location, and produce data insights in the form of dashboards and reports, machine learning outputs, and more.
Scaling access to data within the organization depends heavily on the presence of a solid data infrastructure architecture and the use of the latest data infrastructure tooling, which enables organizations to transform raw data into accessible and valuable data insights.
<center><a href="https://www.datacamp.com/resources/webinars/data-trends-2021"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906487/dataflowsthruorg_xgcspr.png" /></a> 
<h2 id="the-state-of-data-infrastructure-tooling">The state of data infrastructure tooling</h2>
The importance of data infrastructure and tooling is evidenced by the rapid growth and evolution of the data infrastructure market. Many solutions are meant to streamline working with a particular element of an organization’s data infrastructure, which has resulted in an increasingly fragmented landscape of tooling that can be challenging to navigate.
The following visual created by Dataiku aptly illustrates how the data infrastructure market has been segmented into numerous granular categories, with an expanding list of competing solutions present within each segment.
<center><a href="https://blog.dataiku.com/technoslavia-the-fragmented-world-of-data-infrastructure-in-2020"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906493/technoslavia_gxmlfq.png" /></a>
<h5 id="source-dataiku-https-blog-dataiku-com-technoslavia-the-fragmented-world-of-data-infrastructure-in-2020-">Source: <a href="https://blog.dataiku.com/technoslavia-the-fragmented-world-of-data-infrastructure-in-2020">Dataiku</a></h5>
<h2 id="the-future-of-data-infrastructure-tooling">The future of data infrastructure tooling</h2>
Late last year, Andreessen Horowitz sought to provide some clarity in this confusing data infrastructure landscape by developing a unified architecture framework. With the growing need to work with large volumes of data and the plethora of available solutions, this framework offers a practical reference when modernizing, mapping, and building an organization’s data stack.
<center><a href="https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906493/andreeson_bjpxzl.png" /></a>
<h5 id="source-andreessen-horowitz-https-a16z-com-2020-10-15-the-emerging-architectures-for-modern-data-infrastructure-">Source: <a href="https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/">Andreessen Horowitz</a></h5>
As this space and trend evolves, we will see the consolidation and standardization of various tools in the data infrastructure stack. This alongside cloud adoption, data discovery tools, and centralized data governance platforms will allow organizations to provide scalable access to data across teams and functions.
So what does an ideal future data infrastructure deliver? According to <a href="https://www.datacamp.com/resources/webinars/operationalizing-data-within-large-organizations">Maksim Pecherskiy</a>, Data Engineer at the World Bank and former CDO of the city of San Diego, data must be collected, discoverable, reliable, understood, compliant, and actionable. Ultimately, the goal is to make data usable and trustworthy, which in turn drives organization-wide data fluency.
<center><a href="https://www.datacamp.com/resources/webinars/data-trends-2021"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906507/usedandtrusted_wide_ypurb3.png" /></a>
<h2 id="mature-data-infrastructure-includes-cloud-adoption-data-discovery-tools-and-centralized-data-governance-platforms">Mature data infrastructure includes cloud adoption, data discovery tools, and centralized data governance platforms</h2>
As data infrastructure tooling matures around data democratization in 2021 and beyond, so will the adoption and growth of the following technologies:
<h3 id="1-cloud-adoption">1. Cloud Adoption</h3>
The use of cloud technologies is <a href="https://www.oreilly.com/radar/cloud-adoption-in-2020/">becoming the norm</a>, especially in the wake of the accelerated data-driven digital transformation we’ve seen in 2020.
Besides improving data storage and access, cloud solutions also offer numerous user-friendly products for collecting, ingesting, transforming, storing, and creating insights from data.
There are numerous providers in the cloud platform space, and one of the key players is <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), which has <a href="https://aws.amazon.com/products/">more than 100 products</a> that enable enterprises to make the most out of their data at high speed, at scale, and at low cost.
<h3 id="2-data-discovery-tools">2. Data Discovery Tools</h3>
Scaling data infrastructure will be crucial as data tables, reports, and models are added to organizational data landscapes. Stakeholders must be readily able to locate reliable, understood, compliant, and actionable data and insights to do their best work.
To tackle this challenge, <a href="https://datacamp.com/community/data-discovery-tools">data discovery tools</a> (also referred to as metadata management tools or data catalogs) can act as versatile search engines for these troves of data within the organization. Metadata essentially refers to a set of data describing the particular data, and is useful for indexing the large variety of data available.
Lyft open-sourced their own data discovery tool <a href="https://www.amundsen.io/">Amundsen</a>, to offer metadata engine solutions that aim to help organizations discover, understand, share, and trust the data they use for their analyses. This capability is paramount for any organization looking to scale its data competencies.
<h3 id="3-centralized-data-governance-platforms">3. Centralized Data Governance Platforms</h3>
As mentioned earlier, creating trust in data is essential for driving data democratization. This is where data governance plays a major role. With the ever-growing volume and variety of data, centralized oversight of the data quality, consistency, and relevance is crucial.
Centralized data governance platform services offered by companies like <a href="https://www.collibra.com/">Collibra</a> and <a href="https://www.alation.com/solutions/data-governance/">Alation</a> help organizations effectively manage the availability, consistency, and integrity of their data. This then allows all stakeholders to work from a reliable single source of truth.
A maturing data infrastructure is only one of the data trends we foresee for 2021 and beyond. For more, download our white paper on <a href="https://www.datacamp.com/resources/whitepapers/data-trends-and-predictions-2021">Data Trends and Predictions 2021: The Year of Data Fluency</a> or watch our <a href="https://www.datacamp.com/resources/webinars/data-trends-2021">webinar</a>.
<center><a href="https://www.datacamp.com/groups/business"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603718763/banner_style_1_cruuf7.png" /></a>

https://www.datacamp.com/community/blog/data-infrastructure-tools

Unlocking the value of data in an organization starts with having the right data infrastructure and tooling foundations. Here’s a look at the current state and future trends of data infrastructure.

What Mature Data Infrastructure Looks Like | DataCamp

What Mature Data Infrastructure Looks Like

The most successful organizations today rely heavily on data to make decisions. Many of these data-fluent organizations generate millions of rows of data every day that are used and analyzed by a wide range of teams. One of the key challenges to equipping teams with the ability to access and work with data is ensuring that data is <a href="https://www.datacamp.com/resources/webinars/operationalizing-data-within-large-organizations">collected, trusted, reliable, actionable, and discoverable</a>.
To navigate these complex data landscapes, many organizations have developed and open-sourced in-house technologies, called data discovery tools.
In a <a href="https://www.datacamp.com/resources/webinars/the-path-to-data-fluency">recent webinar</a>, Ramnath Vaidyanathan, VP of Product Research at DataCamp describes how organizations can go from data reactive, where no one has the skills or access to work with data, to data fluent, where everyone has access and skills to work with data and make data-driven decisions. Organizations can increase their data maturity through investments in the <a href="https://www.datacamp.com/community/blog/iptop">IPTOP framework</a>: infrastructure, people, tools, organization, and processes. As organizations scale access to data through investments in data infrastructure, they will need to provide easy data discoverability with data discovery tools.
<blockquote>
Data is not useful if people cannot access it, if they don’t understand the context of the data collected, and if they cannot find what they need. This is why data discovery tools are really critical.
Ramnath Vaidyanathan, VP of Product Research at DataCamp
</blockquote>
<center><a href="https://www.datacamp.com/webinars/the-path-to-data-fluency"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615904291/Screen_Shot_2021-03-16_at_10.18.07_AM_mp6zpm.png" /></a> 
<h2 id="how-data-fluent-companies-use-data-discovery-tools">How data-fluent companies use data discovery tools</h2>
Many data-driven companies have implemented data discovery tools in their data infrastructure. Let’s examine the motivations and drivers behind notable data discovery tools like <a href="https://eng.uber.com/metadata-insights-databook/">Uber Databook</a> and <a href="https://www.amundsen.io/">Lyft Amundsen</a>.
<center><a href="https://www.datacamp.com/webinars/the-path-to-data-fluency"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615904178/datadiscoveryengines_pi2uj7_oqvtm7.png" /></a> 
<h3 id="data-discovery-challenges-to-consider">Data discovery challenges to consider</h3>
A key data discovery challenge is productivity and compliance. <a href="https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9">Lyft outlined</a> that it faced these challenges as the size of its data grew exponentially and was projected to continue to grow at a similar rate for at least the next 10 years. When developing Amundsen, they noted that about 25% of an analyst’s time was spent on data discovery. They also saw increasingly more stringent compliance demands in the countries they serviced.
<a href="https://eng.uber.com/metadata-insights-databook/">Uber expressed similar concerns</a> as it began to expand beyond its primary function as a ride-sharing app to services like Uber Eats, Uber Freight, and Jump Bikes. When developing Databook, Uber had over 15 million trips a day and 18,000 employees across different teams. Another challenge was that its data <a href="https://eng.uber.com/databook/">existed in many different forms</a> including <a href="https://hive.apache.org/">Hive</a>, <a href="https://prestodb.io/">Presto</a>, and <a href="https://www.vertica.com/">Vertica</a>. Analysts needed to be able to efficiently access and understand the various data sources in order to achieve data fluency.
<a href="https://engineering.fb.com/2020/10/09/data-infrastructure/nemo/">Facebook built its Nemo data discovery platform</a> to address the need for high-quality, trustworthy data as data size and complexity increased across roles and geographic locations. Due to the privacy concerns associated with the data they analyze, Facebook also faced significant compliance concerns that had to be addressed when offering tables to specific analysts.
John Bodley from Airbnb said <a href="https://www.youtube.com/watch?v=gayXC2FDSiA">in a talk at a 2017 conference</a>, “As Airbnb grows, so do the challenges around the volume, complexity, and obscurity of data.” This means that employees often found data to be “siloed, inaccessible, and lacking context.” As a result, employees were making duplicate tables and did not use data if they didn’t trust it. Airbnb developed <a href="https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770">Dataportal</a> to help make the data discovery process significantly more efficient by removing these barriers to productive data analysis.
These challenges are widespread, industry-agnostic, and apply to any organization looking to scale data-driven decision making. The need for easily discoverable data will become paramount as organizations look to become data fluent.
<h2 id="the-four-key-goals-of-data-discovery-tools">The four key goals of data discovery tools</h2>
Now that we understand data discovery challenges, let’s look at how various data discovery tools address them. <a href="https://eng.uber.com/databook/">Uber</a> breaks down the goals of a successful data discovery platform into four components:
<ol>
<li>Extensibility: The ability to easily add metadata, storage, and entities to tables</li>
<li>Accessibility: The ability to programmatically access all metadata</li>
<li>Scalability: The ability to support a lot of concurrent read requests</li>
<li>Power: The ability to support read and write requests across multiple data centers</li>
</ol>
Attaining these data discovery goals increase as organizations mature their data competencies. As the scale of their data increases, each of these components becomes increasingly more important.
<h2 id="lyft-and-airbnb-provide-detailed-metadata">Lyft and Airbnb provide detailed metadata</h2>
<a href="https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9">Lyft’s engineering team</a> argues that “metadata is the holy grail of future applications.” They break metadata down into two subcategories. The first is a describing set of data made up of application context (i.e., what humans need to know about the data to operate with it), behavior (i.e., who owns a particular dataset and its common usage patterns), and change (i.e., how the dataset is changing over time). The second component is the data being described which is any data stored in the organization in any format such as data stores, dashboards, datastreams, and more.
When developing <a href="https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770">Dataportal</a>, Airbnb’s data discovery platform, metadata was an integral part of the platform’s added value to the data pipeline. Before Dataportal was introduced, many employees did not trust the data they were using due to a lack of context and metadata. Airbnb argues that “the understanding of the entire data ecosystem, from the production of an event log to its consumption in visualization, provides more value than the sum of its parts.”
<center><a href="https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615904101/airbnbimage_irgwjr.png" /></a> 
<h5 id="source-airbnb-https-medium-com-airbnb-engineering-democratizing-data-at-airbnb-852d76c51770-">Source: <a href="https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770">Airbnb</a></h5>
<h2 id="ensure-high-quality-search-results-with-data-discovery">Ensure high-quality search results with data discovery</h2>
Once this context is provided, data discovery requires effective search. No matter how trustworthy and easy-to-understand each table is, data discovery platforms are only useful if analysts can find the data they need quickly. 
<a href="https://www.amundsen.io/">Lyft Amundsen’s</a> landing page allows users to search for tables using natural language queries and provides visibility into the most popular tables used in the organization for quick access. Furthermore, analysts have the option to leave feedback on the table to tune future search results.
Facebook’s data discovery solution, <a href="https://engineering.fb.com/2020/10/09/data-infrastructure/nemo/">Nemo</a>, allows analysts to filter on usage, privacy restrictions, and recency of data while also leveraging Facebook’s competency in natural language processing by allowing users to type questions into the search bar to get relevant tables. 
Uber Databook allows similar search filtering capabilities by allowing analysts to filter on dimensions such as name, owner, column, and nested columns. Uber leverages <a href="https://en.wikipedia.org/wiki/Elasticsearch">Elasticsearch</a> to deliver search results quickly and effectively.
<h2 id="arriving-at-data-democratization-through-data-discovery">Arriving at data democratization through data discovery</h2>
Scaling data fluency and data discoverability go hand in hand, as analysts need to have easy access to data to make data-informed decisions.
<center><a href="https://www.datacamp.com/webinars/the-path-to-data-fluency"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615904005/Screen_Shot_2021-03-16_at_10.13.21_AM_romy3u.png" /></a> 
Organizations with low data maturity should focus on building up data skills and culture and scaling data infrastructure. However, as organizations progress in their data maturity journey and expand data access, enabling extensible and scalable data discovery will be integral to becoming a data-driven organization. For an in-depth discussion on all the levers to grow organizational data maturity, watch our webinar on <a href="https://www.datacamp.com/webinars/the-path-to-data-fluency">the path to data fluency</a>.

https://www.datacamp.com/community/blog/data-discovery-tools

Understand how top-performing companies solve data discovery challenges to drive data fluency at scale.

How Data Discovery Tools Enable Data Democratization | DataCamp

How Data Discovery Tools Enable Data Democratization

One could argue that proper ETL pipelines are a vital organ of data science. Without clean and organized data, it becomes tough to produce quality insights that enhance business decisions.
Therefore, in this tutorial, we will explore what it entails to build a simple ETL pipeline to stream real-time Tweets directly into a SQLite database using R. This is a fairly common task involved in social network analysis for example.
The focus will be on covering the thought process of data collection and storage as well as how to operate the Twitter API using the <a href="https://rtweet.info/">rtweet</a> package in R.
To begin, let&#39;s start with having all the right tools. The first thing that you will need is set up your access to the Twitter API. Overall, you would need to follow these steps:
<ul>
<li>Create a Twitter account, if you don&#39;t have one.</li>
<li>Follow this <a href="https://developer.twitter.com/en/apply-for-access">link</a> and apply for a developer account (note that this process now requires an application that Twitter has to approve).</li>
<li>Create a new app on the following <a href="https://developer.twitter.com/en/apps">webpage</a>$^1$.</li>
<li>Fill in all the details about your app and create your access token$^1$.</li>
<li>Gather your consumer key, consumer secret, access token, and access token secret, which you need to connect to the API.</li>
</ul>
Once you are all set with the Twitter API, you need to get SQLite if you don&#39;t have it. For the full process on how to get SQLite on your computer, follow the <a href="https://www.datacamp.com/community/tutorials/beginners-guide-to-sqlite">Beginners Guide to SQLite tutorial</a> here at DataCamp. The choice of using SQLite for this tutorial was made due to its simplicity of operation.
<h2 id="step-1-creating-a-database-and-table-to-store-the-twitter-data">Step 1: Creating a Database and Table to Store the Twitter Data</h2>
With both the Twitter API access and SQLite installed, we can finally begin building a pipeline to store Tweets as we stream them over time. To start, we are going to create our new SQLite database using R as outlined below:
<pre><code class="lang-{r}"># Import necessary libraries and functions
library(RSQLite)
library(rtweet)
library(tm)
library(dplyr)
library(knitr)
library(wordcloud)
library(lubridate)
library(ggplot2)
source(&quot;transform_and_clean_tweets.R&quot;)
</code></pre>
<pre><code class="lang-{r}"># Create our SQLite database
conn &lt;- dbConnect(RSQLite::SQLite(), &quot;Tweet_DB.db&quot;)
</code></pre>
Afterward, we can create a table inside the database to hold the tweets. In our particular case, we are going to store the following variables:
<ul>
<li>Tweet_ID as an INTEGER primary key</li>
<li>User as TEXT</li>
<li>Tweet_Content as TEXT</li>
<li>Date_Created as INTEGER</li>
</ul>
You may wonder, why do I set the dates as integers? This is due to the fact that SQLite doesn&#39;t have a reserved data type for dates and times. Therefore, the dates will be stored as the number of seconds since 1970-01-01.
Let us now proceed to write the table:
<pre><code class="lang-{r}">dbExecute(conn, &quot;CREATE TABLE Tweet_Data(
 Tweet_ID INTEGER PRIMARY KEY,
 User TEXT,
 Tweet_Content TEXT,
 Date_Created INTEGER)&quot;)
</code></pre>
Once you have created the table, you can go to sqlite3.exe and check that is has indeed been created. You can see a screenshot of this below:
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1568658846/table_creation_m9ldw6.png" />
<h2 id="step-2-stream-tweets-about-your-favourite-topics-">Step 2: Stream Tweets About your Favourite Topics!</h2>
Believe it or not, you are done in terms of requirements and infrastructure needed to have a simple, functional Twitter streaming pipeline. What we need to do now is to stream Tweets using the API. It is worth noting that for the purposes of this tutorial, I will be using the standard free API. There are premium paid versions that may suit your Tweet streaming needs better if you conduct research, for instance.
Without further delay, let us start the process of setting up our Twitter listener. The first thing that you will need is to import the rtweet package and input your application&#39;s access tokens and secrets as described in the beginning:
<pre><code class="lang-{r}">token &lt;- create_token(app = &#39;Your_App_Name&#39;,
 consumer_key = &#39;Your_Consumer_Key&#39;,
 consumer_secret = &#39;Your_Consumer_Secret&#39;,
 access_token = &#39;Your_Access_Token&#39;,
 access_secret = &#39;Your_Access_Secret&#39;)
</code></pre>
With your token in place, the next thing is to decide what tweets do you want to stream (i.e., to listen to). The function stream_tweets in the rtweet package provides us with a variety of options to query the Twitter API. For example, you can stream tweets that contain one or more of a set of given hashtags or keywords (up to 400), a small random subset of all publicly available tweets, track the tweets of a group of user IDs or screen (user) names (up to 5000), or gather tweets by geographic location.
For this tutorial, I decided to go ahead and stream tweets containing hashtags related to data science (see list below). You may notice that the format in which I defined the hashtags to stream is a bit odd. However, that is the format required by the stream_tweets function when you want to listen to hashtags or keywords. This format varies if you intend to listen to a given set of users or based on coordinates. For more details, refer to the <a href="https://www.rdocumentation.org/packages/rtweet/versions/0.6.9/topics/stream_tweets">documentation</a>.
<pre><code class="lang-{r}">keys &lt;- &quot;#nlp,#machinelearning,#datascience,#chatbots,#naturallanguageprocessing,#deeplearning&quot;
</code></pre>
With the keywords defined, it is time to define the tweet streaming loop. There are several ways to do this, but this format has worked well for me in the past:
<pre><code class="lang-{r}"># Initialize the streaming hour tally
hour_counter &lt;- 0

# Initialize a while loop that stops when the number of hours you want to stream tweets for is exceeded
while(hour_counter &lt;= 12){
 # Set the stream time to 2 hours each iteration (7200 seconds)
 streamtime &lt;- 7200
 # Create the file name where the 2 hour stream will be stored. Note that the Twitter API outputs a .json file.
 filename &lt;- paste0(&quot;nlp_stream_&quot;,format(Sys.time(),&#39;%d_%m_%Y__%H_%M_%S&#39;),&quot;.json&quot;)
 # Stream Tweets containing the desired keys for the specified amount of time
 stream_tweets(q = keys, timeout = streamtime, file_name = filename)
 # Clean the streamed tweets and select the desired fields
 clean_stream &lt;- transform_and_clean_tweets(filename, remove_rts = TRUE)
 # Append the streamed tweets to the Tweet_Data table in the SQLite database
 dbWriteTable(conn, &quot;Tweet_Data&quot;, clean_stream, append = T)
 # Delete the .json file from this 2-hour stream
 file.remove(filename)
 # Add the hours to the tally
 hour_counter &lt;- hour_counter + 2
}
</code></pre>
In essence, that loop streams as many tweets as possible mentioning any of the hashtags in the key strings in intervals of 2 hours for a total time of 12 hours. Every 2 hours, the Twitter listener creates a .json file in your current working directory with the name specified in the variable filename.
Then, it passes this filename to the transform_and_clean_tweets function that removes retweets if desired, selects the columns we want to keep from all those given by the Twitter API, and normalizes the text contained in the Tweets.
Then, it appends the resulting dataframe to the Tweet_Data table that we created earlier inside our SQLite database. Finally, it adds 2 to the hour counter tally (since the streams last for 2 hours), and it removes the .json file created. This is done since all the data of interest is now in our database, and keeping the .json files can become a storage burden.
Let us know look closer to the transform_and_clean_tweets function in detail:
<pre><code class="lang-{r}">transform_and_clean_tweets &lt;- function(filename, remove_rts = TRUE){

 # Import the normalize_text function
 source(&quot;normalize_text.R&quot;)

 # Parse the .json file given by the Twitter API into an R data frame
 df &lt;- parse_stream(filename)
 # If remove_rst = TRUE, filter out all the retweets from the stream
 if(remove_rts == TRUE){
 df &lt;- filter(df,df$is_retweet == FALSE)
 }
 # Keep only the tweets that are in English
 df &lt;- filter(df, df$lang == &quot;en&quot;)
 # Select the features that you want to keep from the Twitter stream and rename them
 # so the names match those of the columns in the Tweet_Data table in our database
 small_df &lt;- df[,c(&quot;screen_name&quot;,&quot;text&quot;,&quot;created_at&quot;)]
 names(small_df) &lt;- c(&quot;User&quot;,&quot;Tweet_Content&quot;,&quot;Date_Created&quot;)
 # Finally normalize the tweet text
 small_df$Tweet_Content &lt;- sapply(small_df$Tweet_Content, normalize_text)
 # Return the processed data frame
 return(small_df)
}
</code></pre>
As noted, this function is meant to filter retweets if desired, retain the desired features, and normalize the tweet text. In essence, this can be considered the &quot;T&quot; part of the ETL acronym (the transformation). One key component of this function is the process of cleaning the tweet text.
Usually, text data requires a few preprocessing steps before it can be ready for any sort of analysis. In the case of tweets, these steps can include removal of URLs, stop words and mentions, turning the text to lower case, stemming words, etc. However, not all of these steps are necessary all of the time. Nevertheless, for the time being, I will display the normalize_text function that I used to preprocess these tweets below:
<pre><code class="lang-{r}">normalize_text &lt;- function(text){
 # Keep only ASCII characters
 text = iconv(text, &quot;latin1&quot;, &quot;ASCII&quot;, sub=&quot;&quot;)
 # Convert to lower case characters
 text = tolower(text)
 # Remove any HTML tags
 text = gsub(&quot;&lt;.*?&gt;&quot;, &quot; &quot;, text)
 # Remove URLs
 text = gsub(&quot;\\s?(f|ht)(tp)(s?)(://)([^\\.]*)[\\.|/](\\S*)&quot;, &quot;&quot;, text)
 # Keep letters and numbers only
 text = gsub(&quot;[^[:alnum:]]&quot;, &quot; &quot;, text)
 # Remove stop words
 text = removeWords(text,c(&quot;rt&quot;,&quot;gt&quot;,stopwords(&quot;en&quot;)))
 # Remove any extra white space
 text = stripWhitespace(text) 
 text = gsub(&quot;^\\s+|\\s+$&quot;, &quot;&quot;, text) 

 return(text)
}
</code></pre>
Depending on your use case, these steps may be sufficient. As mentioned earlier, you may choose to add more or different steps such as stemming or lemmatization or to keep only letters instead of letters and numbers. I encourage you to experiment and try out different combinations. This can be a great place to practice regex skills.
After all of these steps, the resulting state is a SQLite database populated with all the streamed tweets. You can validate that everything worked correctly by running a couple of simple queries, for instance:
<pre><code class="lang-{r}">data_test &lt;- dbGetQuery(conn, &quot;SELECT * FROM Tweet_Data LIMIT 20&quot;)
unique_rows &lt;- dbGetQuery(conn, &quot;SELECT COUNT() AS Total FROM Tweet_Data&quot;)
kable(data_test)
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570128184/kable_e3qnut.jpg" />
<pre><code class="lang-{r}">print(as.numeric(unique_rows))
</code></pre>
<pre><code>## [1] 1863
</code></pre><h2 id="step-3-analyze">Step 3 Analyze</h2>
When there is a certainty that the ETL process is working as intended, the final step is to gather some insights and analyze the data that you have collected. For example, for the tweets that we have collected, we can try a couple of different things; a wordcloud of terms mentioned in the tweet&#39;s content, and a timeline to visualize when in our 12-hour streaming period did we obtain the most amount of tweets. This list is obviously not a comprehensive overview of the whole array of research that can be conducted using tweets, which can range from sentiment analysis to psychographic studies and beyond.
With that said, lets jump straight to building a nice wordcloud:
<pre><code class="lang-{r}"># Gather all tweets from the database
all_tweets &lt;- dbGetQuery(conn, &quot;SELECT Tweet_ID, Tweet_Content FROM Tweet_Data&quot;)

# Create a term-document matrix and sort the words by frequency
dtm &lt;- TermDocumentMatrix(VCorpus(VectorSource(all_tweets$Tweet_Content)))
dtm_mat &lt;- as.matrix(dtm)
sorted &lt;- sort(rowSums(dtm_mat), decreasing = TRUE)
freq_df &lt;- data.frame(words = names(sorted), freq = sorted)

# Plot the wordcloud
set.seed(42)
wordcloud(words = freq_df$words, freq = freq_df$freq, min.freq = 10,
 max.words=50, random.order=FALSE, rot.per=0.15,
 colors=brewer.pal(8, &quot;RdYlGn&quot;))
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570046250/2_xjdbkd.png" />
Well, what a tremendous surprise! machinelearning and datascience are the most mentioned words in all our tweets—they just happen to be two of the hashtags that we were streaming for. Thus, this result was expected. The other words though are a bit more fun to look at. For example, bigdata and artificialintelligence were not hashtags in our keys, yet they appeared with high frequency, so one could argue that they are talked about often along with the other two. There are other words of interest that we picked up such as python or tensorflow that gives us a little bit more context into the content of the tweets we collected beyond just the hashtags.
Let us now turn to another simple analysis. In our 12-hour stream, at what time did we collect the most tweets? For that, we will gather our integer dates, turn them into a proper format, and then plot the quantity of tweets over time:
<pre><code class="lang-{r}"># Select the dates in which the tweets were created and convert them into UTC date-time format
all_tweets &lt;- dbGetQuery(conn, &quot;SELECT Tweet_ID, Date_Created FROM Tweet_Data&quot;)
all_tweets$Date_Created &lt;- as.POSIXct(all_tweets$Date_Created, origin = &quot;1970-01-01&quot;, tz = &quot;UTC&quot;)

# Group by the day and hour and count the number of tweets that occurred in each bucket
all_tweets_2 &lt;- all_tweets %&gt;%
 mutate(day = day(Date_Created),
 month = month(Date_Created, label = TRUE),
 hour = hour(Date_Created)) %&gt;%
 mutate(day_hour = paste(month,&quot;-&quot;,day,&quot;-&quot;,hour, sep = &quot;&quot;)) %&gt;%
 group_by(day_hour) %&gt;%
 tally()

# Simple line ggplot
ggplot(all_tweets_2, aes(x = day_hour, y = n)) +
 geom_line(aes(group = 1)) +
 theme_minimal() +
 ggtitle(&quot;Tweet Freqeuncy During the 12-h Streming Period&quot;)+
 ylab(&quot;Tweet Count&quot;)+
 xlab(&quot;Month-Day-Hour&quot;)
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570046251/3_fgkhum.png" />
Sweet! Now we can see that the largest amount of unique tweets that we obtain came from the 2nd of September between 8:00 PM UTC and 8:59PM UTC (This appears in military time as 20).
<h2 id="conclusion">Conclusion</h2>
Congratulations! Now you know how to build a simple ETL pipeline in R. The two analyses we conducted represent very basic analyses conducted using Twitter data. However, as mentioned previously, there are lots of things to do as long as you build a robust pipeline to bring in the data. This aspect was the main focus of this tutorial.
With that said, this tutorial only showed a very small scale case study to walk over the process of building ETL pipelines for Twitter data. Building robust and scalable ETL pipelines for a whole enterprise is a complicated endeavor that requires extensive computing resources and knowledge, especially when big data is involved.
I encourage you to do further research and try to build your own small scale pipelines, which could involve building one in Python$^1$. Maybe, even go ahead and try some big data projects. For example, DataCamp already has some courses like this <a href="https://www.datacamp.com/courses/big-data-fundamentals-via-pyspark">Big Data Fundamentals via PySpark</a> course where they go over big data using tools such as PySpark that can further your knowledge of this field.
If you would like to learn about data engineering, take DataCamp&#39;s <a href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a> course.
<h2 id="references">References</h2>
<ol>
<li>Foley, D. (2019, May 11). Streaming Twitter Data into a MySQL Database. Retrieved from <a href="https://towardsdatascience.com/streaming-twitter-data-into-a-mysql-database-d62a02b050d6">https://towardsdatascience.com/streaming-twitter-data-into-a-mysql-database-d62a02b050d6</a></li>
</ol>

https://www.datacamp.com/community/tutorials/building-a-tweet-etl-pipeline-using-r

Learn how to transform and load (ETL)  a data pipeline from scratch using R and SQLite to gather tweets in real-time and store them for future analyses.

community

datacamp

data engineering

News (10)

Difference between Data Science and Big Data

From Local Machine to Dask Cluster with Terraform

Tutorials (1)

Building Data Pipelines using R

Cheat Sheets (0)

Open Courses (0)

Podcast (0)

Blog Posts (21)

What Mature Data Infrastructure Looks Like

Tech Thoughts (0)