https://www.datacamp.com/community/news/difference-between-data-science-and-big-data-5tm3qkq80bj

Difference Between Data Science and Big Data is really complicated. Data Science is a field that comprises of everything that related to data cleaning, preparation, and analysis. Big Data refers to humongous volumes of data that cannot be processed effectively with the traditional applications that 

Difference between Data Science and Big Data

https://www.datacamp.com/community/news/how-to-become-a-data-scientist-in-one-week-sr4leab48c

How to Become a Data Scientist in One Week?

https://www.datacamp.com/community/news/from-local-machine-to-dask-cluster-with-terraform-d9klqtbw6e6

A comprehensive  tutorial on how to start a dask cluster

From Local Machine to Dask Cluster with Terraform

<ol>
<li>Introduction</li>
<li>What are the responsibilities of a data engineer?</li>
 <ul>
<li>ETL Pipelines</li>
<li>Data Warehouses</li><br>
</ul>
<li>Data engineering vs data science</li>
<li>Why has data engineering increased in popularity?</li>
<li>What are the skills of a data engineer?</li>
<li>How to learn data engineering</li>
</ol>
<h2 id="introduction">Introduction</h2>
<p>Data is everywhere. In recent years, with the growing number of connected devices and the expansion of their use, the world has been producing a vast amount of data that cannot be handled with traditional methods anymore.</p>
<p>According to Statista, as of 2021 there are <a target="_blank" href="https://www.statista.com/statistics/1101442/iot-number-of-connected-devices-worldwide/">23.8 billion devices connected to the internet</a> and 58% of those are IoT devices (smart home devices, connected self-driving cars, industrial devices, etc.) and the remaining 42% are non-IoT devices (laptops, smartphones, etc.). This means that nowadays data can come from various sources. Some of these are:</p>
<ol>
<li><strong>User activities:</strong> Any user action produces data and even the data that reflects where and how often we tap on the mobile applications we use is valuable. Every piece of data that contains this kind of information needs to be kept and distributed to the applications’ data centers.</li>
<li><strong>“Internet of Things” devices:</strong> These devices produce a huge amount of information, such as sensor data generated in many different locations. All this data needs to be retrieved and sent to a data pool for further analysis.</li>
<li><strong>Program logs:</strong> Today’s computer applications consist of many components and all those components produce program logs. These logs are sent to a data engineering pipeline for further analysis.</li>
</ol>
<p>All the different structured data that the world produces needs to be delivered to other components in the software structure, or visualized and interpreted by business analysts and data scientists. However, this delivery is not a simple task. There are some crucial requirements on how to deliver the data:</p>
<ol>
<li>Data needs to be delivered in a standardized structure</li>
<li>The data should not present any conflicts</li>
<li>Data duplication may need to be removed</li>
<li>The data needs to be reliable</li>
<li>The data delivery must be done with minimum delay</li>
</ol>
<p>Therefore, the delivery of this information is a complex process and, if it is not carried out properly, it can be very problematic, especially for medium and large-scale companies. This is where data engineers come into play.</p>
<p>In this article you will learn the basics of data engineering, which are:</p>
<ol>
<li>What data engineering is and why to ensure a robust data pipeline</li>
<li>The difference between data science and data engineering</li>
<li>The reason behind the increasing popularity of data engineering</li>
<li>The skills of a good data engineer</li>
<li>How DataCamp can help you become a data engineer</li>
</ol>
<h2 id="what-are-the-responsibilities-of-a-data-engineer-">What are the responsibilities of a data engineer?</h2>
<p>To put it simply, data engineers:</p>
<ol>
<li>Aim to make data easily accessible and available for data scientists and business intelligence engineers (or anyone working with data)</li>
<li>Optimize the big data system architecture of companies</li>
<li>Design and configure databases</li>
<li>Collaborate with teams of business intelligence engineers, data scientists, and data analysts</li>
<li>Explore and transform data
In order to perform all these tasks, data engineers first need to set up the ETL pipelines.</li>
</ol>
<h2 id="etl-pipelines">ETL pipelines</h2>
<p>ETL pipelines receive complex data at regular time intervals and process it with the aim of storing the data in a usable form. It is the responsibility of the data engineers to establish and maintain these pipelines.</p>
<p>There are 3 types of ETL pipelines:</p>
<ol>
<li><strong>Extract</strong>
A vast number of data created by data sources is collected in different formats.</li>
<li><strong>Transform</strong>
Since the data comes in different formats and due to its inconsistent sources, the data is processed to become as standardized as possible. This increases the discoverability and usability of the data for the next steps.</li>
<li><strong>Load</strong>
Once the data is in a discoverable and usable format, it is usually stored in data warehouses. This makes the data available at all times for further analysis.</li>
</ol>
<h2 id="data-warehouses">Data warehouses</h2>
<p>Data warehouses are where standardized data is stored. They are optimized for filtering and reading large volumes of data. In modern data warehouses, both structured and unstructured data can be stored, because it is also important to store the data that cannot be categorized, such as photos and videos.</p>
<h2 id="data-engineering-vs-data-science">Data engineering vs data science</h2>
<p>The differences between the roles of a data engineer and a data scientist are important. On the one hand, data scientists have an important role in companies because they contribute to data-driven decision making. Nevertheless, the success of data scientists is only as good as the data that they have available.</p>
<p>On the other hand, data engineers are the ones who design the systems that provide data to data scientists in an organized and consistent manner. Once the data engineers make the information available for them, data scientists then try to find patterns in the data to reach conclusions, using the format provided by the data engineers.</p>
<p>Data engineers and data scientists clearly work closely together. We can think of it as a football match. In football, it is the coach who develops the strategies that will create goal opportunities, and it is the coach who sets the players in certain positions. In this context the coach is the data engineer, so we can consider that the ball is the data and the player who scores is the data scientist. The rest of the team is the infrastructure.</p>
<h2 id="why-has-data-engineering-increased-in-popularity-">Why has data engineering increased in popularity?</h2>
<p>According to research conducted on the <a href="https://www.burning-glass.com/products/nova/">Burning Glass Nova</a> platform, job postings for &quot;data engineer&quot; increased by 88.3% in 2019. According to this research, demand for data engineers has been on the rise since 2016. Another interesting piece of information is that the number of job openings for data engineers is almost 5 times higher than the job openings for data scientists. The salary of a data engineer is also 20-30% higher than the salary of a data scientist.</p>
<p>As stated above, data scientists are the final link in the data processing chain. They need the data to be processed by a data engineer in order to perform their job. As the number of data sources and data types multiply, the need for a data engineer grows. For this reason, companies have recently started to invest more in the extraction and distribution of data.</p>
<p>In Figure 4.1, a pyramid chart displays the hierarchy of needs of data science. In this hierarchy, the tasks assigned to data engineers are at the base of the pyramid, which explains how crucial data engineers are for companies.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1640794050/image3_kje7by.png" />Figure 4.1: The data science hierarchy of needs.</center>

<center><a target="_blank" href="https://medium.com/hackernoon/the-ai-hierarchy-of-needs-18f111fcc007">Source: hackernoon</a></center>

<h2 id="what-are-the-skills-of-a-data-engineer-">What are the skills of a data engineer?</h2>
<p>The skills of data engineers are generally intertwined with the skills of software engineers and data scientists, as shown in Figure 5.1.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1640794050/image2_rqg5yb.png" />
Figure 5.1: Data roles and skill sets</center>
<center><a target="_blank" href="https://ryanswanstrom.com/2016/11/28/data-scientists-data-engineers-software-engineers-the-difference-according-to-linkedin/">Source: RyanSwanstrom</a></center>

<h2 id="is-the-role-of-a-data-engineer-closer-to-the-one-of-a-software-engineer-or-to-the-one-of-a-data-scientist-">Is the role of a data engineer closer to the one of a software engineer or to the one of a data scientist?</h2>
<p>The previous occupations of a group of data engineers are displayed in Figure 5.2.  According to this figure, over 40% of them have a software engineering background. Not only does this mean that data engineering and software engineering are significantly closer than we may have first thought, but also that software engineers nowadays are much more likely to switch to data engineering than any other field.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1640794050/image5_d4vwny.png" />
Figure 5.2: Data engineers by prior role (top 10).</center>
<center><a href="https://www.stitchdata.com/resources/the-state-of-data-engineering/">Source: stitchdata</a></center><br>

<p>At DataCamp we believe the following to be the skills of a good data engineer:</p>
<ol>
<li><strong>Software engineering background</strong>
Data engineers need to have a software engineering background, since they need to approach data engineering challenges with their knowledge of object-oriented programming, data structures and algorithms. Data engineers must be experienced in Python, Scala or Java.</li>
<li><strong>Tools</strong>
Data engineers need to be comfortable with data engineering tools like Airflow, Apache or Kafka.</li>
<li><strong>Database Technologies</strong>
A data engineer should have knowledge of multiple kinds of databases.</li>
<li><strong>Cloud Technologies</strong>
Data engineers should be comfortable using multiple cloud platforms like Amazon Web Services, Google Cloud Platform and Microsoft Azure. They need to have experience in cloud architecture and DevOps tools as well, since they are responsible for the automation of the data flow part of the system.</li>
</ol>
<h2 id="how-to-learn-data-engineering">How to learn data engineering</h2>
<p>If you want to improve in the data engineering field, you can start by completing the following recommended DataCamp courses. By completing these courses, you will likely become a more suitable candidate for internships or job positions in the field of data engineering.</p>
<ol>
<li><a target="_blank" href="https://www.datacamp.com/courses/intro-to-python-for-data-science">Introduction to Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/intermediate-python">Intermediate Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-relational-databases-in-sql">Introduction to Relational Databases in SQL</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-scala">Introduction to Scala</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-airflow-in-python">Introduction to Airflow in Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/aws-cloud-concepts">AWS Cloud Concepts</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/building-data-engineering-pipelines-in-python">Building Data Engineering Pipelines in Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/nosql-concepts">NoSQL Concepts</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/etl-in-python">ETL in Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/streaming-concepts">Streaming Concepts</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/streaming-data-with-aws-kinesis-and-lambda">Streaming Data with AWS Kinesis and Lambda</a></li>
</ol>


https://www.datacamp.com/community/blog/what-is-data-engineering

Learn what data engineering is, what is the difference between data science and data engineering, the scope in the field, and how to become one.

Learn what data engineering is, compare it with data science, and find out the skills required for the role and how to acquire them. 

What is Data Engineering?

<p>Data processing, analysis, and visualizations were done using R and its corresponding packages. Notebook and pertinent files used can be found <a target="_blank" href="https://app.datacamp.com/workspace/w/03312344-a7b0-47cf-b97d-06125fabcbd8/edit">here</a>.</p>
<p>According to the Merriam-Webster dictionary, data is factual information used as a basis for reasoning, discussion, or calculation. By this definition, data is basically any piece of collected information that can be used and further processed and analyzed to gain insights. It is commonly associated with computers as data is normally produced and stored in a computer, yet data has been around much longer than we might imagine.</p>
<h2 id="history-of-data">History of Data</h2>
<p>The earliest example of humans storing and analyzing data dates back to 18000 BCE, when it was discovered that prehistoric humans were using tally sticks, as a means of carrying out rudimentary calculations. These Paleolithic tribespeople marked notches on sticks and bones to keep track of their activities, such as trading and monitoring supplies. By 2400 BCE, the abacus, a device used in performing calculations, was invented in Babylon.</p>
<p>Throughout history, constant development of data collection, processing, and analysis was seen through numerous writings in stone tablets, clay, papyrus, wood, and paper scrolls. Eventually, as more forms of data were discovered, the need for processing, collecting, storing, and analyzing it also evolved.</p>
<p>As human society became more advanced, so did its requirement for processing data. In the 1800s, the census started taking place in the United States. The number of data points in the census grew exponentially to the point that the U.S. Census Bureau estimated that it would take years or even decades to collect and analyze all the data in the census. This was largely problematic since compiling and analyzing the current census data would only be completed when the next census was about to start or had already started. Fortunately, a young engineer and inventor named Herman Hollerith developed the Hollerith Tabulating Machine–an electromechanical tabulating machine which reduced the time needed to collect and analyze census data from years, to just mere months. Because of this, Hollerith is regarded as the father of modern automatic computation and would later become known for founding IBM.</p>
<p>Fast forward to the 1900s, and the invention of the computer. With the advent of a more powerful computing machine comes a more sophisticated requirement for data storage. Fritz Pfleumer, a German-Austrian engineer, invented a method of storing information magnetically on tape. Some principles of his invention are still being used today for digital data storage. In this era, the term “business intelligence” gained popularity as the need for emerging software, and systems for analyzing commercial and operational performance, quickly rose.</p>
<p>The revolution of data truly changed when Tim Berners-Lee created the World Wide Web, also known as the Internet, in 1989. This led to automated information sharing between people all over the world. This means that today more data is being shared, created, and stored, leading to new ways for collecting, using, and analyzing data.</p>
<h2 id="transition-to-big-data-">Transition to “Big Data”</h2>
<p>Due to the incredible amount of internet growth in the 1990s and the steady development of personal computers and computing devices in general, the number of devices online (and consequently the amount of data created) grew quickly.</p>
<p>While the idea of Big Data had been around even before the 1990s, it was only in 2005 when Roger Mougalas officially gave it a label. He described it as “a large set of data that is almost impossible to manage and process using traditional business intelligence tools.”</p>
<p>Big Data is a term used to describe the large volume of data–both structured and unstructured–that overwhelms organizations on a day-to-day basis. It encompasses the volume of information, the velocity or speed at which it is created and collected, and the variety or scope of the data points being covered.</p>
<p>Given the size and complexity of Big Data, the process of collecting, organizing, and analyzing it to discover patterns and other useful information has become part of helping a lot of organizations to make their business decisions. In turn this has given rise to data science–an interdisciplinary field which uses scientific methods, processes, algorithms, and systems from a large volume of data, to uncover patterns and enable business leaders to draw informed insights.</p>
<h2 id="what-is-data-science-">What is Data Science?</h2>
<p>According to IBM, data science is a multidisciplinary approach to extracting actionable insights from the large and ever-increasing volumes of data collected and created by today’s organizations. The field typically requires computer science and pure science skills as data scientists apply scientific methods in their approach, and use predictive analytics and artificial intelligence to extract insights from the data.</p>
<p>Nowadays, “data science” is often used by businesses and organizations as a catch-all term for the process of handling large amounts of data, whether it is preparing, cleaning, analyzing, or visualizing the data to reveal patterns. These are just some of the data science careers one can pursue:</p>
<ol>
<li><b>Data Scientist:</b><br>
<p>Data scientists need to be able to apply mathematics, statistics, and the scientific method; use multiple tools and techniques for cleaning and preparing data; perform predictive analytics and artificial intelligence; and explain how these results can be used to provide data-driven solutions to business problems. They require much more technical skills than data analysts. For starters, you may take data science-oriented courses like these on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/intro-to-python-for-data-science">Introduction to Python for Data Science</a> and <a target="_blank" href="https://www.datacamp.com/courses/free-introduction-to-r">Introduction to R</a>.</p>
</li>
<li><b>Data Analyst:</b><br>
<p>Data analysts collect, process, and perform statistical data analyses to arrive at meaningful conclusions for organizations. They transform and manipulate large data sets into usable forms, such as reports or presentations. They also aid in the decision-making process by looking into important patterns, and gleaning insights from the data which they can then effectively communicate to organizational leaders to aid in business decisions. These courses on DataCamp may help you kick-start your data analyst career: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-sql">Introduction to SQL</a> and <a target="_blank" href="https://www.datacamp.com/courses/exploratory-data-analysis-in-sql">Exploratory Data Analysis in SQL</a>.</p>
</li>
<li><b>Data Engineer:</b><br>
<p>Data engineers are responsible for preparation, processing, and managing gathered and stored data for analytical or operational uses. Like the traditional engineers, data engineers build and maintain data “pipelines” which connect data from one system to another, making information accessible for data scientists. Because of this, data engineers are required to understand several programming languages used in data science such as Python, R, and SQL. Topics related to data engineering like these are available on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a> and <a target="_blank" href="https://www.datacamp.com/courses/data-engineering-for-everyone">Data Engineering for Everyone</a>.</p>
</li>
<li><b>Data Architect:</b><br>
<p>Data architects mainly design and create blueprints for data management systems that data engineers then build. Analogous to the traditional architects, data architects are the “visionaries” as they are responsible for visualizing and designing an organization’s data management framework. In addition, data architects improve the performance of existing systems, making sure they are accessible to database administrators and analysts. You may take courses related to data architecture like these that are available on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/aws-cloud-concepts">AWS Cloud Concepts</a> and <a target="_blank" href="https://www.datacamp.com/courses/database-design">Database Design</a></p>
</li>
<li><b>Business Intelligence (BI) Developer:</b><br>
<p>BI developers are specialized engineers who use software tools to transform data into useful insights that help business decisions. They are responsible for simplifying technical information for everyone else in the company to easily understand. In short, they create and run reports containing data they find using business intelligence tools, and transform the information into more layman terms. Courses on using tools related to business intelligence such as Microsoft Power BI and Tableau are available on DataCamp such as these: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-power-bi">Introduction to Power BI</a> and <a target="_blank" href="https://www.datacamp.com/courses/analyzing-data-in-tableau">Analyzing Data in Tableau</a></p>
</li>
<li><b>Statistician:</b><br>
<p>Given that statistics is one of the main foundations of data science, many statisticians can easily transition into the data science field. Statisticians are mainly responsible for the collection and handling of data. They decide what data they need and how to collect it. Additionally, they design experiments, analyze and interpret the data, and report conclusions. You may check-out these introductory courses to statistics on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-statistical-modeling-in-r">Introduction to Statistical Modeling in R</a> and <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-statistics-in-spreadsheets">Introduction to Statistics in Spreadsheets</a>.</p>
</li>
<li><b>Machine Learning Engineer:</b><br>
<p>Machine learning engineers are another group of specialized engineers who are focused on researching, building, and designing artificial intelligence (AI) and machine learning (ML) systems to automate predictive models. Basically, they develop algorithms that use input data and leverage statistical models to predict an output, while continuously updating outputs as new data becomes available. DataCamp has numerous courses related to machine learning which you may find useful such as these: <a target="_blank" href="https://www.datacamp.com/courses/machine-learning-for-everyone">Machine Learning for Everyone</a> and <a target="_blank" href="https://www.datacamp.com/courses/machine-learning-with-caret-in-r">Machine Learning with caret in R.</a></p>
</li>
</ol>
<p>A quick job search online can give you an idea as to which of the data science career paths listed above is the most in demand. The graph below shows the job openings per career path according to <a target="_blank" href="https://www.indeed.com/jobs?q=data%20scientist&amp;vjk=ecfd94ad120e8423">Indeed USA</a> as of December 8, 2021.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image1_hntxle.png" /></center>

<p>Data architect appears to be the most in demand data science career path, due to their importance in creating data management systems which the other data science professionals then use. Next on the list is machine learning engineer, given the importance of utilizing artificial intelligence to predict outcomes in a lot of tech companies.</p>
<p>The least in demand are statisticians, mainly because a lot of the traditional statisticians are now becoming data scientists. This trend is expected to continue given how easy it is for statisticians to move from pure statistics to data science as they already possess the fundamental knowledge needed to become a fully-fledged data scientist.</p>
<h2 id="data-science-is-not-slowing-down">Data science is not slowing down</h2>
<p>The field of data science rose to immense popularity in the 2010s, especially when “Data Scientist” was touted by Harvard Business Review as the “sexiest job of the 21st century.” It became so popular that the title “Data Scientist” became a ubiquitous and trendy expression. While the term “sexiest job” does sound ridiculous at first, it mostly refers to how data scientists are not only in demand, but are also expensive to hire and difficult to retain.</p>
<p>While there is no question that data science is extremely popular nowadays, the better question is, will it remain this popular in the future? Based on employment projections, that does seem to be the case. Data from the 2020-2030 Employment Projections of the <a target="_blank" href="https://www.bls.gov/emp/tables/emp-by-detailed-occupation.htm">U.S. Bureau of Labor Statistics</a> shows that data science careers, which include statistics, data science, and other mathematical and science based occupations such as data engineering, will exhibit a very high growth rate in terms of percent change from 2020 to 2030. Statisticians ranked 14th overall, while data scientists and other mathematical science occupations ranked 31st overall out of 790 jobs included in the data.</p>
<p>While the shares of statisticians and data scientists from the total labor force are small compared to other occupations, those numbers are expected to increase in following years as the data science career path becomes more and more popular. The figure below shows how statisticians, data scientists, and other mathematical science occupations stack up against other jobs with high projected growth.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image3_bwn60b.png" /></center>


<h2 id="a-really-expensive-field">A really expensive field</h2>
<p>One major reason why data science is popular is because it is one of the highest paying jobs in the world. The figure below shows the range of salaries for 10 different occupations–including data scientists and data analysts–in New York City. The data was obtained from <a target="_blank" href="https://teleport.org/cities/new-york/salaries/">Teleport</a>, a website which aggregates data on living conditions (e.g., salaries) in different cities.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image2_qhdjq6.png" /></center>

<p>According to Teleport, data scientists had the 4th highest annual median salary in New York City at \$114,105, just behind C Level Executives and healthcare workers. In fact, in some other cities like Manila, data scientists rank as high as 2nd highest, just behind C Level Executives. This speaks volume as to how lucrative this job is. Data analyst jobs, whilst not as lucrative as data science positions, are still not to be slighted. At an annual median salary of $61,818, data analysts still earn around the average household income in New York City.</p>
<h1 id="ever-growing-in-the-job-market">Ever growing in the job market</h1>
<p>Another major reason why data science is popular is because of how organizations these days integrate the principles of data science in their day-to-day operations. The graph below shows the top 10 industries that involve data science in their work. The sample data of 8,000 data science oriented companies are extracted from <a target="_blank" href="https://www.diffbot.com/products/extract/testdrive/?token=210ced42a6e6d0cc4ac7f15885a94233.">Diffbot</a>, the world’s largest knowledge graph.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image4_spyov2.png" /></center>

<p>It comes to no surprise that Software Companies–including tech companies like Google, Apple, and Uber–came out on top. After all, the exceptional growth of big data was caused by the creation of the internet, which is closely tied to software and technology. Data science really is about processing a lot of information using various tools and technology.</p>
<p>Next on the list are financial services companies, which has been brought about by the rise of fintech companies. A portmanteau of “finance” and “technology,” fintech companies integrate technology and innovation in their services and offerings to improve their delivery to customers and disrupt traditional financial services. Since it involves handling a vast amount of data such as customer information, financial services companies saw the potential of data science tools to help streamline and optimize their processes and improve their services.</p>
<h1 id="in-conclusion-is-it-worth-pursuing-a-career-in-data-science-">In conclusion, is it worth pursuing a career in data science?</h1>
<p>With everything that has been explained above, the answer is a definite yes! As to which specific data science career path you should pursue, this depends primarily on your individual strengths and general interests. What is important is that any of the careers mentioned above are sure to be worthwhile.</p>
<p>A career path which is very lucrative and appears to show no signs of slowing down in terms of popularity, data science will continue to shape and influence how businesses and organizations operate for many years to come.</p>


https://www.datacamp.com/community/blog/blog-data-science-careers-to-pursue

Examining the incredibly popular field that is data science

Explore all the roles under the umbrella Data science.  And discover introductory courses for the same.

An Analysis of the Top 7 Trending Data Science Careers to Pursue 

<p>In our first <a target="_blank" href="https://www.datacamp.com/community/tutorials/Introduction-to-bokeh-tutorial">Data Visualization with Bokeh</a> tutorial, you learned how to build visualizations from scratch using glyphs.  Now, you can extend this concept and build plots from different data structures such as arrays and dataframes. In this tutorial, you will learn how to plot data with NumPy arrays, dataframes in Pandas, and ColumnDataSource using Bokeh. The ColumnDataSource is the essence of Bokeh, making it possible to share data over multiple plots and widgets.</p>
<h2 id="bokeh-plots-from-numpy-arrays">Bokeh Plots from NumPy Arrays</h2>
<h3 id="creating-line-plots-from-numpy-arrays">Creating line plots from NumPy arrays</h3>
<pre><code class="lang-python"># Import packages
import numpy as np
import random
from bokeh.io import output_file, show
from bokeh.plotting import figure

# Create an array
x_array = np.array([10,20,30,40,50,60])
y_array = np.array([50,60,70,80,90,100])

# Create a line plot
plot = figure()
plot.line(x_array, y_array)

# Output plot
show(plot)
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655558/image6_wxmfuh.png"></center>

<h3 id="creating-scatter-plots-from-numpy-arrays">Creating scatter plots from NumPy arrays</h3>
<pre><code class="lang-python"># Import packages
import numpy as np
import random
from bokeh.io import output_file, show
from bokeh.plotting import figure

# Create arrays
x_arr = np.array([10,11,12,13,14,15,16])
y_arr = np.array([17,18,19,20,21,22,23])

# Create plot
plot = figure(title = &quot;Scatter plot&quot;, x_axis_label = &quot;Label name of x axis&quot;, y_axis_label =&quot;Label name of y axis&quot;)
plot.circle(x_arr,y_arr, size = 30, alpha = 0.5)

# show plot
show(plot)
</code></pre><p>This yields the following graph:  </p>

<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655559/image7_osekpy.png"></center>

<h2 id="creating-plots-using-pandas-dataframes">Creating Plots Using Pandas Dataframes</h2>
<h3 id="plotting-lines-with-pandas-dataframes">Plotting lines with Pandas dataframes</h3>
<p>The data in this example is obtained from <a target="_blank" href="https://www.kaggle.com/camnugent/sandp500/data">Kaggle</a>. The dataset contains all the stocks contained in the S&amp;P 500 index for five years up to December 2018. The file contains seven columns:</p>
<ul>
<li>Date - day of the week date in format yy-mm-dd</li>
<li>Open - stock’s opening price</li>
<li>High - stock’s highest price in the day</li>
<li>Low - stock’s lowest price of the day</li>
<li>Close - stock’s closing price for the day</li>
<li>Volume - volume of stocks traded in the day</li>
<li>Name - stock ticker name</li>
</ul>
<p>We want to see the trend for Google’s stock over the five-year period. First, we load the data using Pandas and then check the first five rows of the Pandas dataframe.</p>
<pre><code class="lang-python"># Import packages
import pandas as pd
# Load data
df = pd.read_csv(&#39;all_stocks_5yr.csv&#39;)
df.head()
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655559/image8_jykblu.png"></center>

<pre><code class="lang-python"># Focus on Google&#39;s stock
df_goog = df[df[&#39;Name&#39;] == &#39;GOOG&#39;]
# Convert date column to a time series
df_goog[&#39;date&#39;] = pd.to_datetime(df_goog[&#39;date&#39;])

plot = figure(title= &quot;Pandas Line Plot&quot;, x_axis_type = &#39;datetime&#39;, x_axis_label = &#39;date&#39;, y_axis_label= &#39;Prices&#39;)
plot.line(x = df_goog[&#39;date&#39;], y = df_goog[&#39;high&#39;])

# Output plot
show(plot)
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655558/image3_kxcgou.png"></center>

</br><p>In the chart, it was necessary to convert the date column to datetime to render dates along the x-axis. Additionally, the line function (glyph in Bokeh) is used to generate a time series plot for Google&#39;s high prices.</p>
<h2 id="creating-a-scatter-plot-from-pandas-dataframe">Creating a scatter plot from Pandas dataframe</h2>
<p>In this section, we use the open-source S&amp;P 500 stock data available on <a target="_blank" href="https://www.kaggle.com/camnugent/sandp500/data">Kaggle</a>. In this example, we want to see the correlation between Google&#39;s stock prices and Apple&#39;s stock prices. You can learn more about correlation <a target="_blank" href="https://www.datacamp.com/community/tutorials/tutorial-datails-on-correlation">here</a>.</p>
<pre><code class="lang-python"># Import required packages
# Import packages
import pandas as pd
from bokeh.io import output_file, show
from bokeh.plotting import figure

# Load data
df = pd.read_csv(&#39;all_stocks_5yr.csv&#39;)

# Select Google and Apple stock
df_goog = df[df[&#39;Name&#39;] == &#39;GOOG&#39;]
df_apple = df[df[&#39;Name&#39;] == &#39;AAPL&#39;]

# Select only &#39;date&#39; and &#39;high&#39; columns for Google and Apple
df_goog = df_goog[[&#39;date&#39;,&#39;high&#39;]]
df_apple = df_apple[[&#39;date&#39;,&#39;high&#39;]]
# Rename &#39;high&#39; column to &#39;high_goog&#39; and &#39;high_apple&#39;
df_goog.rename(columns={&#39;high&#39;: &#39;high_goog&#39;}, inplace=True)
df_apple.rename(columns={&#39;high&#39;: &#39;high_apple&#39;}, inplace=True)

df_merged = pd.merge(df_goog, df_apple, how=&#39;inner&#39;, on=&#39;date&#39;)
# See the first five rows
df_merged.head(5)
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655557/image2_avd26p.png"></center>

<br><p>Merging ensures that the daily prices correspond to each other based on date. This mitigates the problem of correlating prices that are from different dates, as this will yield false statistical inference.</p>
<pre><code class="lang-python"># Create scatter plot
plot = figure(title= &quot;Pandas Scatter Plot&quot;, x_axis_label = &quot;Apple high prices&quot;, y_axis_label= &quot;Google high prices&quot;)
plot.circle(x = df_merged[&#39;high_apple&#39;], y = df_merged[&#39;high_goog&#39;],color = &#39;red&#39;, size = 10, alpha = 0.8)

# Output the plot
show(plot)
<p></code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655558/image5_oeqmjz.png"></center></p>


<p>The circles represent the link between Apple and Google&#39;s high prices. There is a general positive correlation between Google and Apple’s stock prices.</p>
<p><h2 id="creating-bokeh-plots-from-columndatasource">Creating Bokeh Plots from Columndatasource</h2></p>
<p>The purpose of ColumnDataSource is to allow you to create different plots using the same data. In short, it allows you to build a foundation of data for calling in multiple plots and analyses. This will save you time, as you won&#39;t have to load data multiple times in Jupyter Notebook.</p>
<p>The ColumnDataSource is akin to a python dictionary, where the value of the data in a column has a key string name for that very column.</p>
<h3 id="creating-line-plots-using-columndatasource">Creating line plots using ColumnDataSource</h3>
<br><pre><code class="lang-python"># Import packages
from bokeh.io import output_file, show
from bokeh.plotting import figure, ColumnDataSource

# Create the ColumnDataSource object
df_goog[&#39;date&#39;] = pd.to_datetime(df_goog[&#39;date&#39;])
data = ColumnDataSource(df_goog)

# Create time series plot
plot = figure(x_axis_type=&#39;datetime&#39;,
              title=&#39;GOOG high prices ts plot&#39;,
             x_axis_label = &quot;Date&quot;,
              y_axis_label = &quot;Prices in $&quot;)
plot.line(x=&#39;date&#39;, y = &#39;high_goog&#39;, source = data, color = &#39;green&#39;)
show(plot)
<br><p></code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655557/image1_zcjhnn.png"></center></p>


<p>The plot shows us how Google’s high prices evolved over a 5-year period with the line indicating each price point of the stock.</p>
<p>To create this plot, we passed the entire Google stock dataframe into the ColumnDataSource function and saved it as a variable named data.</p>
<p>Then, we used the source argument to pass this data into the plot function. This makes it possible to use the column names directly when creating the plot instead of repeatedly calling the dataframe.</p>
<p>Finally, we used the line function to generate the green line to represent the data points.</p>
<h3 id="creating-scatter-plots-using-columndatasource">Creating scatter plots using ColumnDataSource</h3>
<p>In this code, we passed the entire Google stock dataframe into the ColumnDataSource (as in the line plot example). The dataframe is saved into the ColumnDataSource as a variable named data. We first create the line plot by directly calling the column names. Finally, we used the diamond function to generate the red diamonds to represent the individual data points.</p>
<br><pre><code class="lang-python"># Import packages
from bokeh.io import output_file, show
from bokeh.plotting import figure, ColumnDataSource

# Create the ColumnDataSource object
df_goog[&#39;date&#39;] = pd.to_datetime(df_goog[&#39;date&#39;])
data = ColumnDataSource(df_goog)

# Create time series plot
plot = figure(x_axis_type=&#39;datetime&#39;,
              title=&#39;GOOG high prices ts plot&#39;,
             x_axis_label = &quot;Date&quot;,
              y_axis_label = &quot;Prices in $&quot;)
plot.line(x=&#39;date&#39;, y = &#39;high_goog&#39;, source = data, color = &#39;green&#39;)
plot.diamond(x=&#39;date&#39;, y = &#39;high_goog&#39;, source = data, color = &#39;red&#39;)
show(plot)
</code></pre><center><p><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1648655558/image4_xpfdxn.png"></center></p>

<p>The plot shows the evolution of Google&#39;s stock over a 5-year period with the red diamonds indicating each price point of the stock.</p>
<h2 id="conclusion">Conclusion</h2>
<p>This tutorial introduced the most common data structures that you are likely to encounter in Python and how you can use them to bring your data to life. While Bokeh is not your only option for Python visualization, it distinguishes itself from Matplotlib or Seaborn by its ease of use and interactive capabilities.  In our next tutorial, you will gain hands-on experience in using layouts for creating effective and interactive visualizations. If you’d like to do a deep dive into all the data visualization possibilities offered by Bokeh, check out DataCamp’s  <a target="_blank" href="https://www.datacamp.com/courses/interactive-data-visualization-with-bokeh">Interactive Data Visualization with Bokeh</a> course.</p>


https://www.datacamp.com/community/tutorials/tutorial-data-visualization-with-bokeh

Learn basic visualizations of various data structures using Bokeh.

Data Visualization with Bokeh Tutorial: Plotting Data Structures 

<p>One could argue that proper ETL pipelines are a vital organ of data science. Without clean and organized data, it becomes tough to produce quality insights that enhance business decisions.</p>
<p>Therefore, in this tutorial, we will explore what it entails to build a simple ETL pipeline to stream real-time Tweets directly into a SQLite database using R. This is a fairly common task involved in social network analysis for example.</p>
<p>The focus will be on covering the thought process of data collection and storage as well as how to operate the Twitter API using the <a href="https://rtweet.info/">rtweet</a> package in R.</p>
<p>To begin, let&#39;s start with having all the right tools. The first thing that you will need is set up your access to the Twitter API. Overall, you would need to follow these steps:</p>
<ul>
<li>Create a Twitter account, if you don&#39;t have one.</li>
<li>Follow this <a href="https://developer.twitter.com/en/apply-for-access">link</a> and apply for a developer account (note that this process now requires an application that Twitter has to approve).</li>
<li>Create a new app on the following <a href="https://developer.twitter.com/en/apps">webpage</a>$^1$.</li>
<li>Fill in all the details about your app and create your access token$^1$.</li>
<li>Gather your consumer key, consumer secret, access token, and access token secret, which you need to connect to the API.</li>
</ul>
<p>Once you are all set with the Twitter API, you need to get SQLite if you don&#39;t have it. For the full process on how to get SQLite on your computer, follow the <a href="https://www.datacamp.com/community/tutorials/beginners-guide-to-sqlite">Beginners Guide to SQLite tutorial</a> here at DataCamp. The choice of using SQLite for this tutorial was made due to its simplicity of operation.</p>
<h2 id="step-1-creating-a-database-and-table-to-store-the-twitter-data">Step 1: Creating a Database and Table to Store the Twitter Data</h2>
<p>With both the Twitter API access and SQLite installed, we can finally begin building a pipeline to store Tweets as we stream them over time. To start, we are going to create our new SQLite database using R as outlined below:</p>
<pre><code class="lang-{r}"># Import necessary libraries and functions
library(RSQLite)
library(rtweet)
library(tm)
library(dplyr)
library(knitr)
library(wordcloud)
library(lubridate)
library(ggplot2)
source(&quot;transform_and_clean_tweets.R&quot;)
</code></pre>
<pre><code class="lang-{r}"># Create our SQLite database
conn &lt;- dbConnect(RSQLite::SQLite(), &quot;Tweet_DB.db&quot;)
</code></pre>
<p>Afterward, we can create a table inside the database to hold the tweets. In our particular case, we are going to store the following variables:</p>
<ul>
<li>Tweet_ID as an INTEGER primary key</li>
<li>User as TEXT</li>
<li>Tweet_Content as TEXT</li>
<li>Date_Created as INTEGER</li>
</ul>
<p>You may wonder, why do I set the dates as integers? This is due to the fact that SQLite doesn&#39;t have a reserved data type for dates and times. Therefore, the dates will be stored as the number of seconds since 1970-01-01.</p>
<p>Let us now proceed to write the table:</p>
<pre><code class="lang-{r}">dbExecute(conn, &quot;CREATE TABLE Tweet_Data(
                  Tweet_ID INTEGER PRIMARY KEY,
                  User TEXT,
                  Tweet_Content TEXT,
                  Date_Created INTEGER)&quot;)
</code></pre>
<p>Once you have created the table, you can go to <strong>sqlite3.exe</strong> and check that is has indeed been created. You can see a screenshot of this below:</p>
<p><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1568658846/table_creation_m9ldw6.png" /></p>
<h2 id="step-2-stream-tweets-about-your-favourite-topics-">Step 2: Stream Tweets About your Favourite Topics!</h2>
<p>Believe it or not, you are done in terms of requirements and infrastructure needed to have a simple, functional Twitter streaming pipeline. What we need to do now is to stream Tweets using the API. It is worth noting that for the purposes of this tutorial, I will be using the standard free API. There are premium paid versions that may suit your Tweet streaming needs better if you conduct research, for instance.</p>
<p>Without further delay, let us start the process of setting up our Twitter listener. The first thing that you will need is to import the rtweet package and input your application&#39;s access tokens and secrets as described in the beginning:</p>
<pre><code class="lang-{r}">token &lt;- create_token(app = &#39;Your_App_Name&#39;,
                      consumer_key = &#39;Your_Consumer_Key&#39;,
                      consumer_secret = &#39;Your_Consumer_Secret&#39;,
                      access_token = &#39;Your_Access_Token&#39;,
                      access_secret = &#39;Your_Access_Secret&#39;)
</code></pre>
<p>With your token in place, the next thing is to decide what tweets do you want to stream (i.e., to listen to). The function <strong>stream_tweets</strong> in the rtweet package provides us with a variety of options to query the Twitter API. For example, you can stream tweets that contain one or more of a set of given hashtags or keywords (up to 400), a small random subset of all publicly available tweets, track the tweets of a group of user IDs or screen (user) names (up to 5000), or gather tweets by geographic location.</p>
<p>For this tutorial, I decided to go ahead and stream tweets containing hashtags related to data science (see list below). You may notice that the format in which I defined the hashtags to stream is a bit odd. However, that is the format required by the <strong>stream_tweets</strong> function when you want to listen to hashtags or keywords. This format varies if you intend to listen to a given set of users or based on coordinates. For more details, refer to the <a href="https://www.rdocumentation.org/packages/rtweet/versions/0.6.9/topics/stream_tweets">documentation</a>.</p>
<pre><code class="lang-{r}">keys &lt;- &quot;#nlp,#machinelearning,#datascience,#chatbots,#naturallanguageprocessing,#deeplearning&quot;
</code></pre>
<p>With the keywords defined, it is time to define the tweet streaming loop. There are several ways to do this, but this format has worked well for me in the past:</p>
<pre><code class="lang-{r}"># Initialize the streaming hour tally
hour_counter &lt;- 0

# Initialize a while loop that stops when the number of hours you want to stream tweets for is exceeded
while(hour_counter &lt;= 12){
  # Set the stream time to 2 hours each iteration (7200 seconds)
  streamtime &lt;- 7200
  # Create the file name where the 2 hour stream will be stored. Note that the Twitter API outputs a .json file.
  filename &lt;- paste0(&quot;nlp_stream_&quot;,format(Sys.time(),&#39;%d_%m_%Y__%H_%M_%S&#39;),&quot;.json&quot;)
  # Stream Tweets containing the desired keys for the specified amount of time
  stream_tweets(q = keys, timeout = streamtime, file_name = filename)
  # Clean the streamed tweets and select the desired fields
  clean_stream &lt;- transform_and_clean_tweets(filename, remove_rts = TRUE)
  # Append the streamed tweets to the Tweet_Data table in the SQLite database
  dbWriteTable(conn, &quot;Tweet_Data&quot;, clean_stream, append = T)
  # Delete the .json file from this 2-hour stream
  file.remove(filename)
  # Add the hours to the tally
  hour_counter &lt;- hour_counter + 2
}
</code></pre>
<p>In essence, that loop streams as many tweets as possible mentioning any of the hashtags in the key strings in intervals of 2 hours for a total time of 12 hours. Every 2 hours, the Twitter listener creates a .json file in your current working directory with the name specified in the variable <em>filename</em>.</p>
<p>Then, it passes this filename to the <em>transform_and_clean_tweets</em> function that removes retweets if desired, selects the columns we want to keep from all those given by the Twitter API, and normalizes the text contained in the Tweets.</p>
<p>Then, it appends the resulting dataframe to the <em>Tweet_Data</em> table that we created earlier inside our SQLite database. Finally, it adds 2 to the hour counter tally (since the streams last for 2 hours), and it removes the .json file created. This is done since all the data of interest is now in our database, and keeping the .json files can become a storage burden.</p>
<p>Let us know look closer to the <em>transform_and_clean_tweets</em> function in detail:</p>
<pre><code class="lang-{r}">transform_and_clean_tweets &lt;- function(filename, remove_rts = TRUE){

  # Import the normalize_text function
  source(&quot;normalize_text.R&quot;)

  # Parse the .json file given by the Twitter API into an R data frame
  df &lt;- parse_stream(filename)
  # If remove_rst = TRUE, filter out all the retweets from the stream
  if(remove_rts == TRUE){
    df &lt;- filter(df,df$is_retweet == FALSE)
  }
  # Keep only the tweets that are in English
  df &lt;- filter(df, df$lang == &quot;en&quot;)
  # Select the features that you want to keep from the Twitter stream and rename them
  # so the names match those of the columns in the Tweet_Data table in our database
  small_df &lt;- df[,c(&quot;screen_name&quot;,&quot;text&quot;,&quot;created_at&quot;)]
  names(small_df) &lt;- c(&quot;User&quot;,&quot;Tweet_Content&quot;,&quot;Date_Created&quot;)
  # Finally normalize the tweet text
  small_df$Tweet_Content &lt;- sapply(small_df$Tweet_Content, normalize_text)
  # Return the processed data frame
  return(small_df)
}
</code></pre>
<p>As noted, this function is meant to filter retweets if desired, retain the desired features, and normalize the tweet text. In essence, this can be considered the &quot;T&quot; part of the ETL acronym (the transformation). One key component of this function is the process of cleaning the tweet text.</p>
<p>Usually, text data requires a few preprocessing steps before it can be ready for any sort of analysis. In the case of tweets, these steps can include removal of URLs, stop words and mentions, turning the text to lower case, stemming words, etc. However, not all of these steps are necessary all of the time. Nevertheless, for the time being, I will display the <em>normalize_text</em> function that I used to preprocess these tweets below:</p>
<pre><code class="lang-{r}">normalize_text &lt;- function(text){
  # Keep only ASCII characters
  text = iconv(text, &quot;latin1&quot;, &quot;ASCII&quot;, sub=&quot;&quot;)
  # Convert to lower case characters
  text = tolower(text)
  # Remove any HTML tags
  text = gsub(&quot;&lt;.*?&gt;&quot;, &quot; &quot;, text)
  # Remove URLs
  text = gsub(&quot;\\s?(f|ht)(tp)(s?)(://)([^\\.]*)[\\.|/](\\S*)&quot;, &quot;&quot;, text)
  # Keep letters and numbers only
  text = gsub(&quot;[^[:alnum:]]&quot;, &quot; &quot;, text)
  # Remove stop words
  text = removeWords(text,c(&quot;rt&quot;,&quot;gt&quot;,stopwords(&quot;en&quot;)))
  # Remove any extra white space
  text = stripWhitespace(text)                                 
  text = gsub(&quot;^\\s+|\\s+$&quot;, &quot;&quot;, text)                         

  return(text)
}
</code></pre>
<p>Depending on your use case, these steps may be sufficient. As mentioned earlier, you may choose to add more or different steps such as stemming or lemmatization or to keep only letters instead of letters and numbers. I encourage you to experiment and try out different combinations. This can be a great place to practice regex skills.</p>
<p>After all of these steps, the resulting state is a SQLite database populated with all the streamed tweets. You can validate that everything worked correctly by running a couple of simple queries, for instance:</p>
<pre><code class="lang-{r}">data_test &lt;- dbGetQuery(conn, &quot;SELECT * FROM Tweet_Data LIMIT 20&quot;)
unique_rows &lt;- dbGetQuery(conn, &quot;SELECT COUNT() AS Total FROM Tweet_Data&quot;)
kable(data_test)
</code></pre>
<p><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570128184/kable_e3qnut.jpg" /></p>
<pre><code class="lang-{r}">print(as.numeric(unique_rows))
</code></pre>
<pre><code>## [1] 1863
</code></pre><h2 id="step-3-analyze">Step 3 Analyze</h2>
<p>When there is a certainty that the ETL process is working as intended, the final step is to gather some insights and analyze the data that you have collected. For example, for the tweets that we have collected, we can try a couple of different things; a wordcloud of terms mentioned in the tweet&#39;s content, and a timeline to visualize when in our 12-hour streaming period did we obtain the most amount of tweets. This list is obviously not a comprehensive overview of the whole array of research that can be conducted using tweets, which can range from sentiment analysis to psychographic studies and beyond.</p>
<p>With that said, lets jump straight to building a nice wordcloud:</p>
<pre><code class="lang-{r}"># Gather all tweets from the database
all_tweets &lt;- dbGetQuery(conn, &quot;SELECT Tweet_ID, Tweet_Content FROM Tweet_Data&quot;)

# Create a term-document matrix and sort the words by frequency
dtm &lt;- TermDocumentMatrix(VCorpus(VectorSource(all_tweets$Tweet_Content)))
dtm_mat &lt;- as.matrix(dtm)
sorted &lt;- sort(rowSums(dtm_mat), decreasing = TRUE)
freq_df &lt;- data.frame(words = names(sorted), freq = sorted)

# Plot the wordcloud
set.seed(42)
wordcloud(words = freq_df$words, freq = freq_df$freq, min.freq = 10,
          max.words=50, random.order=FALSE, rot.per=0.15,
          colors=brewer.pal(8, &quot;RdYlGn&quot;))
</code></pre>
<p><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570046250/2_xjdbkd.png" /></p>
<p>Well, what a tremendous surprise! <em>machinelearning</em> and <em>datascience</em> are the most mentioned words in all our tweets—they just happen to be two of the hashtags that we were streaming for. Thus, this result was expected. The other words though are a bit more fun to look at. For example, <em>bigdata</em> and <em>artificialintelligence</em> were not hashtags in our keys, yet they appeared with high frequency, so one could argue that they are talked about often along with the other two. There are other words of interest that we picked up such as <em>python</em> or <em>tensorflow</em> that gives us a little bit more context into the content of the tweets we collected beyond just the hashtags.</p>
<p>Let us now turn to another simple analysis. In our 12-hour stream, at what time did we collect the most tweets? For that, we will gather our integer dates, turn them into a proper format, and then plot the quantity of tweets over time:</p>
<pre><code class="lang-{r}"># Select the dates in which the tweets were created and convert them into UTC date-time format
all_tweets &lt;- dbGetQuery(conn, &quot;SELECT Tweet_ID, Date_Created FROM Tweet_Data&quot;)
all_tweets$Date_Created &lt;- as.POSIXct(all_tweets$Date_Created, origin = &quot;1970-01-01&quot;, tz = &quot;UTC&quot;)

# Group by the day and hour and count the number of tweets that occurred in each bucket
all_tweets_2 &lt;- all_tweets %&gt;%
    mutate(day = day(Date_Created),
           month = month(Date_Created, label = TRUE),
           hour = hour(Date_Created)) %&gt;%
    mutate(day_hour = paste(month,&quot;-&quot;,day,&quot;-&quot;,hour, sep = &quot;&quot;)) %&gt;%
    group_by(day_hour) %&gt;%
    tally()

# Simple line ggplot
ggplot(all_tweets_2, aes(x = day_hour, y = n)) +
  geom_line(aes(group = 1)) +
  theme_minimal() +
  ggtitle(&quot;Tweet Freqeuncy During the 12-h Streming Period&quot;)+
  ylab(&quot;Tweet Count&quot;)+
  xlab(&quot;Month-Day-Hour&quot;)
</code></pre>
<p><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570046251/3_fgkhum.png" /></p>
<p>Sweet! Now we can see that the largest amount of unique tweets that we obtain came from the 2nd of September between 8:00 PM UTC and 8:59PM UTC (This appears in military time as 20).</p>
<h2 id="conclusion">Conclusion</h2>
<p>Congratulations! Now you know how to build a simple ETL pipeline in R. The two analyses we conducted represent very basic analyses conducted using Twitter data. However, as mentioned previously, there are lots of things to do as long as you build a robust pipeline to bring in the data. This aspect was the main focus of this tutorial.</p>
<p>With that said, this tutorial only showed a very small scale case study to walk over the process of building ETL pipelines for Twitter data. Building robust and scalable ETL pipelines for a whole enterprise is a complicated endeavor that requires extensive computing resources and knowledge, especially when big data is involved.</p>
<p>I encourage you to do further research and try to build your own small scale pipelines, which could involve building one in Python$^1$. Maybe, even go ahead and try some big data projects. For example, DataCamp already has some courses like this <a href="https://www.datacamp.com/courses/big-data-fundamentals-via-pyspark">Big Data Fundamentals via PySpark</a> course where they go over big data using tools such as PySpark that can further your knowledge of this field.</p>
<p>If you would like to learn about data engineering, take DataCamp&#39;s <a href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a> course.</p>
<h2 id="references">References</h2>
<ol>
<li>Foley, D. (2019, May 11). Streaming Twitter Data into a MySQL Database. Retrieved from <a href="https://towardsdatascience.com/streaming-twitter-data-into-a-mysql-database-d62a02b050d6">https://towardsdatascience.com/streaming-twitter-data-into-a-mysql-database-d62a02b050d6</a></li>
</ol>


https://www.datacamp.com/community/tutorials/building-a-tweet-etl-pipeline-using-r

Learn how to transform and load (ETL)  a data pipeline from scratch using R and SQLite to gather tweets in real-time and store them for future analyses.

community

datacamp

data engineering

News (10)

Difference between Data Science and Big Data

From Local Machine to Dask Cluster with Terraform

Tutorials (2)

Data Visualization with Bokeh Tutorial: Plotting Data Structures

Building Data Pipelines using R

Cheat Sheets (0)

Open Courses (0)

Podcast (0)

Blog Posts (23)

What is Data Engineering?

Tech Thoughts (0)