https://www.datacamp.com/community/news/difference-between-data-science-and-big-data-5tm3qkq80bj

Difference Between Data Science and Big Data is really complicated. Data Science is a field that comprises of everything that related to data cleaning, preparation, and analysis. Big Data refers to humongous volumes of data that cannot be processed effectively with the traditional applications that 

Difference between Data Science and Big Data

https://www.datacamp.com/community/news/how-to-become-a-data-scientist-in-one-week-sr4leab48c

How to Become a Data Scientist in One Week?

https://www.datacamp.com/community/news/from-local-machine-to-dask-cluster-with-terraform-d9klqtbw6e6

A comprehensive  tutorial on how to start a dask cluster

From Local Machine to Dask Cluster with Terraform

AN ANALYSIS OF TOP 7 TRENDING DATA SCIENCE CAREERS TO PURSUE
Examining the incredibly popular field that is data science
Data processing, analysis, and visualizations were done using R and its corresponding packages. Notebook and pertinent files used can be found <a target="_blank" href="https://app.datacamp.com/workspace/w/03312344-a7b0-47cf-b97d-06125fabcbd8/edit">here</a>.
According to the Merriam-Webster dictionary, data is factual information used as a basis for reasoning, discussion, or calculation. By this definition, data is basically any piece of collected information that can be used and further processed and analyzed to gain insights. It is commonly associated with computers as data is normally produced and stored in a computer, yet data has been around much longer than we might imagine.
<h1 id="history-of-data">History of Data</h1>
The earliest example of humans storing and analyzing data dates back to 18000 BCE, when it was discovered that prehistoric humans were using tally sticks, as a means of carrying out rudimentary calculations. These Paleolithic tribespeople marked notches on sticks and bones to keep track of their activities, such as trading and monitoring supplies. By 2400 BCE, the abacus, a device used in performing calculations, was invented in Babylon.
Throughout history, constant development of data collection, processing, and analysis was seen through numerous writings in stone tablets, clay, papyrus, wood, and paper scrolls. Eventually, as more forms of data were discovered, the need for processing, collecting, storing, and analyzing it also evolved.
As human society became more advanced, so did its requirement for processing data. In the 1800s, the census started taking place in the United States. The number of data points in the census grew exponentially to the point that the U.S. Census Bureau estimated that it would take years or even decades to collect and analyze all the data in the census. This was largely problematic since compiling and analyzing the current census data would only be completed when the next census was about to start or had already started. Fortunately, a young engineer and inventor named Herman Hollerith developed the Hollerith Tabulating Machine–an electromechanical tabulating machine which reduced the time needed to collect and analyze census data from years, to just mere months. Because of this, Hollerith is regarded as the father of modern automatic computation and would later become known for founding IBM.
Fast forward to the 1900s, and the invention of the computer. With the advent of a more powerful computing machine comes a more sophisticated requirement for data storage. Fritz Pfleumer, a German-Austrian engineer, invented a method of storing information magnetically on tape. Some principles of his invention are still being used today for digital data storage. In this era, the term “business intelligence” gained popularity as the need for emerging software, and systems for analyzing commercial and operational performance, quickly rose.
The revolution of data truly changed when Tim Berners-Lee created the World Wide Web, also known as the Internet, in 1989. This led to automated information sharing between people all over the world. This means that today more data is being shared, created, and stored, leading to new ways for collecting, using, and analyzing data.
<h1 id="transition-to-big-data-">Transition to “Big Data”</h1>
Due to the incredible amount of internet growth in the 1990s and the steady development of personal computers and computing devices in general, the number of devices online (and consequently the amount of data created) grew quickly.
While the idea of Big Data had been around even before the 1990s, it was only in 2005 when Roger Mougalas officially gave it a label. He described it as “a large set of data that is almost impossible to manage and process using traditional business intelligence tools.”
Big Data is a term used to describe the large volume of data–both structured and unstructured–that overwhelms organizations on a day-to-day basis. It encompasses the volume of information, the velocity or speed at which it is created and collected, and the variety or scope of the data points being covered.
Given the size and complexity of Big Data, the process of collecting, organizing, and analyzing it to discover patterns and other useful information has become part of helping a lot of organizations to make their business decisions. In turn this has given rise to data science–an interdisciplinary field which uses scientific methods, processes, algorithms, and systems from a large volume of data, to uncover patterns and enable business leaders to draw informed insights.
<h1 id="what-is-data-science-">What is Data Science?</h1>
According to IBM, data science is a multidisciplinary approach to extracting actionable insights from the large and ever-increasing volumes of data collected and created by today’s organizations. The field typically requires computer science and pure science skills as data scientists apply scientific methods in their approach, and use predictive analytics and artificial intelligence to extract insights from the data.
Nowadays, “data science” is often used by businesses and organizations as a catch-all term for the process of handling large amounts of data, whether it is preparing, cleaning, analyzing, or visualizing the data to reveal patterns. These are just some of the data science careers one can pursue:
<ol>
<li>Data Scientist: 
Data scientists need to be able to apply mathematics, statistics, and the scientific method; use multiple tools and techniques for cleaning and preparing data; perform predictive analytics and artificial intelligence; and explain how these results can be used to provide data-driven solutions to business problems. They require much more technical skills than data analysts. For starters, you may take data science-oriented courses like these on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/intro-to-python-for-data-science">Introduction to Python for Data Science</a> and <a target="_blank" href="https://www.datacamp.com/courses/free-introduction-to-r">Introduction to R</a>.
</li>
<li>Data Analyst: 
Data analysts collect, process, and perform statistical data analyses to arrive at meaningful conclusions for organizations. They transform and manipulate large data sets into usable forms, such as reports or presentations. They also aid in the decision-making process by looking into important patterns, and gleaning insights from the data which they can then effectively communicate to organizational leaders to aid in business decisions. These courses on DataCamp may help you kick-start your data analyst career: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-sql">Introduction to SQL</a> and <a target="_blank" href="https://www.datacamp.com/courses/exploratory-data-analysis-in-sql">Exploratory Data Analysis in SQL</a>.
</li>
<li>Data Engineer: 
Data engineers are responsible for preparation, processing, and managing gathered and stored data for analytical or operational uses. Like the traditional engineers, data engineers build and maintain data “pipelines” which connect data from one system to another, making information accessible for data scientists. Because of this, data engineers are required to understand several programming languages used in data science such as Python, R, and SQL. Topics related to data engineering like these are available on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a> and <a target="_blank" href="https://www.datacamp.com/courses/data-engineering-for-everyone">Data Engineering for Everyone</a>.
</li>
<li>Data Architect: 
Data architects mainly design and create blueprints for data management systems that data engineers then build. Analogous to the traditional architects, data architects are the “visionaries” as they are responsible for visualizing and designing an organization’s data management framework. In addition, data architects improve the performance of existing systems, making sure they are accessible to database administrators and analysts. You may take courses related to data architecture like these that are available on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/aws-cloud-concepts">AWS Cloud Concepts</a> and <a target="_blank" href="https://www.datacamp.com/courses/database-design">Database Design</a>
</li>
<li>Business Intelligence (BI) Developer: 
BI developers are specialized engineers who use software tools to transform data into useful insights that help business decisions. They are responsible for simplifying technical information for everyone else in the company to easily understand. In short, they create and run reports containing data they find using business intelligence tools, and transform the information into more layman terms. Courses on using tools related to business intelligence such as Microsoft Power BI and Tableau are available on DataCamp such as these: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-power-bi">Introduction to Power BI</a> and <a target="_blank" href="https://www.datacamp.com/courses/analyzing-data-in-tableau">Analyzing Data in Tableau</a>
</li>
<li>Statistician: 
Given that statistics is one of the main foundations of data science, many statisticians can easily transition into the data science field. Statisticians are mainly responsible for the collection and handling of data. They decide what data they need and how to collect it. Additionally, they design experiments, analyze and interpret the data, and report conclusions. You may check-out these introductory courses to statistics on DataCamp: <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-statistical-modeling-in-r">Introduction to Statistical Modeling in R</a> and <a target="_blank" href="https://www.datacamp.com/courses/introduction-to-statistics-in-spreadsheets">Introduction to Statistics in Spreadsheets</a>.
</li>
<li>Machine Learning Engineer: 
Machine learning engineers are another group of specialized engineers who are focused on researching, building, and designing artificial intelligence (AI) and machine learning (ML) systems to automate predictive models. Basically, they develop algorithms that use input data and leverage statistical models to predict an output, while continuously updating outputs as new data becomes available. DataCamp has numerous courses related to machine learning which you may find useful such as these: <a target="_blank" href="https://www.datacamp.com/courses/machine-learning-for-everyone">Machine Learning for Everyone</a> and <a target="_blank" href="https://www.datacamp.com/courses/machine-learning-with-caret-in-r">Machine Learning with caret in R.</a>
</li>
</ol>
A quick job search online can give you an idea as to which of the data science career paths listed above is the most in demand. The graph below shows the job openings per career path according to <a target="_blank" href="https://www.indeed.com/jobs?q=data%20scientist&amp;vjk=ecfd94ad120e8423">Indeed USA</a> as of December 8, 2021.
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image1_hntxle.png" /></center>

Data architect appears to be the most in demand data science career path, due to their importance in creating data management systems which the other data science professionals then use. Next on the list is machine learning engineer, given the importance of utilizing artificial intelligence to predict outcomes in a lot of tech companies.
The least in demand are statisticians, mainly because a lot of the traditional statisticians are now becoming data scientists. This trend is expected to continue given how easy it is for statisticians to move from pure statistics to data science as they already possess the fundamental knowledge needed to become a fully-fledged data scientist.
<h1 id="data-science-is-not-slowing-down">Data science is not slowing down</h1>
The field of data science rose to immense popularity in the 2010s, especially when “Data Scientist” was touted by Harvard Business Review as the “sexiest job of the 21st century.” It became so popular that the title “Data Scientist” became a ubiquitous and trendy expression. While the term “sexiest job” does sound ridiculous at first, it mostly refers to how data scientists are not only in demand, but are also expensive to hire and difficult to retain.
While there is no question that data science is extremely popular nowadays, the better question is, will it remain this popular in the future? Based on employment projections, that does seem to be the case. Data from the 2020-2030 Employment Projections of the <a target="_blank" href="https://www.bls.gov/emp/tables/emp-by-detailed-occupation.htm">U.S. Bureau of Labor Statistics</a> shows that data science careers, which include statistics, data science, and other mathematical and science based occupations such as data engineering, will exhibit a very high growth rate in terms of percent change from 2020 to 2030. Statisticians ranked 14th overall, while data scientists and other mathematical science occupations ranked 31st overall out of 790 jobs included in the data.
While the shares of statisticians and data scientists from the total labor force are small compared to other occupations, those numbers are expected to increase in following years as the data science career path becomes more and more popular. The figure below shows how statisticians, data scientists, and other mathematical science occupations stack up against other jobs with high projected growth.
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image3_bwn60b.png" /></center>


<h1 id="a-really-expensive-field">A really expensive field</h1>
One major reason why data science is popular is because it is one of the highest paying jobs in the world. The figure below shows the range of salaries for 10 different occupations–including data scientists and data analysts–in New York City. The data was obtained from <a target="_blank" href="https://teleport.org/cities/new-york/salaries/">Teleport</a>, a website which aggregates data on living conditions (e.g., salaries) in different cities.
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image2_qhdjq6.png" /></center>

According to Teleport, data scientists had the 4th highest annual median salary in New York City at \$114,105, just behind C Level Executives and healthcare workers. In fact, in some other cities like Manila, data scientists rank as high as 2nd highest, just behind C Level Executives. This speaks volume as to how lucrative this job is. Data analyst jobs, whilst not as lucrative as data science positions, are still not to be slighted. At an annual median salary of $61,818, data analysts still earn around the average household income in New York City.
<h1 id="ever-growing-in-the-job-market">Ever growing in the job market</h1>
Another major reason why data science is popular is because of how organizations these days integrate the principles of data science in their day-to-day operations. The graph below shows the top 10 industries that involve data science in their work. The sample data of 8,000 data science oriented companies are extracted from <a target="_blank" href="https://www.diffbot.com/products/extract/testdrive/?token=210ced42a6e6d0cc4ac7f15885a94233.">Diffbot</a>, the world’s largest knowledge graph.
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1641468289/image4_spyov2.png" /></center>

It comes to no surprise that Software Companies–including tech companies like Google, Apple, and Uber–came out on top. After all, the exceptional growth of big data was caused by the creation of the internet, which is closely tied to software and technology. Data science really is about processing a lot of information using various tools and technology.
Next on the list are financial services companies, which has been brought about by the rise of fintech companies. A portmanteau of “finance” and “technology,” fintech companies integrate technology and innovation in their services and offerings to improve their delivery to customers and disrupt traditional financial services. Since it involves handling a vast amount of data such as customer information, financial services companies saw the potential of data science tools to help streamline and optimize their processes and improve their services.
<h1 id="in-conclusion-is-it-worth-pursuing-a-career-in-data-science-">In conclusion, is it worth pursuing a career in data science?</h1>
With everything that has been explained above, the answer is a definite yes! As to which specific data science career path you should pursue, this depends primarily on your individual strengths and general interests. What is important is that any of the careers mentioned above are sure to be worthwhile.
A career path which is very lucrative and appears to show no signs of slowing down in terms of popularity, data science will continue to shape and influence how businesses and organizations operate for many years to come.

https://www.datacamp.com/community/blog/blog-data-science-careers-to-pursue

Explore your options in the world of data science, the various options available with different types of roles. Get an idea how to start preparing yourself for each of the roles.

Explore all the roles under the umbrella Data science.  And discover introductory courses for the same.

An Analysis of the Top 7 Trending Data Science Careers to Pursue 

<h2 id="the-state-of-data-driven-digital-transformation">The state of data-driven digital transformation</h2>
As organizations undergo accelerated digital transformations amid the COVID-19 pandemic, the case for leveraging data, machine learning and analytics has never been more important.
The 2020 McKinsey Global Survey on the <a href="https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/global-survey-the-state-of-ai-in-2020">State of Artificial Intelligence</a> reported that 50% of businesses have adopted artificial intelligence in at least one business function, and 22% of organizations attribute at least 5% of earnings before interest and tax to artificial intelligence initiatives.
However, in order to unlock the value of data and artificial intelligence at scale, organizations must democratize their data and unlock organization-wide data fluency. One of the ways to do so is by providing access to trusted and actionable data across teams and skill levels so that everyone can make data-informed business decisions.
<h2 id="data-infrastructure-at-the-core-of-data-democratization">Data infrastructure at the core of data democratization</h2>
Before achieving data fluency, organizations need to get the foundations right. This starts with understanding how data flows through an organization and then building toward an organized and unified data infrastructure.
In a <a href="https://www.datacamp.com/resources/webinars/data-trends-2021">webinar on 2021 data trends and predictions</a>, Ramnath Vaidyanathan, Vice President of Product Research at DataCamp, described how data typically flows through an organization. This flow requires data engineering tools to take in raw data, ingest and transform them, store them in a centralized location, and produce data insights in the form of dashboards and reports, machine learning outputs, and more.
Scaling access to data within the organization depends heavily on the presence of a solid data infrastructure architecture and the use of the latest data infrastructure tooling, which enables organizations to transform raw data into accessible and valuable data insights.
<center><a href="https://www.datacamp.com/resources/webinars/data-trends-2021"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906487/dataflowsthruorg_xgcspr.png" /></a> 
<h2 id="the-state-of-data-infrastructure-tooling">The state of data infrastructure tooling</h2>
The importance of data infrastructure and tooling is evidenced by the rapid growth and evolution of the data infrastructure market. Many solutions are meant to streamline working with a particular element of an organization’s data infrastructure, which has resulted in an increasingly fragmented landscape of tooling that can be challenging to navigate.
The following visual created by Dataiku aptly illustrates how the data infrastructure market has been segmented into numerous granular categories, with an expanding list of competing solutions present within each segment.
<center><a href="https://blog.dataiku.com/technoslavia-the-fragmented-world-of-data-infrastructure-in-2020"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906493/technoslavia_gxmlfq.png" /></a>
<h5 id="source-dataiku-https-blog-dataiku-com-technoslavia-the-fragmented-world-of-data-infrastructure-in-2020-">Source: <a href="https://blog.dataiku.com/technoslavia-the-fragmented-world-of-data-infrastructure-in-2020">Dataiku</a></h5>
<h2 id="the-future-of-data-infrastructure-tooling">The future of data infrastructure tooling</h2>
Late last year, Andreessen Horowitz sought to provide some clarity in this confusing data infrastructure landscape by developing a unified architecture framework. With the growing need to work with large volumes of data and the plethora of available solutions, this framework offers a practical reference when modernizing, mapping, and building an organization’s data stack.
<center><a href="https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906493/andreeson_bjpxzl.png" /></a>
<h5 id="source-andreessen-horowitz-https-a16z-com-2020-10-15-the-emerging-architectures-for-modern-data-infrastructure-">Source: <a href="https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/">Andreessen Horowitz</a></h5>
As this space and trend evolves, we will see the consolidation and standardization of various tools in the data infrastructure stack. This alongside cloud adoption, data discovery tools, and centralized data governance platforms will allow organizations to provide scalable access to data across teams and functions.
So what does an ideal future data infrastructure deliver? According to <a href="https://www.datacamp.com/resources/webinars/operationalizing-data-within-large-organizations">Maksim Pecherskiy</a>, Data Engineer at the World Bank and former CDO of the city of San Diego, data must be collected, discoverable, reliable, understood, compliant, and actionable. Ultimately, the goal is to make data usable and trustworthy, which in turn drives organization-wide data fluency.
<center><a href="https://www.datacamp.com/resources/webinars/data-trends-2021"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1615906507/usedandtrusted_wide_ypurb3.png" /></a>
<h2 id="mature-data-infrastructure-includes-cloud-adoption-data-discovery-tools-and-centralized-data-governance-platforms">Mature data infrastructure includes cloud adoption, data discovery tools, and centralized data governance platforms</h2>
As data infrastructure tooling matures around data democratization in 2021 and beyond, so will the adoption and growth of the following technologies:
<h3 id="1-cloud-adoption">1. Cloud Adoption</h3>
The use of cloud technologies is <a href="https://www.oreilly.com/radar/cloud-adoption-in-2020/">becoming the norm</a>, especially in the wake of the accelerated data-driven digital transformation we’ve seen in 2020.
Besides improving data storage and access, cloud solutions also offer numerous user-friendly products for collecting, ingesting, transforming, storing, and creating insights from data.
There are numerous providers in the cloud platform space, and one of the key players is <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS), which has <a href="https://aws.amazon.com/products/">more than 100 products</a> that enable enterprises to make the most out of their data at high speed, at scale, and at low cost.
<h3 id="2-data-discovery-tools">2. Data Discovery Tools</h3>
Scaling data infrastructure will be crucial as data tables, reports, and models are added to organizational data landscapes. Stakeholders must be readily able to locate reliable, understood, compliant, and actionable data and insights to do their best work.
To tackle this challenge, <a href="https://datacamp.com/community/data-discovery-tools">data discovery tools</a> (also referred to as metadata management tools or data catalogs) can act as versatile search engines for these troves of data within the organization. Metadata essentially refers to a set of data describing the particular data, and is useful for indexing the large variety of data available.
Lyft open-sourced their own data discovery tool <a href="https://www.amundsen.io/">Amundsen</a>, to offer metadata engine solutions that aim to help organizations discover, understand, share, and trust the data they use for their analyses. This capability is paramount for any organization looking to scale its data competencies.
<h3 id="3-centralized-data-governance-platforms">3. Centralized Data Governance Platforms</h3>
As mentioned earlier, creating trust in data is essential for driving data democratization. This is where data governance plays a major role. With the ever-growing volume and variety of data, centralized oversight of the data quality, consistency, and relevance is crucial.
Centralized data governance platform services offered by companies like <a href="https://www.collibra.com/">Collibra</a> and <a href="https://www.alation.com/solutions/data-governance/">Alation</a> help organizations effectively manage the availability, consistency, and integrity of their data. This then allows all stakeholders to work from a reliable single source of truth.
A maturing data infrastructure is only one of the data trends we foresee for 2021 and beyond. For more, download our white paper on <a href="https://www.datacamp.com/resources/whitepapers/data-trends-and-predictions-2021">Data Trends and Predictions 2021: The Year of Data Fluency</a> or watch our <a href="https://www.datacamp.com/resources/webinars/data-trends-2021">webinar</a>.
<center><a href="https://www.datacamp.com/groups/business"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603718763/banner_style_1_cruuf7.png" /></a>

https://www.datacamp.com/community/blog/data-infrastructure-tools

Unlocking the value of data in an organization starts with having the right data infrastructure and tooling foundations. Here’s a look at the current state and future trends of data infrastructure.

What Mature Data Infrastructure Looks Like | DataCamp

What Mature Data Infrastructure Looks Like

<ol>
<li>Introduction</li>
<li>What are the responsibilities of a data engineer?</li>
 <ul>
<li>ETL Pipelines</li>
<li>Data Warehouses</li> 
</ul>
<li>Data engineering vs data science</li>
<li>Why has data engineering increased in popularity?</li>
<li>What are the skills of a data engineer?</li>
<li>How to learn data engineering</li>
</ol>
<h1 id="introduction">Introduction</h1>
Data is everywhere. In recent years, with the growing number of connected devices and the expansion of their use, the world has been producing a vast amount of data that cannot be handled with traditional methods anymore.
According to Statista, as of 2021 there are <a target="_blank" href="https://www.statista.com/statistics/1101442/iot-number-of-connected-devices-worldwide/">23.8 billion devices connected to the internet</a> and 58% of those are IoT devices (smart home devices, connected self-driving cars, industrial devices, etc.) and the remaining 42% are non-IoT devices (laptops, smartphones, etc.). This means that nowadays data can come from various sources. Some of these are:
<ol>
<li>User activities: Any user action produces data and even the data that reflects where and how often we tap on the mobile applications we use is valuable. Every piece of data that contains this kind of information needs to be kept and distributed to the applications’ data centers.</li>
<li>“Internet of Things” devices: These devices produce a huge amount of information, such as sensor data generated in many different locations. All this data needs to be retrieved and sent to a data pool for further analysis.</li>
<li>Program logs: Today’s computer applications consist of many components and all those components produce program logs. These logs are sent to a data engineering pipeline for further analysis.</li>
</ol>
All the different structured data that the world produces needs to be delivered to other components in the software structure, or visualized and interpreted by business analysts and data scientists. However, this delivery is not a simple task. There are some crucial requirements on how to deliver the data:
<ol>
<li>Data needs to be delivered in a standardized structure</li>
<li>The data should not present any conflicts</li>
<li>Data duplication may need to be removed</li>
<li>The data needs to be reliable</li>
<li>The data delivery must be done with minimum delay</li>
</ol>
Therefore, the delivery of this information is a complex process and, if it is not carried out properly, it can be very problematic, especially for medium and large-scale companies. This is where data engineers come into play.
In this article you will learn the basics of data engineering, which are:
<ol>
<li>What data engineering is and why to ensure a robust data pipeline</li>
<li>The difference between data science and data engineering</li>
<li>The reason behind the increasing popularity of data engineering</li>
<li>The skills of a good data engineer</li>
<li>How DataCamp can help you become a data engineer</li>
</ol>
<h1 id="what-are-the-responsibilities-of-a-data-engineer-">What are the responsibilities of a data engineer?</h1>
To put it simply, data engineers:
<ol>
<li>Aim to make data easily accessible and available for data scientists and business intelligence engineers (or anyone working with data)</li>
<li>Optimize the big data system architecture of companies</li>
<li>Design and configure databases</li>
<li>Collaborate with teams of business intelligence engineers, data scientists, and data analysts</li>
<li>Explore and transform data
In order to perform all these tasks, data engineers first need to set up the ETL pipelines.</li>
</ol>
<h1 id="etl-pipelines">ETL pipelines</h1>
ETL pipelines receive complex data at regular time intervals and process it with the aim of storing the data in a usable form. It is the responsibility of the data engineers to establish and maintain these pipelines.
There are 3 types of ETL pipelines:
<ol>
<li>Extract
A vast number of data created by data sources is collected in different formats.</li>
<li>Transform
Since the data comes in different formats and due to its inconsistent sources, the data is processed to become as standardized as possible. This increases the discoverability and usability of the data for the next steps.</li>
<li>Load
Once the data is in a discoverable and usable format, it is usually stored in data warehouses. This makes the data available at all times for further analysis.</li>
</ol>
<h1 id="data-warehouses">Data warehouses</h1>
Data warehouses are where standardized data is stored. They are optimized for filtering and reading large volumes of data. In modern data warehouses, both structured and unstructured data can be stored, because it is also important to store the data that cannot be categorized, such as photos and videos.
<h1 id="data-engineering-vs-data-science">Data engineering vs data science</h1>
The differences between the roles of a data engineer and a data scientist are important. On the one hand, data scientists have an important role in companies because they contribute to data-driven decision making. Nevertheless, the success of data scientists is only as good as the data that they have available.
On the other hand, data engineers are the ones who design the systems that provide data to data scientists in an organized and consistent manner. Once the data engineers make the information available for them, data scientists then try to find patterns in the data to reach conclusions, using the format provided by the data engineers.
Data engineers and data scientists clearly work closely together. We can think of it as a football match. In football, it is the coach who develops the strategies that will create goal opportunities, and it is the coach who sets the players in certain positions. In this context the coach is the data engineer, so we can consider that the ball is the data and the player who scores is the data scientist. The rest of the team is the infrastructure.
<h1 id="why-has-data-engineering-increased-in-popularity-">Why has data engineering increased in popularity?</h1>
According to research conducted on the <a href="https://www.burning-glass.com/products/nova/">Burning Glass Nova</a> platform, job postings for &quot;data engineer&quot; increased by 88.3% in 2019. According to this research, demand for data engineers has been on the rise since 2016. Another interesting piece of information is that the number of job openings for data engineers is almost 5 times higher than the job openings for data scientists. The salary of a data engineer is also 20-30% higher than the salary of a data scientist.
As stated above, data scientists are the final link in the data processing chain. They need the data to be processed by a data engineer in order to perform their job. As the number of data sources and data types multiply, the need for a data engineer grows. For this reason, companies have recently started to invest more in the extraction and distribution of data.
In Figure 4.1, a pyramid chart displays the hierarchy of needs of data science. In this hierarchy, the tasks assigned to data engineers are at the base of the pyramid, which explains how crucial data engineers are for companies.
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1640794050/image3_kje7by.png" />Figure 4.1: The data science hierarchy of needs.</center>

<center><a target="_blank" href="https://medium.com/hackernoon/the-ai-hierarchy-of-needs-18f111fcc007">Source: hackernoon</a></center>

<h1 id="what-are-the-skills-of-a-data-engineer-">What are the skills of a data engineer?</h1>
The skills of data engineers are generally intertwined with the skills of software engineers and data scientists, as shown in Figure 5.1.
 <center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1640794050/image2_rqg5yb.png" />
Figure 5.1: Data roles and skill sets</center>
<center><a target="_blank" href="https://ryanswanstrom.com/2016/11/28/data-scientists-data-engineers-software-engineers-the-difference-according-to-linkedin/">Source: RyanSwanstrom</a></center>

<h1 id="is-the-role-of-a-data-engineer-closer-to-the-one-of-a-software-engineer-or-to-the-one-of-a-data-scientist-">Is the role of a data engineer closer to the one of a software engineer or to the one of a data scientist?</h1>
The previous occupations of a group of data engineers are displayed in Figure 5.2. According to this figure, over 40% of them have a software engineering background. Not only does this mean that data engineering and software engineering are significantly closer than we may have first thought, but also that software engineers nowadays are much more likely to switch to data engineering than any other field.
 <center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1640794050/image5_d4vwny.png" />
Figure 5.2: Data engineers by prior role (top 10).</center>
<center><a href="https://www.stitchdata.com/resources/the-state-of-data-engineering/">Source: stitchdata</a></center> 

At DataCamp we believe the following to be the skills of a good data engineer:
<ol>
<li>Software engineering background
Data engineers need to have a software engineering background, since they need to approach data engineering challenges with their knowledge of object-oriented programming, data structures and algorithms. Data engineers must be experienced in Python, Scala or Java.</li>
<li>Tools
Data engineers need to be comfortable with data engineering tools like Airflow, Apache or Kafka.</li>
<li>Database Technologies
A data engineer should have knowledge of multiple kinds of databases.</li>
<li>Cloud Technologies
Data engineers should be comfortable using multiple cloud platforms like Amazon Web Services, Google Cloud Platform and Microsoft Azure. They need to have experience in cloud architecture and DevOps tools as well, since they are responsible for the automation of the data flow part of the system.</li>
</ol>
<h1 id="how-to-learn-data-engineering">How to learn data engineering</h1>
If you want to improve in the data engineering field, you can start by completing the following recommended DataCamp courses. By completing these courses, you will likely become a more suitable candidate for internships or job positions in the field of data engineering.
<ol>
<li><a target="_blank" href="https://www.datacamp.com/courses/intro-to-python-for-data-science">Introduction to Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/intermediate-python">Intermediate Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-relational-databases-in-sql">Introduction to Relational Databases in SQL</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-scala">Introduction to Scala</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/introduction-to-airflow-in-python">Introduction to Airflow in Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/aws-cloud-concepts">AWS Cloud Concepts</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/building-data-engineering-pipelines-in-python">Building Data Engineering Pipelines in Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/nosql-concepts">NoSQL Concepts</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/etl-in-python">ETL in Python</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/streaming-concepts">Streaming Concepts</a></li>
<li><a target="_blank" href="https://www.datacamp.com/courses/streaming-data-with-aws-kinesis-and-lambda">Streaming Data with AWS Kinesis and Lambda</a></li>
</ol>

https://www.datacamp.com/community/tutorials/what-is-data-engineering

 Learn what data engineering is, what is the difference between data science and data engineering , scope in the field and how to become one.

What is Data Engineering?

One could argue that proper ETL pipelines are a vital organ of data science. Without clean and organized data, it becomes tough to produce quality insights that enhance business decisions.
Therefore, in this tutorial, we will explore what it entails to build a simple ETL pipeline to stream real-time Tweets directly into a SQLite database using R. This is a fairly common task involved in social network analysis for example.
The focus will be on covering the thought process of data collection and storage as well as how to operate the Twitter API using the <a href="https://rtweet.info/">rtweet</a> package in R.
To begin, let&#39;s start with having all the right tools. The first thing that you will need is set up your access to the Twitter API. Overall, you would need to follow these steps:
<ul>
<li>Create a Twitter account, if you don&#39;t have one.</li>
<li>Follow this <a href="https://developer.twitter.com/en/apply-for-access">link</a> and apply for a developer account (note that this process now requires an application that Twitter has to approve).</li>
<li>Create a new app on the following <a href="https://developer.twitter.com/en/apps">webpage</a>$^1$.</li>
<li>Fill in all the details about your app and create your access token$^1$.</li>
<li>Gather your consumer key, consumer secret, access token, and access token secret, which you need to connect to the API.</li>
</ul>
Once you are all set with the Twitter API, you need to get SQLite if you don&#39;t have it. For the full process on how to get SQLite on your computer, follow the <a href="https://www.datacamp.com/community/tutorials/beginners-guide-to-sqlite">Beginners Guide to SQLite tutorial</a> here at DataCamp. The choice of using SQLite for this tutorial was made due to its simplicity of operation.
<h2 id="step-1-creating-a-database-and-table-to-store-the-twitter-data">Step 1: Creating a Database and Table to Store the Twitter Data</h2>
With both the Twitter API access and SQLite installed, we can finally begin building a pipeline to store Tweets as we stream them over time. To start, we are going to create our new SQLite database using R as outlined below:
<pre><code class="lang-{r}"># Import necessary libraries and functions
library(RSQLite)
library(rtweet)
library(tm)
library(dplyr)
library(knitr)
library(wordcloud)
library(lubridate)
library(ggplot2)
source(&quot;transform_and_clean_tweets.R&quot;)
</code></pre>
<pre><code class="lang-{r}"># Create our SQLite database
conn &lt;- dbConnect(RSQLite::SQLite(), &quot;Tweet_DB.db&quot;)
</code></pre>
Afterward, we can create a table inside the database to hold the tweets. In our particular case, we are going to store the following variables:
<ul>
<li>Tweet_ID as an INTEGER primary key</li>
<li>User as TEXT</li>
<li>Tweet_Content as TEXT</li>
<li>Date_Created as INTEGER</li>
</ul>
You may wonder, why do I set the dates as integers? This is due to the fact that SQLite doesn&#39;t have a reserved data type for dates and times. Therefore, the dates will be stored as the number of seconds since 1970-01-01.
Let us now proceed to write the table:
<pre><code class="lang-{r}">dbExecute(conn, &quot;CREATE TABLE Tweet_Data(
 Tweet_ID INTEGER PRIMARY KEY,
 User TEXT,
 Tweet_Content TEXT,
 Date_Created INTEGER)&quot;)
</code></pre>
Once you have created the table, you can go to sqlite3.exe and check that is has indeed been created. You can see a screenshot of this below:
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1568658846/table_creation_m9ldw6.png" />
<h2 id="step-2-stream-tweets-about-your-favourite-topics-">Step 2: Stream Tweets About your Favourite Topics!</h2>
Believe it or not, you are done in terms of requirements and infrastructure needed to have a simple, functional Twitter streaming pipeline. What we need to do now is to stream Tweets using the API. It is worth noting that for the purposes of this tutorial, I will be using the standard free API. There are premium paid versions that may suit your Tweet streaming needs better if you conduct research, for instance.
Without further delay, let us start the process of setting up our Twitter listener. The first thing that you will need is to import the rtweet package and input your application&#39;s access tokens and secrets as described in the beginning:
<pre><code class="lang-{r}">token &lt;- create_token(app = &#39;Your_App_Name&#39;,
 consumer_key = &#39;Your_Consumer_Key&#39;,
 consumer_secret = &#39;Your_Consumer_Secret&#39;,
 access_token = &#39;Your_Access_Token&#39;,
 access_secret = &#39;Your_Access_Secret&#39;)
</code></pre>
With your token in place, the next thing is to decide what tweets do you want to stream (i.e., to listen to). The function stream_tweets in the rtweet package provides us with a variety of options to query the Twitter API. For example, you can stream tweets that contain one or more of a set of given hashtags or keywords (up to 400), a small random subset of all publicly available tweets, track the tweets of a group of user IDs or screen (user) names (up to 5000), or gather tweets by geographic location.
For this tutorial, I decided to go ahead and stream tweets containing hashtags related to data science (see list below). You may notice that the format in which I defined the hashtags to stream is a bit odd. However, that is the format required by the stream_tweets function when you want to listen to hashtags or keywords. This format varies if you intend to listen to a given set of users or based on coordinates. For more details, refer to the <a href="https://www.rdocumentation.org/packages/rtweet/versions/0.6.9/topics/stream_tweets">documentation</a>.
<pre><code class="lang-{r}">keys &lt;- &quot;#nlp,#machinelearning,#datascience,#chatbots,#naturallanguageprocessing,#deeplearning&quot;
</code></pre>
With the keywords defined, it is time to define the tweet streaming loop. There are several ways to do this, but this format has worked well for me in the past:
<pre><code class="lang-{r}"># Initialize the streaming hour tally
hour_counter &lt;- 0

# Initialize a while loop that stops when the number of hours you want to stream tweets for is exceeded
while(hour_counter &lt;= 12){
 # Set the stream time to 2 hours each iteration (7200 seconds)
 streamtime &lt;- 7200
 # Create the file name where the 2 hour stream will be stored. Note that the Twitter API outputs a .json file.
 filename &lt;- paste0(&quot;nlp_stream_&quot;,format(Sys.time(),&#39;%d_%m_%Y__%H_%M_%S&#39;),&quot;.json&quot;)
 # Stream Tweets containing the desired keys for the specified amount of time
 stream_tweets(q = keys, timeout = streamtime, file_name = filename)
 # Clean the streamed tweets and select the desired fields
 clean_stream &lt;- transform_and_clean_tweets(filename, remove_rts = TRUE)
 # Append the streamed tweets to the Tweet_Data table in the SQLite database
 dbWriteTable(conn, &quot;Tweet_Data&quot;, clean_stream, append = T)
 # Delete the .json file from this 2-hour stream
 file.remove(filename)
 # Add the hours to the tally
 hour_counter &lt;- hour_counter + 2
}
</code></pre>
In essence, that loop streams as many tweets as possible mentioning any of the hashtags in the key strings in intervals of 2 hours for a total time of 12 hours. Every 2 hours, the Twitter listener creates a .json file in your current working directory with the name specified in the variable filename.
Then, it passes this filename to the transform_and_clean_tweets function that removes retweets if desired, selects the columns we want to keep from all those given by the Twitter API, and normalizes the text contained in the Tweets.
Then, it appends the resulting dataframe to the Tweet_Data table that we created earlier inside our SQLite database. Finally, it adds 2 to the hour counter tally (since the streams last for 2 hours), and it removes the .json file created. This is done since all the data of interest is now in our database, and keeping the .json files can become a storage burden.
Let us know look closer to the transform_and_clean_tweets function in detail:
<pre><code class="lang-{r}">transform_and_clean_tweets &lt;- function(filename, remove_rts = TRUE){

 # Import the normalize_text function
 source(&quot;normalize_text.R&quot;)

 # Parse the .json file given by the Twitter API into an R data frame
 df &lt;- parse_stream(filename)
 # If remove_rst = TRUE, filter out all the retweets from the stream
 if(remove_rts == TRUE){
 df &lt;- filter(df,df$is_retweet == FALSE)
 }
 # Keep only the tweets that are in English
 df &lt;- filter(df, df$lang == &quot;en&quot;)
 # Select the features that you want to keep from the Twitter stream and rename them
 # so the names match those of the columns in the Tweet_Data table in our database
 small_df &lt;- df[,c(&quot;screen_name&quot;,&quot;text&quot;,&quot;created_at&quot;)]
 names(small_df) &lt;- c(&quot;User&quot;,&quot;Tweet_Content&quot;,&quot;Date_Created&quot;)
 # Finally normalize the tweet text
 small_df$Tweet_Content &lt;- sapply(small_df$Tweet_Content, normalize_text)
 # Return the processed data frame
 return(small_df)
}
</code></pre>
As noted, this function is meant to filter retweets if desired, retain the desired features, and normalize the tweet text. In essence, this can be considered the &quot;T&quot; part of the ETL acronym (the transformation). One key component of this function is the process of cleaning the tweet text.
Usually, text data requires a few preprocessing steps before it can be ready for any sort of analysis. In the case of tweets, these steps can include removal of URLs, stop words and mentions, turning the text to lower case, stemming words, etc. However, not all of these steps are necessary all of the time. Nevertheless, for the time being, I will display the normalize_text function that I used to preprocess these tweets below:
<pre><code class="lang-{r}">normalize_text &lt;- function(text){
 # Keep only ASCII characters
 text = iconv(text, &quot;latin1&quot;, &quot;ASCII&quot;, sub=&quot;&quot;)
 # Convert to lower case characters
 text = tolower(text)
 # Remove any HTML tags
 text = gsub(&quot;&lt;.*?&gt;&quot;, &quot; &quot;, text)
 # Remove URLs
 text = gsub(&quot;\\s?(f|ht)(tp)(s?)(://)([^\\.]*)[\\.|/](\\S*)&quot;, &quot;&quot;, text)
 # Keep letters and numbers only
 text = gsub(&quot;[^[:alnum:]]&quot;, &quot; &quot;, text)
 # Remove stop words
 text = removeWords(text,c(&quot;rt&quot;,&quot;gt&quot;,stopwords(&quot;en&quot;)))
 # Remove any extra white space
 text = stripWhitespace(text) 
 text = gsub(&quot;^\\s+|\\s+$&quot;, &quot;&quot;, text) 

 return(text)
}
</code></pre>
Depending on your use case, these steps may be sufficient. As mentioned earlier, you may choose to add more or different steps such as stemming or lemmatization or to keep only letters instead of letters and numbers. I encourage you to experiment and try out different combinations. This can be a great place to practice regex skills.
After all of these steps, the resulting state is a SQLite database populated with all the streamed tweets. You can validate that everything worked correctly by running a couple of simple queries, for instance:
<pre><code class="lang-{r}">data_test &lt;- dbGetQuery(conn, &quot;SELECT * FROM Tweet_Data LIMIT 20&quot;)
unique_rows &lt;- dbGetQuery(conn, &quot;SELECT COUNT() AS Total FROM Tweet_Data&quot;)
kable(data_test)
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570128184/kable_e3qnut.jpg" />
<pre><code class="lang-{r}">print(as.numeric(unique_rows))
</code></pre>
<pre><code>## [1] 1863
</code></pre><h2 id="step-3-analyze">Step 3 Analyze</h2>
When there is a certainty that the ETL process is working as intended, the final step is to gather some insights and analyze the data that you have collected. For example, for the tweets that we have collected, we can try a couple of different things; a wordcloud of terms mentioned in the tweet&#39;s content, and a timeline to visualize when in our 12-hour streaming period did we obtain the most amount of tweets. This list is obviously not a comprehensive overview of the whole array of research that can be conducted using tweets, which can range from sentiment analysis to psychographic studies and beyond.
With that said, lets jump straight to building a nice wordcloud:
<pre><code class="lang-{r}"># Gather all tweets from the database
all_tweets &lt;- dbGetQuery(conn, &quot;SELECT Tweet_ID, Tweet_Content FROM Tweet_Data&quot;)

# Create a term-document matrix and sort the words by frequency
dtm &lt;- TermDocumentMatrix(VCorpus(VectorSource(all_tweets$Tweet_Content)))
dtm_mat &lt;- as.matrix(dtm)
sorted &lt;- sort(rowSums(dtm_mat), decreasing = TRUE)
freq_df &lt;- data.frame(words = names(sorted), freq = sorted)

# Plot the wordcloud
set.seed(42)
wordcloud(words = freq_df$words, freq = freq_df$freq, min.freq = 10,
 max.words=50, random.order=FALSE, rot.per=0.15,
 colors=brewer.pal(8, &quot;RdYlGn&quot;))
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570046250/2_xjdbkd.png" />
Well, what a tremendous surprise! machinelearning and datascience are the most mentioned words in all our tweets—they just happen to be two of the hashtags that we were streaming for. Thus, this result was expected. The other words though are a bit more fun to look at. For example, bigdata and artificialintelligence were not hashtags in our keys, yet they appeared with high frequency, so one could argue that they are talked about often along with the other two. There are other words of interest that we picked up such as python or tensorflow that gives us a little bit more context into the content of the tweets we collected beyond just the hashtags.
Let us now turn to another simple analysis. In our 12-hour stream, at what time did we collect the most tweets? For that, we will gather our integer dates, turn them into a proper format, and then plot the quantity of tweets over time:
<pre><code class="lang-{r}"># Select the dates in which the tweets were created and convert them into UTC date-time format
all_tweets &lt;- dbGetQuery(conn, &quot;SELECT Tweet_ID, Date_Created FROM Tweet_Data&quot;)
all_tweets$Date_Created &lt;- as.POSIXct(all_tweets$Date_Created, origin = &quot;1970-01-01&quot;, tz = &quot;UTC&quot;)

# Group by the day and hour and count the number of tweets that occurred in each bucket
all_tweets_2 &lt;- all_tweets %&gt;%
 mutate(day = day(Date_Created),
 month = month(Date_Created, label = TRUE),
 hour = hour(Date_Created)) %&gt;%
 mutate(day_hour = paste(month,&quot;-&quot;,day,&quot;-&quot;,hour, sep = &quot;&quot;)) %&gt;%
 group_by(day_hour) %&gt;%
 tally()

# Simple line ggplot
ggplot(all_tweets_2, aes(x = day_hour, y = n)) +
 geom_line(aes(group = 1)) +
 theme_minimal() +
 ggtitle(&quot;Tweet Freqeuncy During the 12-h Streming Period&quot;)+
 ylab(&quot;Tweet Count&quot;)+
 xlab(&quot;Month-Day-Hour&quot;)
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1570046251/3_fgkhum.png" />
Sweet! Now we can see that the largest amount of unique tweets that we obtain came from the 2nd of September between 8:00 PM UTC and 8:59PM UTC (This appears in military time as 20).
<h2 id="conclusion">Conclusion</h2>
Congratulations! Now you know how to build a simple ETL pipeline in R. The two analyses we conducted represent very basic analyses conducted using Twitter data. However, as mentioned previously, there are lots of things to do as long as you build a robust pipeline to bring in the data. This aspect was the main focus of this tutorial.
With that said, this tutorial only showed a very small scale case study to walk over the process of building ETL pipelines for Twitter data. Building robust and scalable ETL pipelines for a whole enterprise is a complicated endeavor that requires extensive computing resources and knowledge, especially when big data is involved.
I encourage you to do further research and try to build your own small scale pipelines, which could involve building one in Python$^1$. Maybe, even go ahead and try some big data projects. For example, DataCamp already has some courses like this <a href="https://www.datacamp.com/courses/big-data-fundamentals-via-pyspark">Big Data Fundamentals via PySpark</a> course where they go over big data using tools such as PySpark that can further your knowledge of this field.
If you would like to learn about data engineering, take DataCamp&#39;s <a href="https://www.datacamp.com/courses/introduction-to-data-engineering">Introduction to Data Engineering</a> course.
<h2 id="references">References</h2>
<ol>
<li>Foley, D. (2019, May 11). Streaming Twitter Data into a MySQL Database. Retrieved from <a href="https://towardsdatascience.com/streaming-twitter-data-into-a-mysql-database-d62a02b050d6">https://towardsdatascience.com/streaming-twitter-data-into-a-mysql-database-d62a02b050d6</a></li>
</ol>

https://www.datacamp.com/community/tutorials/building-a-tweet-etl-pipeline-using-r

Learn how to transform and load (ETL)  a data pipeline from scratch using R and SQLite to gather tweets in real-time and store them for future analyses.

community

datacamp

data engineering

News (10)

Difference between Data Science and Big Data

From Local Machine to Dask Cluster with Terraform

Tutorials (2)

What is Data Engineering?

Building Data Pipelines using R

Cheat Sheets (0)

Open Courses (0)

Podcast (0)

Blog Posts (22)

An Analysis of the Top 7 Trending Data Science Careers to Pursue

Tech Thoughts (0)