Skip to main content

What does Data Engineering mean?

In this blog, you will learn what data engineering entails along with learning about our future data engineering course offerings.
Sep 25, 2018  · 4 min read

Due to popular demand, DataCamp is getting ready to build a Data Engineering track. Like most terms in the ever-expanding Data Science Universe, there’s a lot of ambiguity around the definition of “Data Engineering.” Some Data Engineers do a lot of reporting and dashboarding. Some spend most of their time working on data pipelines. Others take Python code from Data Scientists and optimize it to run in Java or C.

In order to start course creation, we’ll need to pick a single definition of “Data Engineer” to work from. After much deliberation and thought, we chose to paraphrase the American television show “Law and Order”:

Data Engineers vs Data Analysts vs Data Scientists

In the world of Data Science, the data are represented by three separate yet equally important professions:

  • The Data Engineers, who use programming languages to ensure clean, reliable, and performative access to data and databases
  • The Data Analysts, who use programming languages, spreadsheets, and business intelligence tools to describe and categorize the data that currently exist
  • The Data Scientists, who use algorithms to predict future data based on existing information

Examples

For example, imagine that a company sells many different types of sofas on their website. Each time a visitor to the website clicks on a particular sofa, a new piece of data is created. A Data Engineer would define how to collect this data, what types of metadata should be appended to each click event, and how to store the data in an easy-to-access format. A Data Analyst would create visualizations to help sales and marketing track who is buying each sofa and how much money the company is making. A Data Scientist would take the data on which customers bought each sofa and use it to predict the perfect sofa for each new visitor to the website.

What is a data engineer?

For many organizations, data engineers are the first hires on a data team. Before collected data can be analyzed and leveraged with predictive methods, it needs to be organized and cleaned. Data Engineers begins this process by making a list of what data is stored, called a data schema. Next, they need to pick a reliable, easily accessible location, called a data warehouse, for storing the data. Examples of data warehousing systems include Amazon Redshift or Google Cloud. Finally, Data Engineers create ETL (Extract, Transform and Load) processes to make sure that the data gets into the data warehouse.

As an organization grows, Data Engineers are responsible for integrating new data sources into the data ecosystem, and sending the stored data into different analysis tools. When the data warehouse becomes very large, Data Engineers have to find new ways of making analyses performative, such as parallelizing analysis or creating smaller subsets for fast querying.

The relationship between the three professions

Within the Data Science universe, there is always overlap between the three professions. Data Engineers are often responsible for simple Data Analysis projects or for transforming algorithms written by Data Scientists into more robust formats that can be run in parallel. Data Analysts and Data Scientists need to learn basic Data Engineering skills, especially if they’re working in an early-stage startup where engineering resources are scarce.

Learn data engineering with DataCamp!

At DataCamp, we’re excited to build out our Data Engineering course offerings. We know what we want to teach, and we’re starting to recruit instructors to design these courses. If you’re interested, check out our application and the list of courses we are currently prioritizing.

Topics
Related

blog

What is Data Engineering?

Learn what data engineering is, what is the difference between data science and data engineering, the scope in the field, and how to learn data engineering.
Çağlar Uslu's photo

Çağlar Uslu

8 min

blog

Using Data Science to Explore Software Development

What can data science mean for software development? In this blog post, you'll discover some interesting case studies of data science in software engineering!
Greg Wilson's photo

Greg Wilson

9 min

Data Science Concept

blog

What is Data Science? Understanding Data Science from Scratch

In this blog, we delve into what data science is and explore the answers to frequently asked questions about the various aspects of data science.
DataCamp Team's photo

DataCamp Team

16 min

blog

What Does a Data Engineer Do?

Curious about what a data engineer does? We break down the different data engineer roles & career paths and look at a typical data engineering project.
Joleen Bothma's photo

Joleen Bothma

9 min

blog

5 Essential Data Engineering Skills

Discover the data engineering skills you need to thrive in the industry. Find out about the roles and responsibilities of a data engineer, and how you can develop your own skills.
Joleen Bothma's photo

Joleen Bothma

11 min

tutorial

An Introduction to Data Pipelines for Aspiring Data Professionals

This tutorial covers the basics of data pipelines and terminology for aspiring data professionals, including pipeline uses, common technology, and tips for pipeline building.
Amberle McKee's photo

Amberle McKee

22 min

See MoreSee More