Skip to main content

Introduction to Data Engineering

Learn about the world of data engineering with an overview of all its relevant topics and tools!

Start Course for Free
4 Hours15 Videos57 Exercises68,169 Learners
4100 XP

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).

Loved by learners at thousands of companies

Course Description

Have you heard people talk about data engineers and wonder what it is they do? Do you know what data engineers do but you're not sure how to become one yourself? This course is the perfect introduction. It touches upon all things you need to know to streamline your data processing. This introductory course will give you enough context to start exploring the world of data engineering. It's perfect for people who work at a company with several data sources and don't have a clear idea of how to use all those data sources in a scalable way. Be the first one to introduce these techniques to your company and become the company star employee.

  1. 1

    Introduction to Data Engineering

    Free

    In this first chapter, you will be exposed to the world of data engineering! Explore the differences between a data engineer and a data scientist, get an overview of the various tools data engineers use and expand your understanding of how cloud technology plays a role in data engineering.

    Play Chapter Now
    What is data engineering?
    50 xp
    Tasks of the data engineer
    50 xp
    Data engineer or data scientist?
    100 xp
    Data engineering problems
    50 xp
    Tools of the data engineer
    50 xp
    Kinds of databases
    50 xp
    Processing tasks
    50 xp
    Scheduling tools
    50 xp
    Cloud providers
    50 xp
    Why cloud computing?
    50 xp
    Big players in cloud computing
    100 xp
    Cloud services
    100 xp
  2. 2

    Data engineering toolbox

    Now that you know the primary differences between a data engineer and a data scientist, get ready to explore the data engineer's toolbox! Learn in detail about different types of databases data engineers use, how parallel computing is a cornerstone of the data engineer's toolkit, and how to schedule data processing jobs using scheduling frameworks.

    Play Chapter Now
    Databases
    50 xp
    SQL vs NoSQL
    100 xp
    The database schema
    100 xp
    Joining on relations
    100 xp
    Star schema diagram
    50 xp
    What is parallel computing
    50 xp
    Why parallel computing?
    50 xp
    From task to subtasks
    100 xp
    Using a DataFrame
    100 xp
    Parallel computation frameworks
    50 xp
    Spark, Hadoop and Hive
    100 xp
    A PySpark groupby
    100 xp
    Running PySpark files
    50 xp
    Workflow scheduling frameworks
    50 xp
    Airflow, Luigi and cron
    50 xp
    Airflow DAGs
    100 xp
  3. 3

    Extract, Transform and Load (ETL)

    Having been exposed to the toolbox of data engineers, it's now time to jump into the bread and butter of a data engineer's workflow! With ETL, you will learn how to extract raw data from various sources, transform this raw data into actionable insights, and load it into relevant databases ready for consumption!

    Play Chapter Now
    Extract
    50 xp
    Data sources
    50 xp
    Fetch from an API
    100 xp
    Read from a database
    100 xp
    Transform
    50 xp
    Splitting the rental rate
    100 xp
    Prepare for transformations
    50 xp
    Joining with ratings
    100 xp
    Loading
    50 xp
    OLAP or OLTP
    50 xp
    Writing to a file
    100 xp
    Load into Postgres
    100 xp
    Putting it all together
    50 xp
    Defining a DAG
    100 xp
    Setting up Airflow
    50 xp
    Interpreting the DAG
    50 xp
  4. 4

    Case Study: DataCamp

    Cap off all that you've learned in the previous three chapters by completing a real-world data engineering use case from DataCamp! You will perform and schedule an ETL process that transforms raw course rating data, into actionable course recommendations for DataCamp students!

    Play Chapter Now
    Course ratings
    50 xp
    Exploring the schema
    50 xp
    Querying the table
    100 xp
    Average rating per course
    100 xp
    From ratings to recommendations
    50 xp
    Filter out corrupt data
    100 xp
    Using the recommender transformation
    100 xp
    Scheduling daily jobs
    50 xp
    The target table
    100 xp
    Defining the DAG
    100 xp
    Enable the DAG
    50 xp
    Querying the recommendations
    100 xp
    Congratulations
    50 xp

In the following tracks

Data Engineer

Datasets

datacamp_application.sql

Collaborators

Adel Nehme

Prerequisites

Intermediate PythonIntroduction to SQL
Vincent Vankrunkelsven Headshot

Vincent Vankrunkelsven

Data and Software Engineer @DataCamp

Vincent has a Master's degree in Computer Science and has several years of experience scaling up the DataCamp's platform as a Software Engineer. He experienced first-hand the difficulties that come with building scalable data products. This made him passionate about teaching people how to do tackle these problems the right way.
See More

What do other learners have to say?

I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.

Devon Edwards Joseph
Lloyds Banking Group

DataCamp is the top resource I recommend for learning data science.

Louis Maiden
Harvard Business School

DataCamp is by far my favorite website to learn from.

Ronald Bowers
Decision Science Analytics, USAA

Join over 9 million learners and start Introduction to Data Engineering today!

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).