Skip to main content

Introduction to Data Engineering

4.4+
20 reviews
Intermediate

Learn about the world of data engineering in this short course, covering tools and topics like ETL and cloud computing.

Start Course for Free
4 Hours15 Videos57 Exercises
100,687 Learners

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.

Loved by learners at thousands of companies

Course Description

Get Started in Data Engineering

Are you curious about a career in data engineering but don’t know where to start? Or perhaps you want more information on what data engineers do before you take the next steps? This four-hour course is an introduction to data engineering and the core concepts, techniques, and tools you need to understand to do the job.

Learn Data Engineering Concepts and Techniques

You’ll start by learning the differences between a data engineer and a data scientist (and how they work together) before finding out more about the tools of the trade, specifically talking about cloud computing and parallel computing. By the end of the second chapter, you’ll understand the applications of SQL and NoSQL, using DataFrames, and why parallel computing is so important.

Perform ETL in Hands-on Exercises

The ETL process is core to a data engineer’s workflow. You will learn how data is extracted, transformed, and loaded to get it ready for analysis and generating insights. At the end of the course, you’ll put all this knowledge into practice by performing and scheduling an ETL process yourself using real-world data.

Our exercises and interactive tests allow you to review and cement your new knowledge, so you’re confident discussing and applying it once you’ve received your Statement of Accomplishment.

This introductory course is part of a data engineering Track, which offers you pathways to improve your understanding of data engineering and a clear set of next steps to becoming a professional data engineer.
  1. 1

    Introduction to Data Engineering

    Free

    In this first chapter, you will be exposed to the world of data engineering! Explore the differences between a data engineer and a data scientist, get an overview of the various tools data engineers use and expand your understanding of how cloud technology plays a role in data engineering.

    Play Chapter Now
    What is data engineering?
    50 xp
    Tasks of the data engineer
    50 xp
    Data engineer or data scientist?
    100 xp
    Data engineering problems
    50 xp
    Tools of the data engineer
    50 xp
    Kinds of databases
    50 xp
    Processing tasks
    50 xp
    Scheduling tools
    50 xp
    Cloud providers
    50 xp
    Why cloud computing?
    50 xp
    Big players in cloud computing
    100 xp
    Cloud services
    100 xp
  2. 2

    Data engineering toolbox

    Now that you know the primary differences between a data engineer and a data scientist, get ready to explore the data engineer's toolbox! Learn in detail about different types of databases data engineers use, how parallel computing is a cornerstone of the data engineer's toolkit, and how to schedule data processing jobs using scheduling frameworks.

    Play Chapter Now
    Databases
    50 xp
    SQL vs NoSQL
    100 xp
    The database schema
    100 xp
    Joining on relations
    100 xp
    Star schema diagram
    50 xp
    What is parallel computing
    50 xp
    Why parallel computing?
    50 xp
    From task to subtasks
    100 xp
    Using a DataFrame
    100 xp
    Parallel computation frameworks
    50 xp
    Spark, Hadoop and Hive
    100 xp
    A PySpark groupby
    100 xp
    Running PySpark files
    50 xp
    Workflow scheduling frameworks
    50 xp
    Airflow, Luigi and cron
    50 xp
    Airflow DAGs
    100 xp
  3. 3

    Extract, Transform and Load (ETL)

    Having been exposed to the toolbox of data engineers, it's now time to jump into the bread and butter of a data engineer's workflow! With ETL, you will learn how to extract raw data from various sources, transform this raw data into actionable insights, and load it into relevant databases ready for consumption!

    Play Chapter Now
    Extract
    50 xp
    Data sources
    50 xp
    Fetch from an API
    100 xp
    Read from a database
    100 xp
    Transform
    50 xp
    Splitting the rental rate
    100 xp
    Prepare for transformations
    50 xp
    Joining with ratings
    100 xp
    Loading
    50 xp
    OLAP or OLTP
    50 xp
    Writing to a file
    100 xp
    Load into Postgres
    100 xp
    Putting it all together
    50 xp
    Defining a DAG
    100 xp
    Setting up Airflow
    50 xp
    Interpreting the DAG
    50 xp
  4. 4

    Case Study: DataCamp

    Cap off all that you've learned in the previous three chapters by completing a real-world data engineering use case from DataCamp! You will perform and schedule an ETL process that transforms raw course rating data, into actionable course recommendations for DataCamp students!

    Play Chapter Now
    Course ratings
    50 xp
    Exploring the schema
    50 xp
    Querying the table
    100 xp
    Average rating per course
    100 xp
    From ratings to recommendations
    50 xp
    Filter out corrupt data
    100 xp
    Using the recommender transformation
    100 xp
    Scheduling daily jobs
    50 xp
    The target table
    100 xp
    Defining the DAG
    100 xp
    Enable the DAG
    50 xp
    Querying the recommendations
    100 xp
    Congratulations
    50 xp

Datasets

datacamp_application.sql

Collaborators

Collaborator's avatar
Adel Nehme

Prerequisites

Intermediate PythonIntermediate SQL
Vincent Vankrunkelsven HeadshotVincent Vankrunkelsven

Data and Software Engineer @DataCamp

Vincent has a Master's degree in Computer Science and has several years of experience scaling up the DataCamp's platform as a Software Engineer. He experienced first-hand the difficulties that come with building scalable data products. This made him passionate about teaching people how to do tackle these problems the right way.
See More

Don’t just take our word for it

*4.4
from 20 reviews
60%
25%
10%
5%
0%
Sort by
  • Ridhouane M.
    3 months

    Good overview

  • Adibya R.
    6 months

    Cool

  • Manh D.
    7 months

    Great introduction to Data Engineering concepts! I enjoyed it a lot!

  • Vatsala N.
    7 months

    Fantastic course.

  • Tazeen K.
    8 months

    i like it

"Good overview"

Ridhouane M.

"Cool"

Adibya R.

"Great introduction to Data Engineering concepts! I enjoyed it a lot!"

Manh D.

FAQs

Join over 12 million learners and start Introduction to Data Engineering today!

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.