In 2019, the average salary for data engineers overtook data scientists. How did this happen? Companies wanting to find the gold within their data realized it wasn’t possible if they hadn’t yet built the mine. Data engineers lay the foundations that make data science possible.
 
In this course, you’ll learn about a data engineer’s core responsibilities, how they differ from data scientists, and facilitate the flow of data through an organization. Through hands-on exercises you’ll follow Spotflix, a fictional music streaming company, to understand how their data engineers collect, clean, and catalog their data. By the end of the course, you’ll understand what your company's data engineers do, be ready to have a conversation with a data engineer, and have a solid foundation to start your own data engineer journey.

Discover how data engineers lay the groundwork that makes data science possible. No coding involved!

Data Engineering for Everyone

As a Data Scientist, the majority of your time should be spent gleaning actionable insights from data -- not waiting for your code to finish running. Writing efficient Python code can help reduce runtime and save computational resources, ultimately freeing you up to do the things you love as a Data Scientist. In this course, you'll learn how to use Python's built-in data structures, functions, and modules to write cleaner, faster, and more efficient code. We'll explore how to time and profile code in order to find bottlenecks. Then, you'll practice eliminating these bottlenecks, and other bad design patterns, using Python's Standard Library, NumPy, and pandas. After completing this course, you'll have the necessary tools to start writing efficient Python code!

Learn to write efficient code that executes quickly and allocates resources skillfully to avoid unnecessary overhead.

Writing Efficient Python Code

You’ve already used SQL to query data from databases. But did you know that there's a lot more you can do with databases? You can model different phenomena in your data, as well as the relationships between them. This gives your data structure and consistency, which results in better data quality. In this course, you'll experience this firsthand by working with a real-life dataset that was used to investigate questionable university affiliations. Column by column, table by table, you'll get to unlock and admire the full potential of databases. You'll learn how to create tables and specify their relationships, as well as how to enforce data integrity. You'll also discover other unique features of database systems, such as constraints.

Learn how to create one of the most efficient ways of storing data - relational databases! 

Learn how to create tables and specify their relationships in SQL, as well as how to enforce data integrity and other unique features of database systems.

Introduction to Relational Databases in SQL

In this course, you'll learn how to use Spark from Python! Spark is a tool for doing parallel computation with large datasets and it integrates well with Python. PySpark is the Python package that makes the magic happen. You'll use this package to work with data about flights from Portland and Seattle. You'll learn to wrangle this data and build a whole machine learning pipeline to predict whether or not flights will be delayed. Get ready to put some Spark in your Python code and dive into the world of high-performance machine learning!

Learn to implement distributed data management and machine learning in Spark using the PySpark package.

Learn to wrangle data and build a machine learning pipeline to make predictions with  PySpark Python package. Practice your skills with real-world data.

Introduction to PySpark

Object-oriented programming (OOP) is a widely used programming paradigm that reduces development times—making it easier to read, reuse, and maintain your code. OOP shifts the focus from thinking about code as a sequence of actions to looking at your program as a collection of objects that interact with each other. In this course, you’ll learn how to create classes, which act as the blueprints for every object in Python. You’ll then leverage principles called inheritance and polymorphism to reuse and optimize code. Dive in and learn how to create beautiful code that’s clean and efficient!

Dive in and learn how to create classes and leverage inheritance and polymorphism to reuse and optimize code.

Object-Oriented Programming in Python

You've done your analysis, built your report, and trained a model. What's next? Well, if you want to deploy your model into production, your code will need to be more reliable than exploratory scripts in a Jupyter notebook. Writing Functions in Python will give you a strong foundation in writing complex and beautiful functions so that you can contribute research and engineering skills to your team. You'll learn useful tricks, like how to write context managers and decorators. You'll also learn best practices around how to write maintainable reusable functions with good documentation. They say that people who can do good research and write high-quality code are unicorns. Take this course and discover the magic!

Learn to use best practices to write maintainable, reusable, complex functions with good documentation.

Writing Functions in Python

Have you heard people talk about data engineers and wonder what it is they do? Do you know what data engineers do but you're not sure how to become one yourself? This course is the perfect introduction. It touches upon all things you need to know to streamline your data processing. This introductory course will give you enough context to start exploring the world of data engineering. It's perfect for people who work at a company with several data sources and don't have a clear idea of how to use all those data sources in a scalable way. Be the first one to introduce these techniques to your company and become the company star employee.

Learn about the world of data engineering with an overview of all its relevant topics and tools!

Introduction to Data Engineering

The Unix command line has survived and thrived for almost 50 years because it lets people do complex things with just a few keystrokes. Sometimes called "the universal glue of programming," it helps users combine existing programs in new ways, automate repetitive tasks, and run programs on clusters and clouds that may be halfway around the world. This course will introduce its key elements and show you how to use them efficiently.

The Unix command line helps users combine existing programs in new ways, automate repetitive tasks, and run programs on clusters and clouds. 

Discover the Unix command line and how to use it to combine existing programs, automate repetitive tasks, and run programs on clusters and clouds in Shell.

Introduction to Shell

A good database design is crucial for a high-performance application. Just like you wouldn't start building a house without the benefit of a blueprint, you need to think about how your data will be stored beforehand. Taking the time to design a database saves time and frustration later on, and a well-designed database ensures ease of access and retrieval of information. While choosing a design, a lot of considerations have to be accounted for. In this course, you'll learn how to process, store, and organize data in an efficient way. You'll see how to structure data through normalization and present your data with views. Finally, you'll learn how to manage your database and all of this will be done on a variety of datasets from book sales, car rentals, to music reviews.

Learn to design databases using star and snoflake schemas, and OLTP and OLAP. You will gain essential skills for creating and managing databases in SQL.

Database Design

Before you can analyze data, you first have to acquire it. This course teaches you how to build pipelines to import data kept in common storage formats. You’ll use pandas, a major Python library for analytics, to get data from a variety of sources, from spreadsheets of survey responses, to a database of public service requests, to an API for a popular review site. Along the way, you’ll learn how to fine-tune imports to get only what you need and to address issues like incorrect data types. Finally, you’ll assemble a custom dataset from a mix of sources.

Learn to acquire data from common file formats and systems such as CSV files, spreadsheets, JSON, SQL databases, and APIs.

Streamlined Data Ingestion with pandas

There's been a lot of buzz about Big Data over the past few years, and it's finally become mainstream for many  companies. But what is this Big Data? This course covers the fundamentals of Big Data via PySpark. Spark is a "lightning fast cluster computing" framework for Big Data. It provides a general data processing platform engine and lets you run programs up to 100x faster in memory, or 10x faster on disk,  than Hadoop. You’ll use PySpark, a Python package for Spark programming and its powerful, higher-level libraries such as SparkSQL, MLlib (for machine learning), etc. You will explore the works of William Shakespeare, analyze Fifa 2018 data and perform clustering on genomic datasets. At the end of this course, you will have gained an in-depth understanding of PySpark and its application to general Big Data analysis.

Learn the fundamentals of working with big data with PySpark.

Learn to conduct Big Data analysis using the Python package for Spark programming, PySpark, as well as higher level libraries like SparkSQL and MLlib.

Big Data Fundamentals with PySpark

Bash is a concise, superfast, and robust scripting language for data and file manipulation. It’s a vital skill for building analytics pipelines in the cloud, favored by Linux users to work with data stored across multiple files.

In this course, we’ll guide you through the basics of Bash scripting. We begin with an introduction to Bash script structures, including inputting arguments and outputting results. You’ll then work through data structures, such as variables and arrays, and control statements, including loops and conditionals. You’ll then put what you’ve learned into practice, by creating your own Bash functions and scheduling automated scripts to run like clockwork with cron.

Bash scripting allows you to build analytics pipelines in the cloud and work with data stored across multiple files.

Learn how you can use Bash scripting to build automated scripts, create analytics pipelines in the cloud, and work with data stored across multiple files.

Introduction to Bash Scripting

Every data science project needs unit testing. It comes with huge benefits - saving a lot of development and maintenance time, improving documentation, increasing end-user trust and reducing downtime of productive systems. As a result, unit testing has become a must-have skill in the industry, used by almost every company. This course teaches unit testing in Python using the most popular testing framework pytest. By the end of this course, you will have written a complete test suite for a data science project. In the process, you will learn to write unit tests for data preprocessors, models and visualizations, interpret test results and fix any buggy code. You will also learn advanced concepts like TDD, test organization, fixtures and mocking so that you can test your own data science projects properly.

Learn how to write unit tests for your Data Science projects in Python using pytest.

Unit Testing for Data Science in Python

Delivering data on a schedule can be a manual process. You write scripts, add complex cron tasks, and try various ways to meet an ever-changing set of requirements—and it’s even trickier to manage everything when working with teammates. Airflow can remove this headache by adding scheduling, error handling, and reporting to your workflows. In this course, you’ll master the basics of Airflow and learn how to implement complex data engineering pipelines in production. You'll also learn how to use Directed Acyclic Graphs (DAGs), automate data engineering workflows, and implement data engineering tasks in an easy and repeatable fashion—helping you to maintain your sanity.

Learn how to implement and schedule data engineering workflows.

Learn to implement and schedule data engineering workflows by adding scheduling, error handling, and reporting using Airflow and Airflow DAGs in Python. 

Introduction to Airflow in Python

Working with data is tricky - working with millions or even billions of rows is worse.
Did you receive some data processing code written on a laptop with fairly pristine data?
Chances are you’ve probably been put in charge of moving a basic data process from prototype to production.
You may have worked with real world datasets, with missing fields, bizarre formatting, and orders of magnitude more data. Even if this is all new to you, this course helps you learn what’s needed to prepare data processes using Python with Apache Spark.
You’ll learn terminology, methods, and some best practices to create a performant, maintainable, and understandable data processing platform.

Learn how to clean data with Apache Spark in Python.

Learn how to use PySpark to clean your data in Python with DataFrames and data pipelines. You'll also learn why clean data is so important for analysis

Cleaning Data with PySpark

Get an introduction to the programming language Scala. You'll learn why and how companies like Netflix, Airbnb, and Morgan Stanley are choosing Scala for large-scale applications and data engineering infrastructure. You'll learn the basics of the language, including syntax and style, focusing on the most commonly used features in the Scala standard library. You'll learn by writing code for a real program that plays a computer version of the popular card game Twenty-One. You’ll get a taste of the value of a hybrid object-oriented and functional programming language, of which Scala is the foremost example. We recommend this course for learners with intermediate-level programming experience, which can be acquired in the listed prerequisites.

Begin your journey with Scala, a popular language for scalable applications and data engineering infrastructure.

Learn the basics of the Sacla language, including syntax and style, focusing on the most commonly used features in the Scala standard library.

Introduction to Scala

A mission critical assignment is depending on your SQL coding skills. You’ve been given some code to fix.  It is giving the results you need but it’s running too slow, and it’s poorly formatted making it hard to read.  The deadline is tomorrow.  You’ll need to reformat the code and try different methods to improve performance.  The pressure is on!!!  In this course we’ll be using SQL on real world datasets, from sports and geoscience, to look at good coding practices and different ways how we can can improve the performance of queries to achieve the same outcome.

In this course, students will learn to write queries that are both efficient and easy to read and understand.

Discover how you can reformat code and try different methods to improve the performance of queries in SQL. Then apply your skills to real-world datasets.

Improving Query Performance in SQL Server

We live in a busy world with tight deadlines. As a result, we fall back on what is familiar and easy, favoring GUI interfaces like Anaconda and RStudio. However, taking the time to learn data analysis on the command line is a great long-term investment because it makes us stronger and more productive data people.
 
 
In this course, we will take a practical approach to learn simple, powerful, and data-specific command-line skills. Using publicly available Spotify datasets, we will learn how to download, process, clean, and transform data, all via the command line. We will also learn advanced techniques such as command-line based SQL database operations. Finally, we will combine the powers of command line and Python to build a data pipeline for automating a predictive model.

Learn powerful command-line skills to download, process,  and transform data, including machine learning pipeline.

Learn powerful command-line skills to download, process, and transform data, including a machine learning pipeline.

Data Processing in Shell

It is critical to know how to handle errors and manage transactions when programming SQL scripts. Unhandled errors can be very harmful and can cause unexpected situations, such as inconsistent data in your database, or even worse, errors can lead you to make wrong business decisions.
In this course, you will learn how to handle errors and discover how to manage transactions in case of an error. Additionally, you will study what happens when two or more people interact at the same time with the same data. You will practice all these concepts using two datasets, one of them based on bank accounts and the other one on an electric bike store.

Learn to write scripts that will catch and handle errors and control for multiple operations happening at once.

Transactions and Error-Handling in SQL Server

Transactions and Error Handling in SQL Server

In any data-driven company, you will undoubtedly cross paths with data engineers. Among other things, they facilitate some of your work by making data readily available to everyone within the organization, and possibly in bringing machine learning models into production. One way to speed up this process is through building an understanding of what it means to bring processes into production and what features are of high-grade code. In this course, we’ll be looking at various data pipelines the data engineer is building, and how some of the tools he or she is using can help you in getting your models into production or run repetitive tasks consistently and efficiently.
 
In this course, we illustrate common elements of data engineering pipelines. In Chapter 1, you will learn how to ingest data. Chapter 2 will go one step further with cleaning and transforming data. In Chapter 3, you will learn how to safely deploy code. Finally, in Chapter 4 you will schedule complex dependencies between applications.
 
Building Data Engineering Pipelines covers new technologies and material, so we recommend that you have a strong understanding of the prerequisites to get the most out of this course.

Learn how to build data engineering pipelines in Python.

Learn how to build data engineering pipelines in Python by cleaning and transforming data before safely deploying code and scheduling complex dependencies.

Building Data Engineering Pipelines in Python

What if you were no longer constrained by the capabilities of your laptop? What if you could get an SMS when a city garbage truck camera spots a missing a cat? This is all possible with cloud technology. This course will teach you how to integrate Amazon Web Services (AWS) into your data workflow. You’ll learn how to upload data to S3, AWS cloud storage. You’ll use triggers from your analysis to send text messages with AWS SNS. You will use Rekognition to detect objects in an image. And you will use Comprehend to decide if a piece of feedback is negative. By the time you’re done, you will learn how to build a pipeline, subscribe people to it, and send them text messages when an image contains a cat!

Learn about AWS Boto and harnessing cloud technology to optimize your data workflow. 

Learn how to integrate Amazon Web Services (AWS) into your data workflow, upload data to S3 cloud storage, and use Rekognition to detect parts of an image.

Introduction to AWS Boto in Python

Auditing your SQL Server database and keeping data integrity can be a challenging task for DBAs and database developers. SQL Server triggers are special types of stored procedures designed to help you achieve consistency and integrity of your database. This course will teach you how to work with triggers and use them in real-life examples. Specifically, you will learn about the use cases and limitations of triggers and get practice designing and implementing them. You will also learn to optimize triggers to fit your specific needs.

Learn how to design and implement triggers in SQL Server using real-world examples.

Building and Optimizing Triggers in SQL Server

MongoDB is a tool to explore data structured as you see fit. As a NoSQL database, it doesn't follow the strict relational format imposed by SQL. By providing capabilities that typically require adding layers to SQL, it collapses complexity. With dynamic schema, you can handle vastly different data together and consolidate analytics. The flexibility of MongoDB empowers you to keep improving and fix issues as your requirements evolve. In this course, you will learn the MongoDB language and apply it to search and analytics. Working with unprocessed data from the official nobelprize.org API, you will explore and answer questions about Nobel Laureates and prizes.

Learn to manipulate and analyze flexibly structured data with MongoDB.

Introduction to Using MongoDB for Data Science in Python

Introduction to MongoDB in Python

You're familiar with SQL, and have heard great things about Apache Spark. Then this course is for you! Apache Spark is a computing framework for processing big data. Spark SQL is a component of Apache Spark that works with tabular data. Window functions are an advanced feature of SQL that take Spark to a new level of usefulness.  You will use Spark SQL to analyze time series.  You will extract the most common sequences of words from a text document. You will create feature sets from natural language text and use them to predict the last word in a sentence using logistic regression. Spark combines the power of distributed computing with the ease of use of Python and SQL.

The course uses a natural language text dataset that is easy to understand. Sentences are sequences of words. Window functions are very suitable for manipulating sequence data. The same techniques taught here can be applied to sequences of song identifiers, video ids, or podcast ids.  Exercises include discovering frequent word sequences, and converting word sequences into machine learning feature set data for training a text classifier.

Learn how to manipulate data and create machine learning feature sets in Spark using SQL in Python.

Learn how to use Spark SQL to analyze time series, extract common sequences of words, create feature sets from text to make predictions, and much more.

Introduction to Spark SQL in Python

Did you know that data scientists and data analysts spend a large amount of time cleaning data before they can analyze it? This is because real-world data is messy. To help you navigate messy data this course teaches you how to clean data stored in an SQL Server database. You’ll learn how to solve common problems such as how to clean messy strings, deal with empty values, compare the similarity between strings, and much more. You’ll get hands-on with all these tasks using a wide range of interesting and messy datasets, including monthly airline flights by airport, TV series and paper shop sales. Are you ready to get your hands messy?

Develop the skills you need to clean raw data and transform it into accurate insights.

Develop the skills you need to go from raw data to awesome insights as quickly and accurately as possible.

Cleaning Data in SQL Server Databases

In this track, you’ll discover how to build an effective data architecture, streamline data processing, and maintain large-scale data systems. In addition to working with Python, you’ll also grow your language skills as you work with Shell, SQL, and Scala, to create data engineering pipelines, automate common file system tasks, and build a high-performance database.
 
Through hands-on exercises, you’ll add cloud and big data tools such as AWS Boto, PySpark, Spark SQL, and MongoDB, to your data engineering toolkit to help you create and query databases, wrangle data, and configure schedules to run your pipelines. By the end of this track, you’ll have mastered the critical database, scripting, and process skills you need to progress your career.
 
Please note this track assumes a fundamental knowledge of <a href="https://learn.datacamp.com/skill-tracks/python-programming">Python</a> and <a href="https://www.datacamp.com/courses/intermediate-sql">SQL</a>.

Gain the in-demand data engineering skills businesses are looking for and learn how to efficiently ingest, manage, and warehouse data.

with Python

Data Engineer

Python is a general-purpose programming language that is becoming ever more popular for data science. Companies worldwide are using Python to harvest insights from their data and gain a competitive edge. Unlike other Python tutorials, this course focuses on Python specifically for data science. In our Introduction to Python course, you’ll learn about powerful ways to store and manipulate data, and helpful data science tools to begin conducting your own analyses. Start DataCamp’s online Python curriculum now.

Master the basics of data analysis in Python. Expand your skillset by learning scientific computing with NumPy.

Discover how to use Python for data science, learning about the ways to store and manipulate data in the environment to begin conducting your own analyses.

Introduction to Python

Understanding the basic principles of finance is essential for making important financial decisions ranging from taking out a student loan to constructing an investment portfolio. Combining basic financial knowledge with Python will allow you to construct some very powerful tools. You'll come out of this course understanding the time value of money, how to compare potential projects and how to make rational, data-driven financial decisions.

Using Python and NumPy, learn the most fundamental financial concepts. 

Learn about the time value of money, how to compare potential projects, and how to make rational, data-driven, financial decisions with Python and Numpy.

Introduction to Financial Concepts in Python

Experimental design is a crucial part of data analysis in any field, whether you work in business, health or tech. If you want to use data to answer a question, you need to design an experiment! In this course you will learn about basic experimental design, including block and factorial designs, and commonly used statistical tests, such as the t-tests and ANOVAs. You will use built-in R data and real world datasets including the CDC NHANES survey, SAT Scores from NY Public Schools, and Lending Club Loan Data. Following the course, you will be able to design and analyze your own experiments!

In this course you'll learn about basic experimental design, a crucial part of any data analysis.

Learn about basic experimental design, including block and factorial designs, and commonly used statistical tests, such as the t-tests and ANOVAs in R.

Experimental Design in R

Data scientists can experience huge benefits by learning concepts from the field of software engineering, allowing them to more easily reutilize their code and share it with collaborators. In this course, you'll learn all about the important ideas of modularity, documentation, & automated testing, and you'll see how they can help you solve Data Science problems quicker and in a way that will make future you happy. You'll even get to use your acquired software engineering chops to write your very own Python package for performing text analytics.

Learn about modularity, documentation, and automated testing to help you solve data science problems more quickly and reliably.

Software Engineering for Data Scientists in Python

Managing risk using Quantitative Risk Management is a vital task across the banking, insurance, and asset management industries. It’s essential that financial risk analysts, regulators, and actuaries can quantitatively balance rewards against their exposure to risk.


This course introduces you to financial portfolio risk management through an examination of the 2007—2008 financial crisis and its effect on investment banks such as Goldman Sachs and J.P. Morgan. You’ll learn how to use Python to calculate and mitigate risk exposure using the Value at Risk and Conditional Value at Risk measures, estimate risk with techniques like Monte Carlo simulation, and use cutting-edge  technologies such as neural networks to conduct real time portfolio rebalancing.

Learn about risk management, value at risk and more applied to the 2008 financial crisis using Python. 

Learn about risk management, value at risk and more applied to the 2008 financial crisis in Python. 

Data Science Courses

Data Science Tracks

Our Learning Experience

@darrenkeeley-0606220f-c648-4ef1-83bb-f30e0a54d3b2

Workspace Publications (0)

Completed Track(1)

Completed courses (25)

Completed Projects (0)