In 2019, the average salary for data engineers overtook data scientists. How did this happen? Companies wanting to find the gold within their data realized it wasn’t possible if they hadn’t yet built the mine. Data engineers lay the foundations that make data science possible.
 
In this course, you’ll learn about a data engineer’s core responsibilities, how they differ from data scientists, and facilitate the flow of data through an organization. Through hands-on exercises you’ll follow Spotflix, a fictional music streaming company, to understand how their data engineers collect, clean, and catalog their data. By the end of the course, you’ll understand what your company's data engineers do, be ready to have a conversation with a data engineer, and have a solid foundation to start your own data engineer journey.

Discover how data engineers lay the groundwork that makes data science possible. No coding involved!

Data Engineering for Everyone

As a Data Scientist, the majority of your time should be spent gleaning actionable insights from data -- not waiting for your code to finish running. Writing efficient Python code can help reduce runtime and save computational resources, ultimately freeing you up to do the things you love as a Data Scientist. In this course, you'll learn how to use Python's built-in data structures, functions, and modules to write cleaner, faster, and more efficient code. We'll explore how to time and profile code in order to find bottlenecks. Then, you'll practice eliminating these bottlenecks, and other bad design patterns, using Python's Standard Library, NumPy, and pandas. After completing this course, you'll have the necessary tools to start writing efficient Python code!

Learn to write efficient code that executes quickly and allocates resources skillfully to avoid unnecessary overhead.

Writing Efficient Python Code

In this course, you'll learn how to use Spark from Python! Spark is a tool for doing parallel computation with large datasets and it integrates well with Python. PySpark is the Python package that makes the magic happen. You'll use this package to work with data about flights from Portland and Seattle. You'll learn to wrangle this data and build a whole machine learning pipeline to predict whether or not flights will be delayed. Get ready to put some Spark in your Python code and dive into the world of high-performance machine learning!

Learn to implement distributed data management and machine learning in Spark using the PySpark package.

Learn to wrangle data and build a machine learning pipeline to make predictions with  PySpark Python package. Practice your skills with real-world data.

Introduction to PySpark

Object-oriented programming (OOP) is a widely used programming paradigm that reduces development times—making it easier to read, reuse, and maintain your code. OOP shifts the focus from thinking about code as a sequence of actions to looking at your program as a collection of objects that interact with each other. In this course, you’ll learn how to create classes, which act as the blueprints for every object in Python. You’ll then leverage principles called inheritance and polymorphism to reuse and optimize code. Dive in and learn how to create beautiful code that’s clean and efficient!

Dive in and learn how to create classes and leverage inheritance and polymorphism to reuse and optimize code.

Object-Oriented Programming in Python

You've done your analysis, built your report, and trained a model. What's next? Well, if you want to deploy your model into production, your code will need to be more reliable than exploratory scripts in a Jupyter notebook. Writing Functions in Python will give you a strong foundation in writing complex and beautiful functions so that you can contribute research and engineering skills to your team. You'll learn useful tricks, like how to write context managers and decorators. You'll also learn best practices around how to write maintainable reusable functions with good documentation. They say that people who can do good research and write high-quality code are unicorns. Take this course and discover the magic!

Learn to use best practices to write maintainable, reusable, complex functions with good documentation.

Writing Functions in Python

Have you heard people talk about data engineers and wonder what it is they do? Do you know what data engineers do but you're not sure how to become one yourself? This course is the perfect introduction. It touches upon all things you need to know to streamline your data processing. This introductory course will give you enough context to start exploring the world of data engineering. It's perfect for people who work at a company with several data sources and don't have a clear idea of how to use all those data sources in a scalable way. Be the first one to introduce these techniques to your company and become the company star employee.

Learn about the world of data engineering with an overview of all its relevant topics and tools!

Introduction to Data Engineering

The Unix command line has survived and thrived for almost 50 years because it lets people do complex things with just a few keystrokes. Sometimes called "the universal glue of programming," it helps users combine existing programs in new ways, automate repetitive tasks, and run programs on clusters and clouds that may be halfway around the world. This course will introduce its key elements and show you how to use them efficiently.

The Unix command line helps users combine existing programs in new ways, automate repetitive tasks, and run programs on clusters and clouds. 

Discover the Unix command line and how to use it to combine existing programs, automate repetitive tasks, and run programs on clusters and clouds in Shell.

Introduction to Shell

Before you can analyze data, you first have to acquire it. This course teaches you how to build pipelines to import data kept in common storage formats. You’ll use pandas, a major Python library for analytics, to get data from a variety of sources, from spreadsheets of survey responses, to a database of public service requests, to an API for a popular review site. Along the way, you’ll learn how to fine-tune imports to get only what you need and to address issues like incorrect data types. Finally, you’ll assemble a custom dataset from a mix of sources.

Learn to acquire data from common file formats and systems such as CSV files, spreadsheets, JSON, SQL databases, and APIs.

Streamlined Data Ingestion with pandas

Every data science project needs unit testing. It comes with huge benefits - saving a lot of development and maintenance time, improving documentation, increasing end-user trust and reducing downtime of productive systems. As a result, unit testing has become a must-have skill in the industry, used by almost every company. This course teaches unit testing in Python using the most popular testing framework pytest. By the end of this course, you will have written a complete test suite for a data science project. In the process, you will learn to write unit tests for data preprocessors, models and visualizations, interpret test results and fix any buggy code. You will also learn advanced concepts like TDD, test organization, fixtures and mocking so that you can test your own data science projects properly.

Learn how to write unit tests for your Data Science projects in Python using pytest.

Unit Testing for Data Science in Python

Delivering data on a schedule can be a manual process. You write scripts, add complex cron tasks, and try various ways to meet an ever-changing set of requirements—and it’s even trickier to manage everything when working with teammates. Airflow can remove this headache by adding scheduling, error handling, and reporting to your workflows. In this course, you’ll master the basics of Airflow and learn how to implement complex data engineering pipelines in production. You'll also learn how to use Directed Acyclic Graphs (DAGs), automate data engineering workflows, and implement data engineering tasks in an easy and repeatable fashion—helping you to maintain your sanity.

Learn how to implement and schedule data engineering workflows.

Learn to implement and schedule data engineering workflows by adding scheduling, error handling, and reporting using Airflow and Airflow DAGs in Python. 

Introduction to Airflow in Python

Bash is a concise, superfast, and robust scripting language for data and file manipulation. It’s a vital skill for building analytics pipelines in the cloud, favored by Linux users to work with data stored across multiple files.

In this course, we’ll guide you through the basics of Bash scripting. We begin with an introduction to Bash script structures, including inputting arguments and outputting results. You’ll then work through data structures, such as variables and arrays, and control statements, including loops and conditionals. You’ll then put what you’ve learned into practice, by creating your own Bash functions and scheduling automated scripts to run like clockwork with cron.

Bash scripting allows you to build analytics pipelines in the cloud and work with data stored across multiple files.

Learn how you can use Bash scripting to build automated scripts, create analytics pipelines in the cloud, and work with data stored across multiple files.

Introduction to Bash Scripting

We live in a busy world with tight deadlines. As a result, we fall back on what is familiar and easy, favoring GUI interfaces like Anaconda and RStudio. However, taking the time to learn data analysis on the command line is a great long-term investment because it makes us stronger and more productive data people.
 
 
In this course, we will take a practical approach to learn simple, powerful, and data-specific command-line skills. Using publicly available Spotify datasets, we will learn how to download, process, clean, and transform data, all via the command line. We will also learn advanced techniques such as command-line based SQL database operations. Finally, we will combine the powers of command line and Python to build a data pipeline for automating a predictive model.

Learn powerful command-line skills to download, process,  and transform data, including machine learning pipeline.

Learn powerful command-line skills to download, process, and transform data, including a machine learning pipeline.

Data Processing in Shell

Python is a general-purpose programming language that is becoming ever more popular for data science. Companies worldwide are using Python to harvest insights from their data and gain a competitive edge. Unlike other Python tutorials, this course focuses on Python specifically for data science. In our Introduction to Python course, you’ll learn about powerful ways to store and manipulate data, and helpful data science tools to begin conducting your own analyses. Start DataCamp’s online Python curriculum now.

Master the basics of data analysis in Python. Expand your skillset by learning scientific computing with NumPy.

Discover how to use Python for data science, learning about the ways to store and manipulate data in the environment to begin conducting your own analyses.

Introduction to Python

In any data-driven company, you will undoubtedly cross paths with data engineers. Among other things, they facilitate some of your work by making data readily available to everyone within the organization, and possibly in bringing machine learning models into production. One way to speed up this process is through building an understanding of what it means to bring processes into production and what features are of high-grade code. In this course, we’ll be looking at various data pipelines the data engineer is building, and how some of the tools he or she is using can help you in getting your models into production or run repetitive tasks consistently and efficiently.
 
In this course, we illustrate common elements of data engineering pipelines. In Chapter 1, you will learn how to ingest data. Chapter 2 will go one step further with cleaning and transforming data. In Chapter 3, you will learn how to safely deploy code. Finally, in Chapter 4 you will schedule complex dependencies between applications.
 
Building Data Engineering Pipelines covers new technologies and material, so we recommend that you have a strong understanding of the prerequisites to get the most out of this course.

Learn how to build data engineering pipelines in Python.

Learn how to build data engineering pipelines in Python by cleaning and transforming data before safely deploying code and scheduling complex dependencies.

Data Science Courses

Data Science Tracks

Our Learning Experience

@emrecelik

Workspace Publications (0)

Completed Tracks (0)

Completed courses (12)

Completed Projects (0)