Skip to main content

Streaming Data with AWS Kinesis and Lambda

Learn how to work with streaming data using serverless technologies on AWS.

Start Course for Free
4 Hours22 Videos56 Exercises2,594 Learners
4500 XP

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA. You confirm you are at least 16 years old (13 if you are an authorized Classrooms user).

Loved by learners at thousands of companies

Course Description

What powers the systems that we use every day without realizing, like fraud detection to keep our transactions secure or traffic signals that keep traffic flowing smoothly? The answer is streaming data (data that is continuously generated by different sources) and serverless technologies—like Amazon Kinesis and AWS Lambda.

In this course, you’ll learn how to leverage these powerful technologies by helping a fictional data engineer named Cody. Your goal is to help her to collect real-time streaming data from city-owned vehicles, analyze the data, and send relevant alerts like speed warnings to drivers. Using Amazon Kinesis and Firehose, you’ll learn how to ingest data from millions of sources before using Kinesis Analytics to analyze data as it moves through the stream. You’ll also spin up serverless functions in AWS Lambda that will conditionally trigger actions based on the data received. By the end of this training you’ll know how to create live ElasticSearch dashboards with AWS QuickSight and CloudWatch—and hopefully helped Cody complete her ambitious project.

  1. 1

    Streaming in the cloud

    Free

    In this first chapter, you will learn about the differences between batch and stream processing, create your first stream, manage its permissions, write to it and read from it.

    Play Chapter Now
    What is streaming and why does it matter?
    50 xp
    Batch vs stream
    100 xp
    Producers and destinations
    100 xp
    Managing Firehose delivery streams
    100 xp
    Getting ready for the first stream
    50 xp
    Creating roles
    50 xp
    IAM users vs. roles
    100 xp
    Permissions practice
    100 xp
    S3 bucket creation
    100 xp
    Working with the Firehose delivery stream
    50 xp
    Create your first Firehose stream
    100 xp
    Writing to a Firehose stream
    100 xp
    Reading Firehose data
    100 xp
  2. 2

    Going serverless

    The next step in your streaming data journey is learning how to use transformational lambda functions to go serverless. Through hands-on exercises, you will add lambda layers and trigger lambda functions on specific conditions.

    Play Chapter Now
    Going serverless
    50 xp
    Creating and running Lambda functions
    50 xp
    Serverless vs servers
    100 xp
    Lambda function components
    100 xp
    Reading data from S3 based on an event
    100 xp
    Your first live lambda!
    50 xp
    Adding a lambda layer
    50 xp
    Lambda flow
    100 xp
    Using environment variables
    100 xp
    Serverless data workflow
    50 xp
    Fast vs slow lambdas
    100 xp
    A time-triggered lambda
    100 xp
    Time-triggered lambda code
    100 xp
    Serverless APIs
    50 xp
    Create an API lambda
    100 xp
    A serverless API
    100 xp
  3. 3

    Analyzing streaming data

    You're now ready to encode and decode streaming data and analyze data directly in the stream. You will even use multiple streams to get daily vehicle top speeds.

    Play Chapter Now
    A transformational Lambda
    50 xp
    Transforming data inside a stream
    50 xp
    Encoding and decoding base64
    100 xp
    Create a transformational lambda
    100 xp
    A barebones transformational lambda
    100 xp
    Analyzing data in the stream
    50 xp
    Creating a Kinesis data analytics application
    50 xp
    Kinesis data analytics vs transformational Lambda
    100 xp
    Building a Kinesis data analytics application
    100 xp
    Kinesis data analytics SQL components
    100 xp
    Using multiple streams
    50 xp
    Delivering data from Kinesis Analytics
    50 xp
    Get the daily top speed
    100 xp
    Using Kinesis data analytics for alerts
    100 xp
  4. 4

    Monitoring and visualizing streaming data

    In this final chapter, you will discover how to monitor your stream's performance using logs, metrics, alarms and dashboards. You will use Elasticsearch and build your own Kibana dashboard.

    Play Chapter Now
    Streaming data case study
    50 xp
    Creating an Elasticsearch cluster
    50 xp
    RedShift vs Elasticsearch
    100 xp
    Monitoring performance
    50 xp
    Cloudwatch dasbhoards and alarms
    50 xp
    Cloudwatch components
    100 xp
    Cloudwatch monitoring flow
    100 xp
    Visualizing streaming data
    50 xp
    Working with ElasticSearch using Kibana
    50 xp
    Cloudwatch vs Elasticsearch
    100 xp
    Elasticsearch visualization flow
    100 xp
    An alternative approach
    100 xp
    Another alternative approach
    100 xp

Collaborators

Hadrien LacroixJen BrickerLis Sulmont

Prerequisites

Introduction to AWS Boto in PythonIntroduction to ShellStreaming Concepts
Maksim Pecherskiy Headshot

Maksim Pecherskiy

Data Engineer

Maksim is a Data Engineer at the World Bank, and formerly the Chief Data Officer at the City of San Diego. He also works with tech companies to help them effectively connect technology to government. he came to the US from Ukraine when he was 10 years old, lived in 5 countries and speaks 3 languages.
See More

What do other learners have to say?

I've used other sites—Coursera, Udacity, things like that—but DataCamp's been the one that I've stuck with.

Devon Edwards Joseph
Lloyds Banking Group

DataCamp is the top resource I recommend for learning data science.

Louis Maiden
Harvard Business School

DataCamp is by far my favorite website to learn from.

Ronald Bowers
Decision Science Analytics, USAA