What is Amazon Kinesis? Use Cases, Pricing, and Cost Optimization Tips

Data streaming is a constant and continuous flow of data, often in high volumes. One of the most important aspects of a data stream is that it is time-sensitive. So, after a certain amount of time, the contents of a data stream can become less relevant.

This means that companies can gain immense value from real-time analytics and insights from their data streams. This is where specialized data streaming architecture comes in. One of the most popular and powerful options for data streaming is AWS Kinesis.

In this article, we introduce AWS Kinesis and explore the two core services offered under the Kinesis umbrella: Kinesis Data Streams and Kinesis Video Streams. At the end of the article, we also compare the main differences between two data streaming services: AWS Kinesis and Apache Kafka (and Amazon MSK).

If you're new to AWS, check out our introductory AWS course that will take you through the basic concepts behind today's leading cloud provider.

What is AWS Kinesis?

Amazon Kinesis was launched in 2013 in response to a shift in the market towards real-time data ingestion and analysis.

However, AWS took this a step further since Amazon Kinesis is able to process large data streams in a fully managed, scalable, and cloud-based service.

In the years that followed the release, AWS expanded their services offered under Amazon Kinesis. This started with the release of Amazon Kinesis Data Firehose in October 2015 and later released Amazon Kinesis Data Analytics in August 2016.

Initially, AWS only offered its Kinesis Data Streams service for processing and analyzing large amounts of data in real-time. It was only in 2017 that AWS introduced Kinesis Video Streams for capturing, processing, and storing video streams.

However, Amazon Kinesis continued to evolve and change over the years. Fast forward to today and only the two core services remain: Kinesis Data Streams and Kinesis Video streams. We'll discuss these changes in more detail later in this section.

Source: AWS

What is Amazon Kinesis used for?

Amazon Kinesis is designed for real-time data streaming.

This means that any organization wanting to leverage large volumes of real-time (or near real-time) data would benefit immensely from the Amazon Kinesis offering.

Here are some use cases that Amazon Kinesis is best suited for:

• Streaming log and event data from one or more sources. Data from applications, servers, and devices can be ingested in real time.
• Collecting video streams from camera-enabled devices. Think home and business surveillance systems, traffic light cameras, and so on.
• Real-time analysis of streaming data. Use cases involving fraud detection, for example, are highly time-sensitive and rely on a solid data streaming architecture.

Amazon Kinesis Video Streams Example Use Case (Source: AWS)

Recent changes to Amazon Kinesis services

In response to user feedback, AWS recently made a couple of changes to its services.

Amazon Kinesis Data Analytics > Amazon Managed Service for Apache Flink

The Amazon Kinesis Data Analytics was renamed to Amazon Managed Service for Apache Flink (MSF) on August 30, 2023.

Not only that but support for Amazon Kinesis Data Analytics for SQL applications will be discontinued in the future. Customers will need to migrate their applications over to MSF for their SQL-based applications.

MSF is a fully managed and serverless service to build and run real-time streaming applications using Apache Flink. MSF includes support for SQL along with other programming languages, such as Java and Scala.

Amazon Kinesis Data Firehose > Amazon Data Firehose

On February 9, 2024, AWS announced that it would be renaming Amazon Kinesis Data Firehose to just Amazon Data Firehose.

AWS made this change to reflect the growth in their data sources and connectors within Amazon Data Firehose. In addition to Amazon Kinesis Data Streams, integration support was added for Amazon MSK and more than 40 other AWS source and destination connectors.

AWS Kinesis Data Streams

AWS Kinesis Data Streams is Amazon's flagship service under the Amazon Kinesis umbrella.

In this section, we explore this service and the use cases for which it is best suited.

What is Kinesis Data Streams?

Kinesis Data Streams can continuously ingest large volumes of data (at the speed of gigabytes per second) to facilitate real-time analytics use cases.

In addition to being fully integrated with other AWS services, here are a few standout features of Kinesis Data Streams:

• It is serverless, so there is no need to provision or manage servers ahead of time, reducing the up-front costs of getting started with data streaming.
• With Kinesis, data collection is separate from data processing. This separation makes it easy for applications to adapt to rapid changes in information.
• Kinesis Data Streams can capture large volumes of data in milliseconds which makes it an incredible low latency solution.
• You can choose between on-demand and provisioned capacity modes to suit your data streaming requirements.
• Kinesis Data Streams are both secure and compliant. Your data can be encrypted and secured at rest to ensure maximum security.

Amazon Kinesis Data Streams consists of three key components: shards, producers, and consumers. Let’s break these down using a real-world example of a streaming analytics system for an ecommerce website.

Shards

A shard is a unit within Kinesis that determines how much data can be processed at a time.

For example, if the ecommerce website processes 100 transactions per second, it might require multiple shards to handle the incoming data smoothly. Each shard can handle up to 1 MB of incoming data per second, so if the traffic spikes, adding more shards will ensure data is processed efficiently.

Producers 

A producer is any system that sends data to the Kinesis stream.

For example, the e-commerce site generates real-time logs every time a user adds an item to their cart, completes a purchase, or leaves a review. Each of these events is sent as a data record into Kinesis.

Consumers

A consumer is any application or system that reads data from the Kinesis stream and processes it. For example: 

• A fraud detection system that analyzes incoming transactions in real time to flag suspicious activity.
• A dashboard that updates live sales metrics for store managers.
• A recommendation engine that suggests products to users based on their recent activity.

Let’s see how these all interact with an example: 

1. A customer adds a product to their cart → The event is sent to Kinesis (Producer)
2. The event enters a shard (depending on traffic load)
3. A consumer application reads the event → Updates the inventory system, triggers an AI recommendation model, or flags a fraud risk

What are Kinesis Data Streams used for?

Kinesis data streams are best suited to applications that require rapid data processing in real-time.

For example, suppose you have an application or device that generates logs or data feeds. You would use Kinesis Data Streams to ingest this stream of data and apply some processing or transformation to it, such as calculating some real-time metrics or analytics. You can then push this processed data to a data store for later use.

This is just one common example of the usefulness of Kinesis Data Streams.

AWS Kinesis Video Streams

AWS Video Streams is a more recent Amazon Kinesis Service with powerful applications for various kinds of media data.

In this section, we explore this service and go over what applications it is best suited to.

What is Kinesis Video Streams?

Kinesis Video Streams allow you to ingest large volumes of time-encoded media data from connected devices.

More concretely, time-encoded data is a type of media data where records are related to each other in a time series. Video and audio are both examples of time-encoded data since each frame or sequence in this data is related to the previous and next frame.

Amazon Kinesis Video Streaming (Source: AWS)

It is important to note that Kinesis Video Streams only handles the ingestion and storage of media data. Processing and analysis must take place within a different service. Examples of other services are Amazon Rekognition Video or Amazon SageMaker for machine learning applications.

Check out our complete tutorial on Amazon SageMaker to start building machine learning applications in AWS.

What are Kinesis Video Streams used for?

It is very easy (and secure) to connect to and stream media to AWS using Kinesis Video Streams. There are SDKs and real-time APIs purpose-built for Amazon Kinesis.

This means that you can get up and running quickly and easily with Amazon Kinesis Video Streams for your media streaming applications.

One of the most popular uses of video streams is to build real-time computer vision-enabled machine learning applications. These can be used in a wide range of areas, such as security and surveillance, smart homes, city emergency response, and so on.

AWS Kinesis Pricing

One of the best things about AWS is its pay-as-you-go pricing model. Most, if not all, AWS services follow this model, and AWS Kinesis is no different.

There are no upfront costs or minimum fees. You only pay for the resources that you use.

Let's go over how pricing works for AWS Kinesis and how you can apply some best practices to optimize your costing.

Overview of pricing

The two core services under AWS Kinesis, Kinesis Data Streams and Kinesis Video Streams, have different costs and billing options based on how you set up the services.

Kinesis Data Streams has two capacity modes, each with different costs. Here’s a brief summary:

• On demand: You are charged for the volume of data that you ingest and retrieve, as well as an hourly rate for each data stream you set up in your AWS account.
• Provisioned: Using the provision capacity mode, you must specify the number of shards required for your application and you will be charged for each shard at an hourly rate.

The on-demand capacity mode is a more flexible and simplistic approach that is good when you’re just getting started with Kinesis Data Streams.

However, once your application reaches the point where your traffic is more predictable and easier to forecast, the provisioned capacity mode can be a more cost effective option.

Kinesis Video Streams offers a flexible, on-demand pricing option based on three factors:

• The volume of data that you ingest (in gigabytes).
• The volume of data that you consume (in gigabytes).
• The volume of data that you store in a month (in gigabytes).

Cost optimization tips and best practices

Both Kinesis Data Streams and Kinesis Video Streams are not available in AWS Free Tier. This means that you will be charged from the moment you set up one of these streams in AWS.

However, there are a number of best practices and a few tips you can follow as you set up AWS Kinesis for your applications to keep costs low.

To optimize costs, here are three top tips to follow:

• Switch to provision capacity mode as soon as your traffic becomes predictable. While the on-demand mode removes the headache of monitoring and managing shards, it can be much more expensive.
• You can easily switch between on-demand and provisioned capacity modes based on traffic patterns. For example, if you expect spikes during certain hours of the day, you can switch to on-demand and then switch back to provisioned mode when you know traffic is more consistent. Be careful with this though, you can only make this switch twice in a 24 hour period.
• Set up your data retention policies early on. Kinesis Data Streams is not designed for long-term storage of data. You will be charged for data stored longer than 24 hours (storage is free in those first 24 hours).

AWS Kinesis vs Kafka

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is another popular data streaming service offered by AWS.

You can find our comprehensive beginner's guide to Amazon MSK in a separate article, as well as our guide to Kafka.

When comparing Amazon MSK to Kinesis, there is a lot of overlap. Both services offer scalable, secure, and highly available solutions for data streaming applications.

The only real difference when deciding which service to use comes down to your own preferences, requirements, and current infrastructure.

If you are already using Apache Kafka, you may benefit from migrating to a fully managed cloud solution like Amazon MSK. Similarly, if you have a team that is comfortable with Apache Kafka, then Amazon MSK would be preferable over Amazon Kinesis.

On the other hand, if you do not have any existing data streaming infrastructure, then you may find Amazon Kinesis to be more beginner-friendly and easier to get started with.

Conclusion

Real-time data analytics is becoming increasingly popular and important for many industries and business. To achieve this, it is necessary to set up a good data streaming system that can handle large amounts of data without any lags or downtime.

AWS recognized this growing need back in 2013 and they have continued to develop and fine-tune one of their best data streaming solutions: Amazon Kinesis. Today, Amazon Kinesis is the best-in-class solution to ingesting and processing data streams at incredible speeds.

If you’re new to AWS, start with our beginner-friendly AWS course to learn more about AWS and its services. If you're ready to start a career using cloud-based services like AWS, check out our article on AWS certifications and how to choose the best one for your goals. Then, take the next step by preparing for an interview and practicing the top AWS interview questions.