AWS MSK for Beginners: A Comprehensive Getting-Started Guide

Setting Up AWS MSK

To get started with AWS MSK, first, create your AWS account. If it’s your first time using AWS, learn how to set up and configure your AWS account with our comprehensive tutorial.

Sign in to the AWS Management Console and open the MSK console. Click "Create cluster" to start the setup process. 

Getting started with AWS MSK. Image source: AWS

Select "Quick create" for default settings, then enter a descriptive cluster name.

From there, you have many additional options to select, which all depend on your own requirements for your cluster. Here’s a quick overview of the choices:

• Cluster type: “Provisioned” or “Serverless”
• Apache Kafka version
• Broker type: “Standard” or “Express”
• Broker size
• EBS storage volume

AWS MSK configuration options

The cluster is always created within an Amazon VPC. You can choose to use the default VPC or configure and specify a custom VPC.

Now, you just need to wait for your cluster to get activated, which can take 15 to 30 minutes. You can monitor the status of your cluster from the cluster summary page, where you will see the status change from “Creating” to “Active”.

Ingesting and Processing Data with AWS MSK

Once your MSK cluster is set up, you’ll need to create a client machine to produce and consume data across one or more topics. Since Apache Kafka integrates so well with many data producers (such as websites, IoT devices, Amazon EC2 instances, etc.), MSK also shares this benefit.

Apache Kafka organizes data in structures called topics. Each topic consists of single or many partitions. Partitions are the degree of parallelism in Apache Kafka. The data is distributed across brokers using data partitioning.

Key terms to know when dealing with Apache Kafka clusters:

• Topics are the fundamental way of organizing data in Kafka.
• Producers are applications that publish data to topics—they generate and write data to Kafka. They write data on specific topics and partitions.
• Consumers are applications that read and process data from topics. They pull data from topics to which they are subscribed.

When building an event-driven architecture with AWS MSK, you need to configure several layers, of which MSK is the main data ingestion component. Here’s an overview of the layers that may be required:

1. Data ingestion setup
2. Processing layer
3. Storage layer
4. Analytics layer

Example of an event-driven architecture with Amazon MSK and Amazon EventBridge. Image source: AWS

If you’re interested in leveraging Python in your data pipeline workflows, check out our Introduction to AWS Boto in Python course.

Best Practices for Using AWS MSK

AWS MSK is relatively simple to set up and start using right away. However, some essential best practices will improve the performance of your clusters and save you time later down the road.

Right-size your cluster

You will need to choose the right number of partitions per broker and the right number of brokers per cluster. 

A number of factors can influence your decisions here; however, AWS has provided some handy recommendations and resources to guide you through this process.

In addition, AWS provides an easy-to-use sizing and pricing spreadsheet to help you estimate the right size of your cluster and the associated costs of using AWS MSK versus a similar self-managed EC2 Kafka cluster.

Build highly available clusters

AWS recommends that you set up your clusters to be highly available. This is especially important when performing an update (such as updating the Apache Kafka version) or when AWS is replacing a broker. 

To ensure that your clusters are highly available, there are three things you must do:

1. Set up your clusters across three availability zones (also called a three-AZ cluster).
2. Set the replication factor to 3 or more.
3. Set the minimum number of in-sync replicas to RF-1.

The great thing about AWS is that they commit to strict SLAs for multi-AZ deployments; otherwise, you get your credits back.

Monitor disk and CPU usage

Two key metrics to monitor through AWS CloudWatch are disk and CPU usage. Doing this will not only ensure that your system runs smoothly but will also help to keep costs down. 

The best way to manage disk usage and the associated storage costs is to set up a CloudWatch alarm that alerts you when disk usage exceeds a certain value, such as 85%, and to adjust your retention policies. Setting a retention time for messages in your log can go a long way toward helping free up disk space automatically.

Additionally, to maintain the performance of your cluster and avoid bottlenecks, AWS recommends that you maintain the total CPU usage for your brokers under 60%. You can monitor this using AWS CloudWatch and then take corrective action by updating your broker size, for example.

Protect your data using encryption in transit

By default, AWS encrypts data in transit between brokers in your MSK cluster. You can disable this if your system is experiencing high CPU usage or latency. However, it is strongly recommended that you keep in-transit encryption enabled at all times and find other ways of improving performance if that is a problem for you.

Check out our AWS Security and Cost Management course to learn more about how to secure and optimize your AWS cloud environment and manage costs and resources in AWS.

Understanding AWS MSK Pricing

AWS MSK pricing is pay-as-you-go, meaning you only pay for the resources you use. However, pricing varies based on cluster type, broker instance size, storage, and data transfer. Below, we break down the key cost factors and how to optimize your MSK spend.

AWS MSK pricing components

AWS MSK pricing by cluster type

AWS MSK offers two cluster types with different pricing models:

Provisioned MSK cluster (standard pricing model)

• You pay for broker instances, storage, and data transfer.
• Best for workloads that need fine-tuned performance control.

Serverless MSK cluster (pay-per-request model)

• Pricing is based on the volume of data processed (GB ingested/retrieved).
• Best for applications with unpredictable traffic patterns.

Which one should you choose?

• Use provisioned MSK → If you have predictable workloads and need control over instance sizes.
• Use serverless MSK → If your workload fluctuates and you want to avoid managing brokers.

Try the AWS Pricing Calculator to estimate costs based on your workload.

Comparing AWS MSK to Other Streaming Tools

When deciding which tool is best for a project, we often need to evaluate several options. Here are the most common alternatives to AWS MSK and how they compare. 

AWS MSK vs Apache Kafka on EC2

The main trade-off between MSK and a self-hosted option using EC2 is between convenience and control:

• AWS MSK:
  • Fully managed service, reducing operational complexity.
  • Automatic provisioning and configuration—no upfront infrastructure costs.
  • Seamless integration with AWS services.
  • Robust security features.
  • Less flexibility compared to self-hosting.
• Kafka on EC2:
  • Requires manual setup, configuration, and maintenance.
  • Offers full control over infrastructure and customization.
  • Potentially higher operational costs.
  • Requires more expertise and ongoing management.

AWS MSK vs. Kinesis

Choosing between AWS MSK and Kinesis depends on your priorities:

• Use Kinesis if:
  • You need a fully serverless architecture.
  • AWS-managed service with minimal operational overhead.
  • Uses shards for data streaming but has data retention limits.
  • Best suited for simple data streaming requirements.
• Use AWS MSK if:
  • You need Kafka’s topic and partition model.
  • Virtually unlimited data retention (depends on storage).
  • More flexible and customizable.
  • Allows migration away from AWS if needed.

If you’re not familiar with Kinesis, we have a course that walks you through working with streaming data using AWS Kinesis and Lambda.

AWS MSK vs. Confluent

The choice between AWS MSK and Confluent depends on feature requirements and cloud strategy:

• Use Confluent if:
  • You need an extensive feature set with built-in connectors.
  • You prefer a more straightforward deployment process.
  • You're handling spiky workloads that require dynamic scaling.
  • Higher cost but offers a free tier with limited features.
• Use AWS MSK if:
  • You want a cost-effective option for consistent workloads.
  • You are deeply integrated into the AWS ecosystem.
  • You have in-house Kafka expertise.
  • You’re fine with integrating additional AWS services for extended functionality.

The following table offers a comparison of AWS MSK and its alternatives:

Closing Thoughts

Apache Kafka is the go-to choice for situations where you need a large-scale, reliable solution that cannot afford data loss and requires connecting multiple data sources or building complex data pipelines. AWS MSK prevents many of the headaches of setting up and configuring Kafka clusters, allowing developers to focus more on building and improving applications instead of infrastructure.

Getting an AWS certification is an excellent way to start your AWS career. You can build your AWS skills by checking out our course catalog and getting hands-on experience through projects!