Apache Arrow: A Beginner’s Guide with Practical Examples

Apache Arrow helps different tools and systems share data quickly and easily. It stores data in a special column-based format that stays in memory, which makes it really fast to work with.

In this tutorial, I’ll walk you through the basics of Apache Arrow. You’ll get hands-on examples to understand how it works and how to use it in your projects.

What is Apache Arrow?

Apache Arrow is an open-source, cross-language platform for high-performance in-memory data processing. It defines a standardized, columnar memory format optimized for modern CPUs to enable efficient analytics and vectorized computation.

Arrow also removes the need for serialization between systems and languages, which allows for zero-copy reads and cuts down processing overhead. 

Because it works well with tools like pandas, Apache Spark, and Dask, you can move data around with minimal effort. That makes it an excellent choice for anyone building fast, flexible systems that need to handle lots of data.

Now, let’s explore Arrow with some practical examples!

Installing Apache Arrow

Apache Arrow supports multiple languages, including C, C++, C#, Python, Rust, Java, JavaScript, Julia, Go, Ruby, and R. 

It works on Windows, macOS, and popular Linux distributions, such as Debian, Ubuntu, CentOS, and Red Hat Enterprise Linux.

In this guide, we’ll use it with Python.

Step-by-step installation guide

Let’s see how to install Apache Arrow for Python using the official binary wheels published on PyPI. These steps work the same across Windows, macOS, and Linux.

Step 1: Check if Python is installed

Open your terminal and run:

python --version

If this returns a version number, you're good to go. If not, install Python from the official site.

Step 2: Install pyarrow using pip

To install Apache Arrow for Python, run:

pip install 'pyarrow==19.0.* '  

The * ensures you get the latest patch release in the 19.0 series. 

I also recommend using pyarrow==19.0.* in your requirements.txt file to keep things consistent across environments. This will install pyarrow along with the required Apache Arrow and Parquet C++ binary libraries bundled with the wheel.

To install Arrow for another language, check out the complete documentation.

Testing the installation

You can run this quick Python snippet to check if everything works:

import pyarrow as pa
# Create a simple Arrow array
data = pa.array([1, 2, 3, 4, 5])
print("Apache Arrow is installed and working!")
print("Arrow Array:", data)

If pyarrow is installed correctly, you’ll see the array printed in your terminal. If not, the code will throw an error.

Apache Arrow Fundamentals

Before we get deeper into code, let’s understand how Apache Arrow is built.

Understanding columnar memory format

The in-memory columnar format is one of Apache Arrow’s most important features. It defines a language-independent, in-memory data structure. In addition, it includes metadata serialization, a protocol for data transport, and a standard way to represent data across systems.

In a columnar format, data is stored in columns instead of rows, which is different from traditional row-based storage. Columnar layouts are popular for analytical workloads because they’re much faster to scan and process.

The image below explains the format:

Columnar format. Source: Apache Arrow docs

Apache Arrow’s format boosts efficiency by laying columns in contiguous memory blocks. This supports modern vectorized (SIMD) operations, which speed up data processing.

Using a standard columnar layout, Arrow removes the need for every system to define its internal data format. That reduces both serialization overhead and duplication of effort. 

Systems can share data more easily, and developers can reuse algorithms across different languages.

Without a shared format like Arrow, each database or tool needs to serialize and deserialize data before passing it on. That adds cost and complexity. And common algorithms have to be rewritten for each system’s internal data structure. 

Arrow changes that by standardizing how data is represented in memory.

While columnar formats are great for analytical performance and memory locality, they’re not designed for frequent updates.

That’s because columnar storage is optimized for fast reading, not modifying. Making changes to data often means copying or reallocating memory, which can be slow and resource-intensive.

Arrow focuses on efficient in-memory representation and serialization. If your use case needs frequent data mutation, you’ll need to handle that at the implementation level.

To dive deeper into Arrow-compatible formats, check out this guide to Apache Parquet, ideal for columnar data storage.        

Arrow data structures

Apache Arrow provides two key data structures: Array and Table: 

•  Arrow Array: An Array represents a single column of data. It stores values in a contiguous memory block, allowing fast access, efficient scanning, and support for processor-level optimizations like SIMD. Each array can also carry metadata such as null indicators (bitmaps) and data type information.
• Arrow Table: Data often comes in two-dimensional formats, like database tables or CSV files. So an Arrow Table represents a 2D dataset. Each column is a chunked array, and all columns follow a shared schema that defines field names and types. While the chunks can differ across columns, the total number of elements must stay aligned to keep the rows consistent.

You can think of it like a DataFrame in pandas or R. It’s tabular, efficient, and supports operations across multiple columns simultaneously.

Example of basic data structures

Let’s look at a few examples of creating an Arrow Array and Table in Python. 

Make sure you have pyarrow installed before running the code below, following the previous instructions.

Creating an Arrow Array

import pyarrow as pa

# Create an Arrow Array (single column)
data = pa.array([1, 2, 3, 4, 5])
print("Arrow Array:")
print(data)

Print an Arrow Array. Image by author.

Creating an Arrow Table

import pyarrow as pa

# Create an Arrow Table (multiple columns)
table = pa.table({
    'column1': pa.array([1, 2, 3]),
    'column2': pa.array(['a', 'b', 'c'])
})

print("\nArrow Table:")
print(table)

Print an Arrow Table. Image by author.          

Working with Apache Arrow Arrays

Let’s see how to work with Arrow Arrays and perform common operations, such as slicing, filtering, and checking metadata.

Creating Arrow Arrays

Arrow Arrays work with different data types, including strings, floating-point numbers, and integers. Here's how to create them:

import pyarrow as pa

# Integer Array
int_array = pa.array([10, 20, 30], type=pa.int32())
print("Integer Array:")
print(int_array)

# String Array
string_array = pa.array(["apple", "banana", "cherry"], type=pa.string())
print("\nString Array:")
print(string_array)

# Floating-point Array
float_array = pa.array([1.1, 2.2, 3.3], type=pa.float64())
print("\nFloating-point Array:")
print(float_array)

Printing Arrow Arrays with different data types. Image by author.

Basic operations on Arrow Arrays

Let’s look at a few common operations you can perform with Arrow Arrays.

Slicing

Slicing lets you grab a subsection of an array. For example, you can extract elements at specific index ranges:

import pyarrow as pa
arr = pa.array([10, 20, 30, 40, 50])
# Slice from index 1 to 4 (3 elements: 20, 30, 40)
sliced = arr.slice(1, 3)
print("Original Array:", arr)
print("Sliced Array (1:4):", sliced)

Printing a sliced Arrow Array. Image by Author.

Filtering and metadata

Filtering uses a Boolean mask to keep only values that match a condition. You can also access metadata like type, length, and null counts. 

import pyarrow.compute as pc

arr = pa.array([10, 20, 30, 40, 50])

# Create a boolean mask: Keep elements > 25
mask = pc.greater(arr, pa.scalar(25))

# Apply the filter
filtered = pc.filter(arr, mask)

print("Filtered Array (values > 25):", filtered)

print("Array Type:", arr.type)
print("Array Length:", len(arr))
print("Null Count:", arr.null_count)
print("Is Null Bitmap Buffers:", arr.buffers())

Printing a filtered Arrow Array. Image by Author.

Printing Metadata of an Arrow Array. Image by Author.

Using Apache Arrow Tables

You can create Arrow Tables from Arrow Arrays or directly from native Python data structures like dictionaries. Let’s see how. 

Creating and viewing tables

There are two common ways to create an Arrow Table:

1. Using multiple Arrow Arrays
2. Using a Python dictionary with pa.table()

Let’s explore both ways one by one.

1. Using Arrow Arrays

import pyarrow as pa

# Create individual Arrow Arrays
ids = pa.array([1, 2, 3])
names = pa.array(['Alice', 'Bob', 'Charlie'])
scores = pa.array([85.0, 92.5, 78.3])

# Combine them into a Table
table = pa.table({
    'id': ids,
    'name': names,
    'score': scores
})

print("Arrow Table from Arrays:")
print(table)

In the code above:

• We create three arrays (integer, string, float)
• We define a schema by giving each column a name
• We combine them into a Table with pa.table()

Creating an Arrow Table using multiple Arrow Arrays. Image by author.

2. Using a Python dictionary

Now, let’s see how to create a Table from a Python dictionary.

# Create a table directly from Python data
data = {
    'id': [4, 5, 6],
    'name': ['Diana', 'Eve', 'Frank'],
    'score': [88.0, 79.5, 91.2]
}

table2 = pa.table(data)

print("\nArrow Table from Dictionary:")
print(table2)

This method is shorter and more compact. We don’t have to define arrays manually; Arrow handles that for us. 

Creating an Arrow Table directly from Python dictionaries. Image by author.

Converting between Arrow Tables and pandas DataFrames

One of Arrow’s biggest strengths is its interoperability. You can easily move between an Arrow Table and a pandas DataFrame. Let’s see how to do that:

Converting an Arrow Table to a Pandas DataFrame

import pyarrow as pa
import pandas as pd

# Create an Arrow Table
data = {
    'numbers': [1, 2, 3],
    'fruits': ['apple', 'banana', 'cherry'],
    'prices': [1.1, 2.2, 3.3]
}
arrow_table = pa.table(data)

# Convert Arrow Table to pandas DataFrame
df = arrow_table.to_pandas()

# Show the DataFrame
print(df)

Converting an Arrow Table to a Pandas DataFrame. Image by author.

Converting a Pandas DataFrame to an Arrow Table

Now, let’s see how to do the opposite: 

import pyarrow as pa
import pandas as pd

# Create a pandas DataFrame
df = pd.DataFrame({
    'numbers': [1, 2, 3],
    'fruits': ['apple', 'banana', 'cherry'],
    'prices': [1.1, 2.2, 3.3]
})

# Convert pandas DataFrame to Arrow Table
arrow_table = pa.Table.from_pandas(df)

# Show the Arrow Table
print(arrow_table)

Converting pandas DataFrame to an Array Table. Image by author.