Python Cache: Two Simple Methods

In this article, we'll learn about caching in Python. We'll understand what it is and how to use it effectively.

Caching is a technique used to improve application performance by temporarily storing results obtained by the program to reuse them if needed later.

In this tutorial, we'll learn different techniques for caching in Python, including the @lru_cache and @cache decorators in the functools module.

For those of you in a hurry, let's start with a very short caching implementation and then continue with more details.

Short Answer: Python Caching Implementation

To create a cache in Python, we can use the @cache decorator from the functools module. In the code below, notice that the print() function is only executed once:

import functools

@functools.cache
def square(n):
    print(f"Calculating square of {n}")
    return n * n

# Testing the cached function
print(square(4))  # Output: Calculating square of 4 \n 16
print(square(4))  # Output: 16 (cached result, no recalculation)

Calculating square of 4
16
16

What is Caching in Python?

Suppose we need to solve a mathematical problem and spend an hour getting the right answer. If we had to solve the same problem the following day, it’d be helpful to reuse our previous work rather than start all over again.

Caching in Python follows a similar principle—it stores values when they're calculated within function calls to reuse them when needed again. This type of caching is also referred to as memoization.

Let's look at a short example that works out the sum of a large range of numbers twice:

output = sum(range(100_000_001))
print(output)
output = sum(range(100_000_001))
print(output)

5000000050000000
5000000050000000

The program has to work out the sum each time. We can confirm this by timing the two calls:

import timeit

print(
    timeit.timeit(
        "sum(range(100_000_001))",
        globals=globals(),
        number=1,
    )
)

print(
    timeit.timeit(
        "sum(range(100_000_001))",
        globals=globals(),
        number=1,
    )
)

1.2157779589979327
1.1848394999979064

The output shows that both calls take roughly the same amount of time (depending on our setup, we may get faster or slower execution times).

However, we can use a cache to avoid working out the same value more than once. We can redefine the name sum using the cache() function in the built-in functools module:

import functools
import timeit

sum = functools.cache(sum)

print(
    timeit.timeit(
        "sum(range(100_000_001))",
        globals=globals(),
        number=1,
    )
)

print(
    timeit.timeit(
        "sum(range(100_000_001))",
        globals=globals(),
        number=1,
    )
)

1.2760689580027247
2.3330067051574588e-06

The second call now takes a couple of microseconds instead of over one second because the result of finding the sum of the numbers from 0 to 100,000,000 has already been calculated and cached—the second call uses the value that was calculated and stored earlier.

Above, we use the decorator functools.cache() to include a cache to the sum() built-in function. As a side note, a decorator in Python is a function that modifies the behavior of another function without permanently changing its code. You can learn more about decorators in this Python Decorators Tutorial.

The functools.cache() decorator was added to Python in version 3.9, but we can use functools.lru_cache() for older versions. In the next section, we'll explore both of these ways to create a cache, including using the more frequently used decorator notation, such as @cache.

Python Caching: Different Methods

Python's functools module has two decorators for applying caching to functions. Let's explore functools.lru_cache() and functools.cache() with an example.

Let’s write a function sum_digits() that takes in a sequence of numbers and returns the sum of the digits of those numbers. For example, if we use the tuple (23, 43, 8) as an input, then:

• The sum of the digits of 23 is five.
• The sum of the digits of 43 is seven.
• The sum of the digits of 8 is eight.
• Therefore, the overall sum is 20.

This is one way we can write our sum_digits() function:

def sum_digits(numbers):
    return sum(
        int(digit) for number in numbers for digit in str(number)
    )

numbers = 23, 43, 8

print(sum_digits(numbers))

20

Let's use this function to explore different ways to create a cache.

Python manual caching

Let’s first create the cache manually. While we could also automate this easily, creating a cache manually helps us understand the process.

Let’s create a dictionary and add key-value pairs each time we call the function with a new value to store the results. If we call the function with a value that's already stored in this dictionary, the function will return the stored value without working it out again:

import random
import timeit

def sum_digits(numbers):
    if numbers not in sum_digits.my_cache:
        sum_digits.my_cache[numbers] = sum(
            int(digit) for number in numbers for digit in str(number)
        )
    return sum_digits.my_cache[numbers]
sum_digits.my_cache = {}

numbers = tuple(random.randint(1, 1000) for _ in range(1_000_000))

print(
    timeit.timeit(
        "sum_digits(numbers)",
        globals=globals(),
        number=1
    )
)

print(
    timeit.timeit(
        "sum_digits(numbers)",
        globals=globals(),
        number=1
    )
)

0.28875587500078836
0.0044607500021811575

The second call to sum_digits(numbers) is much quicker than the first one because it uses the cached value.

Let’s now explain the code above in more detail. First, notice that we create the dictionary sum_digits.my_cache after defining the function even though we use it in the function definition.

The sum_digits() function checks whether the argument passed to the function is already one of the keys in the sum_digits.my_cache dictionary. The sum of all digits is only evaluated if the argument is not already in the cache.

Since the argument we use when calling the function serves as the key in the dictionary, it must be a hashable data type. A list is not hashable, so we can’t use it as the key in a dictionary. For example, let’s try replacing numbers with a list instead of a tuple—this will raise a TypeError:

# ...

numbers = [random.randint(1, 1000) for _ in range(1_000_000)]

# ...

Traceback (most recent call last):
...
TypeError: unhashable type: 'list'

Creating a cache manually is great for learning purposes, but let’s now explore faster ways to do it.

Python caching with functools.lru_cache()

Python has had the lru_cache() decorator since version 3.2. The “lru” at the beginning of the function name stands for "least recently used." We can think of cache as a box for storing frequently used things—when it fills up, the LRU strategy throws away the item we haven't used in the longest time to make space for something new.

Let's decorate our sum_digits() function with @functools.lru_cache:

import functools
import random
import timeit

@functools.lru_cache
def sum_digits(numbers):
    return sum(
        int(digit) for number in numbers for digit in str(number)
    )

numbers = tuple(random.randint(1, 1000) for _ in range(1_000_000))

print(
    timeit.timeit(
        "sum_digits(numbers)",
        globals=globals(),
        number=1
    )
)

print(
    timeit.timeit(
        "sum_digits(numbers)",
        globals=globals(),
        number=1
    )
)

0.28326129099878017
0.002184917000704445

Thanks to caching, the second call takes significantly less time to run.

By default, the cache stores the first 128 values calculated. Once all 128 places are full, the algorithm deletes the least recently used (LRU) value to make space for new values.

We can set a different maximum cache size when we decorate the function using the maxsize parameter:

import functools
import random
import timeit

@functools.lru_cache(maxsize=5)
def sum_digits(numbers):
    return sum(
        int(digit) for number in numbers for digit in str(number)
    )

# ...

In this case, the cache only stores five values. We can also set the maxsize argument to None if we don't want to limit the cache's size.

Python caching with functools.cache()

Python 3.9 includes a simpler and faster caching decorator—functools.cache(). This decorator has two main characteristics:

• It doesn’t have a maximum size—it’s similar to calling functools.lru_cache(maxsize=None).
• It stores all the function calls and their results (it doesn’t use the LRU strategy). This is suitable for functions with relatively small outputs or when we don't need to worry about cache size limitations.

Let’s use the @functools.cache decorator on the sum_digits() function:

import functools
import random
import timeit

@functools.cache
def sum_digits(numbers):
    return sum(
        int(digit) for number in numbers for digit in str(number)
    )

numbers = tuple(random.randint(1, 1000) for _ in range(1_000_000))

print(
    timeit.timeit(
        "sum_digits(numbers)",
        globals=globals(),
        number=1
    )
)

print(
    timeit.timeit(
        "sum_digits(numbers)",
        globals=globals(),
        number=1
    )
)

0.16661812500024098
0.0018135829996026587

Decorating sum_digits() with @functools.cache is equivalent to assigning sum_digits to functools.cache():

# ...

def sum_digits(numbers):
    return sum(
        int(digit) for number in numbers for digit in str(number)
    )

sum_digits = functools.cache(sum_digits)

Note that we can also use a different import style:

from functools import cache

This way, we can decorate our functions using just @cache.

Other caching strategies

Python's own tools implement the LRU caching strategy, where the least recently used entries are deleted to make space for new values.

Let's have a look at a few other caching strategies:

• First-in, first-out (FIFO): When the cache is full, the first item added is removed to make space for new values. The difference between LRU and FIFO is that LRU keeps recently used items in the cache, while FIFO discards the oldest item regardless of use.
• Last-in, first-out (LIFO): The most recently added item is removed when the cache is full. Imagine a stack of plates in a cafeteria. The plate we put on the stack most recently (last in) is the one we'll take off first (first out).
• Most-recently used (MRU): The value that's been used most recently is discarded when space is needed in the cache.
• Random replacement (RR): This strategy randomly discards an item to make space for a new one.

These strategies can also be combined with measures of the valid lifetime—this refers to how long a piece of data in the cache is considered valid or relevant. Imagine a news article in a cache. It might be frequently accessed (LRU would keep it), but after a week, the news might be outdated.

Python Caching: Common Use Cases

So far, we’ve used simplistic examples for learning purposes. However, caching has many real-world applications.

In data science, we often execute repeated operations on large datasets. Using cached results reduces the time and cost associated with performing the same calculations repeatedly on the same datasets.

We can also use caching for saving external resources such as web pages or databases. Let’s consider an example and cache a DataCamp article. But first we’ll need to install the third-party requests module by executing the following line in the terminal:

$ python -m pip install requests

Once requests is installed, we can try the following code, which attempts to fetch the same DataCamp article twice while using the @lru_cache decorator:

import requests
from functools import lru_cache

@lru_cache(maxsize=10)
def get_article(url):
    print(f"Fetching article from {url}")
    response = requests.get(url)
    return response.text

print(get_article("https://www.datacamp.com/tutorial/decorators-python"))
print(get_article("https://www.datacamp.com/tutorial/decorators-python"))

Fetching article from https://www.datacamp.com/tutorial/decorators-python
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>...
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>...

As a side note, we truncated the output because it’s very long. Notice, however, that only the first call to get_article() prints the phrase Fetching article from {url}.

This is because the webpage is accessed only the first time the call is made. The result is stored in the function's cache. When we request the same webpage the second time, the data stored in the cache is returned instead.

Caching ensures there are no unnecessary delays when fetching the same data repeatedly. External APIs often also have rate limits and costs associated with fetching data. Caching reduces APIs costs and the likelihood of hitting rate limits.

Another common use case is in machine learning applications where several expensive computations need to be repeated. For example, if we need to tokenize and vectorize a text before using it in a machine learning model, we can store the output in a cache. This way we won’t need to repeat the computationally-expensive operations.

Common Challenges When Caching in Python

We've learned about the advantages of caching in Python. There are also some challenges and drawbacks to keep in mind when implementing a cache:

• Cache invalidation and consistency: Data may change with time. Therefore, values stored in a cache may also need to be updated or removed.
• Memory management: Storing large amounts of data in a cache requires memory, and this can cause performance issues if a cache grows indefinitely.
• Complexity: Adding caches introduces complexity to the system when creating and maintaining the cache. Often, the benefits outweigh these costs, but this increased complexity could lead to bugs that are hard to find and correct.

Conclusion

We can use caching to optimize performance when computationally intensive operations are repeated on the same data.

Python has two decorators to create a cache when calling functions: @lru_cache and @cache in the functools module.

We need to ensure, however, that we keep the cache up to date and we properly manage the memory.

If you want to learn more caching and Python, check out this six-course Python Programming skill track.