Skip to main content

Python Environment Variables: What You Need to Know

Learn the ins and outs of managing Python environment variables with os and python-dotenv libraries.
Apr 29, 2024  · 9 min read

Environment variables are powerful mechanisms for storing configuration details of your development project flexibly and securely. They store key-value pairs of data accessible by programming languages such as Python and keep that information hidden from prying eyes. In this tutorial, you will learn must-know concepts and techniques to create and manage environment variables in Python.

What Are Environment Variables?

Environment variables are name-value pairs stored somewhere safe in your operation system. Most often, they look like this:

API_KEY=1akn4nka-aqwej3

In Python, you can use such variables to store sensitive information related to your development project. Sensitive information can be:

  • API keys to access others’ applications
  • Username and passwords to log in to applications through scripts
  • Database credentials
  • anything that might cause a security issue if exposed accidentally.

For example, if you write a Python script that openly uses your AWS credentials and accidentally commit that script to GitHub, there is a chance malicious parties discover it and significantly increase your AWS bill (this is known to happen).

Another benefit of environment variables is configurability. You can easily adjust settings (database URLs, file paths) by modifying environment variables without changing your code. This feature is especially helpful if you use the same setting in multiple parts of your project.

So, let’s learn how to work with them in Python.

How To Retrieve Environment Variables in Python’s os Module

To retrieve existing environment variables in your system, you can use the os module. It has the .getenv function to retrieve variables:

import os
os.getenv("USER")
'bexgboost'

Above, I am retrieving the system user name, which is built into all systems. There are many others like the home path:

os.getenv("HOME")
'/home/bexgboost'

Or the path to your Conda executable:

os.getenv("CONDA_EXE")
'/home/bexgboost/anaconda3/bin/conda'

So, when you run conda commands, that's how your terminal knows which application to run.

But what happens if you try to retrieve a variable that doesn't exist:

os.getenv("MYDATABASE")

The .getenv() function doesn't return anything, but we can change that behavior. You can use its default parameter to return a custom value if a variable doesn't exist:

os.getenv("COMPUTER_PASSWORD", default="Nope!")
'Nope!'

The .getenv() function is the best method to retrieve existing variables.

However, in other sources, you might see different methods like the .environ attribute, which returns a dictionary containing all environment variables:

current_directory = os.environ.get("PWD", None)
current_directory
'/home/bexgboost/articles/2024/4_april'

Since it is a dictionary, you can use brackets notation (not recommended) or the .get() function to retrieve a value.

os also has access to perhaps the most important system variable called PATH. PATH holds the absolute paths to all the executables installed on your system, so it is pretty long:

os.getenv("PATH")[:46]
'/home/bexgboost/.local/bin:/home/bexgboost/bin'

Each path in PATH is separated by a colon. Let's count the length of PATH on my system:

def count_path_items():
   items = os.getenv("PATH").split(":")

   return len(items)

count_path_items()
79

79! Not bad.

Using the python-dotenv Library to Manage Environment Variables in Python Effectively

Setting custom environment variables

Now that we know how to extract existing variables, let’s see how to set custom ones. The first method is using the os.environ dictionary:

# Create a new environment variable
os.environ["NEW_VARIABLE"] = "42"  # Always expects a string value

os.getenv("NEW_VARIABLE")
'42'

However, this method isn’t very useful as all new variables will be lost once the current session ends.

Fortunately, there is a more lasting method of storing environment variables by using .env files (pronounced dot-env). These files have the same syntax, which means they will work on any operating system. Here is what a sample .env file looks like:

CUSTOM_API_LINK=https://myapi.com/v1/api
SNOWFLAKE_USERNAME=bexgboost
MYSQL_DATABASE_PASSWORD=as3ndf03

For now, it defines only three variables. To read this file and extract its contents, we will use a library called python-dotenv:

$ pip install python-dotenv

Then, in a Python script or a Jupyter Notebook we will import its load_dotenv function and call it:

from dotenv import load_dotenv

load_dotenv()
True

If it can’t find a .env file within the current directory, it will search all parent directories and return True if found.

I stored the above .env file in my home directory ~/.env; that's why load_dotenv could find it.

To create your own files, you can use the touch command in the terminal or use a text editor like VSCode:

$ touch ~/.env

The syntax of .env files is pretty flexible. For example, you can set variables with multi-line values (must have quotes):

LONG_ENV="This is an environment
variable with a long message"

Use escape characters:

LONG_ENV="This value uses an \"escape character"

You can also add comments to the file for future reference:

# You can also add comments anywhere with a hashtag
CUSTOM_API_LINK=https://myapi.com/v1/api
SNOWFLAKE_USERNAME=bexgboost
MYSQL_DATABASE_PASSWORD=as3ndf03

It is also possible to use the value of one variable inside another using the ${VAR_NAME} syntax:

FIRST_NAME=John
LAST_NAME=Doe
FULL_NAME="I am ${FIRST_NAME} ${LAST_NAME}"

Retrieving custom environment variables with python-dotenv

Retrieving custom variables is the same as before - just use the os module:

import os

os.getenv("CUSTOM_API_LINK")
'https://myapi.com/v1/api'
os.getenv("SNOWFLAKE_USERNAME")
'bexgboost'

Make sure that you call load_dotenv first.

If your .env file is located somewhere unreachable to load_dotenv, you can pass a direct path to it like below:

load_dotenv("/home/bexgboost/somewhere/unreachable/.env")
True

Working with .env files in Jupyter

Once you install Python-dotenv, you don't have to import it into Python if you are using Jupyter. The library comes with a Jupyter magic method that automatically loads it:

# Run this anywhere in the notebook to load EVs from .env
%load_ext dotenv
%dotenv

Then, you can use the os module again:

import os

os.getenv("SNOWFLAKE_USERNAME")
'bexgboost'

Best Practices For Using Environment Variables in Python

In this section, we will list some best practices to ensure environment variables stay safe and fully usable.

1. Always add .env files to .gitignore so that you don't accidentally commit them to GitHub.

# Create .gitignore if it doesn't exist
$ touch .gitignore
$ echo ".env" >> .gitignore

2. While defining variables use descriptive, all UPPERCASE names.

# Good
DEBUG=True
LOGGING_VERBOSITY=3

# Bad
debug=True
LV=3

3. Create a single .env file to store system-wide variables you need in every project.

$ touch ~/.env

4. Control the scope of your .env files. Create separate files for different parts of your development workflow like below:

$ touch development/.env.development
$ touch testing/.env.testing
$ touch production/.env.production

The load_dotenv function will recognize them.

If you create multiple .env files with different names, change your .gitignore file so that all of them are excluded from git indexing:

# Pay attention to the asterisk *
$ echo ".env*" >> .gitignore

Conclusion

In this article, we have learned all essential concepts and techniques to effectively manage environment variables in Python. We discovered how to retrieve existing variables with the Python OS module and how to create custom values with python-dotenv library. We also learned how to write .env files that conform to best practices.

If you want to learn more about Python software engineering, check out our course, Software Engineering Principles in Python.

You can also check out our Python Programming Skill Track, which will take you from a beginner to an advanced Pythonista with hands-on exercises and projects.


Photo of Bex Tuychiev
Author
Bex Tuychiev
LinkedIn

I am a data science content creator with over 2 years of experience and one of the largest followings on Medium. I like to write detailed articles on AI and ML with a bit of a sarcastıc style because you've got to do something to make them a bit less dull. I have produced over 130 articles and a DataCamp course to boot, with another one in the makıng. My content has been seen by over 5 million pairs of eyes, 20k of whom became followers on both Medium and LinkedIn. 

Topics

Continue Your Python Journey Today!

course

Software Engineering Principles in Python

4 hr
47.6K
Learn about modularity, documentation, and automated testing to help you solve data science problems more quickly and reliably.
See DetailsRight Arrow
Start Course
See MoreRight Arrow
Related

blog

A Data Scientist's Guide to Environment Variables

As a data scientist, you might have to configure environment variables for data science projects locally or globally, on a platform. This guide will show you all you need to know!
Eric Ma's photo

Eric Ma

9 min

blog

Anaconda vs Python: Exploring Their Key Differences

Learn all about the key differences between Python and Anaconda in this complete guide.
Austin Chia's photo

Austin Chia

9 min

tutorial

Python Setup: The Definitive Guide

In this tutorial, you'll learn how to set up your computer for Python development, and explain the basics for having the best application lifecycle.

J. Andrés Pizarro

15 min

tutorial

Virtual Environment in Python

In this tutorial, you'll learn about the Virtual Environment and two different ways of creating it.

Olivia Smith

7 min

tutorial

Encapsulation in Python Object-Oriented Programming: A Comprehensive Guide

Learn the fundamentals of implementing encapsulation in Python object-oriented programming.
Bex Tuychiev's photo

Bex Tuychiev

11 min

tutorial

Python Private Methods Explained

Learn about private methods in Python, their syntax, how and when to use them in your projects using examples, and the best practices.
Arunn Thevapalan's photo

Arunn Thevapalan

9 min

See MoreSee More