Free access is closing! 700+ free courses are waiting.
Ends in:
Start Now
Gymnasium needs specific versions (not the latest releases) of various dependency programs like NumPy and PyTorch. Thus, we recommend creating a fresh Conda or venv environment or a fresh notebook to install, use Gymnasium, and run RL programs.
You can use this DataLab workbook to follow along with the tutorial.
To install Gymnasium on a server or local machine, run:
To install using a Notebook like Google’s Colab or DataCamp’s DataLab, use:
The above command installs Gymnasium and the correct versions of dependencies.
As of November 2024, Gymnasium includes over 60 inbuilt environments. To browse available inbuilt environments, use the gym.envs.registry.all() function, as illustrated in the example below:
You can also visit the Gymnasium homepage. The left-hand column has links to all the environments. The webpage of each environment includes details about it, such as actions, states, etc.
Environments are organized into categories like Classic Control, Box2D, and more. Below, I list some of the common environments in each group:
In addition to the built-in environments, Gymnasium can be used with many external environments using the same API.
We’ll use one of the canonical Classic Control environments in this tutorial. To import a specific environment, use the .make() command and pass the name of the environment as an argument. For example, to create a new environment based on CartPole (version 1), use the command below:
In a nutshell, Reinforcement Learning consists of an agent (like a robot) that interacts with its environment. A policy decides the agent’s actions. Depending on the agent’s actions, the environment gives a reward (or penalty) at each timestep. The agent uses RL to figure out the optimal policy that maximizes the total rewards the agent earns.
The following are the key components of an RL environment:
The observation is the information that the agent gathers about the environment. An agent, for example, a robot, could collect environmental information using sensors. Ideally, the agent should be able to observe the complete state, which describes all the aspects of the environment. In practice, the agent uses its observations as a proxy for the state. Thus, the observations decide the agent’s actions.
A space is analogous to a mathematical set. The space of items X includes all possible instances of X. The space of X also defines the structure (syntax and format) of all items of type X. Each Gymnasium environment has two spaces, the action space, action_space, and the observation space, observation_space. Both the action and observation spaces derive from the parent gymnasium.spaces.Space superclass.
The observation space is the space that includes all possible observations. It also defines the format in which observations are stored. The observation space is typically represented as an object of datatype Box. This is an ndarray which describes the parameters of the observations. The box specifies the bounds of each dimension. You can view the observation space for an environment using the observation_space method:
In the case of the CartPole-v1 environment, the output looks like the example below:
In this example, the CartPole-v1 observation space has 4 dimensions. The 4 elements of the observation array are:
To see an example of an individual observation array, use the .reset() command.
In the case of the CartPole-v1 environment, the output looks like the example below:
The four elements of this array correspond to the four observed quantities (cart position, cart velocity, pole angle, pole angular velocity ), as explained earlier.
The action space includes all possible actions that the agent can take. The action space also defines the format in which actions are represented. You can view the action space for an environment using the action_space method:
In the case of the CartPole-v1 environment, the output looks like the example below:
In the case of the CartPole-v1 environment, the action space is discrete. There are a total of two actions that the agent can take:
In the previous sections, we explored the basic concepts of RL and Gymnasium. This section shows you how to use Gymnasium to build an RL agent.
The first step is to create an instance of the environment. To create new environments, use the .make() method.
The agent’s interactions change the environment’s state. The .reset() method resets the environment to an initial state. By default, the environment is initialized to a random state. You can use a SEED parameter with the .reset() method to initialize the environment to the same state every time the program is run. The code below shows how to do this:
The sampling of actions also involves randomness. To control this randomness and get a fully reproducible training path, we can seed the random generators of NumPy and PyTorch:
In each step in a Markov process, the agent can randomly choose an action and explore the environment until it arrives at a terminal state. By choosing actions at random:
Training the agent to optimize the selection of actions based on previous experiences (of interacting with the environment) is more efficient to maximize long-term rewards.
The untrained agent starts with random actions based on a randomly initialized policy. This policy is typically represented as a neural network. During training, the agent learns the optimal policy that maximizes the rewards. In RL, the training process is also called policy optimization.
There are various methods of policy optimization. The Bellman equations describe how to calculate the value of RL policies and determine the optimal policy. In this tutorial, we’ll use a simple technique called policy gradients. Other methods exist, such as Proximal Policy Optimization (PPO).
To build an RL agent that uses policy gradients, we create a neural network to implement the policy, write functions to calculate the returns and loss from the stepwise rewards and the action probabilities, and iteratively update the policy using standard backpropagation techniques.
We use a neural network to implement the policy. Because CartPole-v1 is a simple environment, we use a neural network with:
Thus, the function of the policy network is to map observed states to actions. Given an input observation, it predicts the right action. The code below implements the policy network:
class PolicyNetwork(nn.Module):
def __init__(self, input_dim, hidden_dim, output_dim, dropout):
super().__init__()
self.layer1 = nn.Linear(input_dim, hidden_dim)
self.layer2 = nn.Linear(hidden_dim, output_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, x):
x = self.layer1(x)
x = self.dropout(x)
x = F.relu(x)
x = self.layer2(x)
return xAs mentioned, in each step of the Markov process, the environment gives a reward based on the agent’s action and state. The goal in RL is to maximize the total return.
In practice, while accumulating rewards, it is common to:
The code below shows how to do this:
def calculate_stepwise_returns(rewards, discount_factor):
returns = []
R = 0
for r in reversed(rewards):
R = r + R * discount_factor
returns.insert(0, R)
returns = torch.tensor(returns)
normalized_returns = (returns - returns.mean()) / returns.std()
return normalized_returnsThe forward pass consists of running the agent based on the current policy until it reaches a terminal state and collecting the stepwise rewards and action probabilities. The steps below explain how to implement the forward pass:
.step() function to iteratively run the agent in the environment until it terminates:Softmax function to estimate the probability of taking the predicted action.def forward_pass(env, policy, discount_factor):
log_prob_actions = []
rewards = []
done = False
episode_return = 0
policy.train()
observation, info = env.reset()
while not done:
observation = torch.FloatTensor(observation).unsqueeze(0)
action_pred = policy(observation)
action_prob = F.softmax(action_pred, dim = -1)
dist = distributions.Categorical(action_prob)
action = dist.sample()
log_prob_action = dist.log_prob(action)
observation, reward, terminated, truncated, info = env.step(action.item())
done = terminated or truncated
log_prob_actions.append(log_prob_action)
rewards.append(reward)
episode_return += reward
log_prob_actions = torch.cat(log_prob_actions)
stepwise_returns = calculate_stepwise_returns(rewards, discount_factor)
return episode_return, stepwise_returns, log_prob_actionsThe loss represents the quantity on which we apply gradient descent. The goal in RL is to maximize the returns. So, we use the expected return value as a proxy for the loss. The expected return value is calculated as the product of the stepwise expected returns and the log probability of the stepwise actions. The code below calculates the loss:
def calculate_loss(stepwise_returns, log_prob_actions):
loss = -(stepwise_returns * log_prob_actions).sum()
return lossTo update the policy, you run backpropagation with respect to the loss function. The update_policy() method below invokes the calculate_loss() method. It then runs backpropagation on this loss to update the policy parameters, i.e., model weights of the policy network.
def update_policy(stepwise_returns, log_prob_actions, optimizer):
stepwise_returns = stepwise_returns.detach()
loss = calculate_loss(stepwise_returns, log_prob_actions)
optimizer.zero_grad()
loss.backward()
optimizer.step()
return loss.item()Updating the policy based on the gradient of the returns is called the policy gradient method.
We now have all the components needed to train and evaluate the policy. We implement the training loop as explained in the following steps:
Before starting, we declare the hyperparameters, instantiate a policy, and create an optimizer:
MAX_EPOCHS is the maximum number of iterations we are prepared to run to train the policy. DISCOUNT_FACTOR decides the relative significance of rewards from future time steps. A discount factor of 1 means all rewards are equally important, while a value of 0 means only the reward from the current time step is important. N_TRIALS is the number of episodes over which we average the returns to evaluate the agent’s performance. We decide the training is successful if the average return over N_TRIALS episodes is above the threshold. REWARD_THRESHOLD: If the policy can achieve a return greater than the threshold, it is considered successful. DROPOUT decides the fraction of the weights that should be randomly zeroed. The dropout function randomly sets a fraction of the model weights to zero. This reduces reliance on specific neurons and prevents overfitting, making the network more robust.LEARNING_RATE decides how much the policy parameters can be modified in each step. The update to the parameters in each iteration is the product of the gradient and the learning rate. PolicyNetwork class (implemented earlier). To train the policy, we iteratively run the training steps till the average return (over N_TRIALS) is greater than the reward threshold:
N_TRIALS exceeds the reward threshold. The code below implements these steps:
def main():
MAX_EPOCHS = 500
DISCOUNT_FACTOR = 0.99
N_TRIALS = 25
REWARD_THRESHOLD = 475
PRINT_INTERVAL = 10
INPUT_DIM = env.observation_space.shape[0]
HIDDEN_DIM = 128
OUTPUT_DIM = env.action_space.n
DROPOUT = 0.5
episode_returns = []
policy = PolicyNetwork(INPUT_DIM, HIDDEN_DIM, OUTPUT_DIM, DROPOUT)
LEARNING_RATE = 0.01
optimizer = optim.Adam(policy.parameters(), lr = LEARNING_RATE)
for episode in range(1, MAX_EPOCHS+1):
episode_return, stepwise_returns, log_prob_actions = forward_pass(env, policy, DISCOUNT_FACTOR)
_ = update_policy(stepwise_returns, log_prob_actions, optimizer)
episode_returns.append(episode_return)
mean_episode_return = np.mean(episode_returns[-N_TRIALS:])
if episode % PRINT_INTERVAL == 0:
print(f'| Episode: {episode:3} | Mean Rewards: {mean_episode_return:5.1f} |')
if mean_episode_return >= REWARD_THRESHOLD:
print(f'Reached reward threshold in {episode} episodes')
breakLastly, invoke the main() function to train the policy:
Use this DataLab workbook to run the above algorithm directly and solve the CartPole environment using RL.
Having demonstrated how to implement an RL algorithm, we now discuss some advanced techniques commonly used in practice.
Implementing RL algorithms from scratch is a long and difficult process, especially for complex environments and state-of-the-art policies.
A more practical alternative is to use software like Stable Baselines3. It comes with tried and tested implementations of RL algorithms. It includes pre-trained agents, training scripts, evaluation tools, and modules to plot graphs and record videos.
Ray RLib is another popular tool for RL. RLib is designed as a scalable solution, making it easy to implement RL algorithms on multi-GPU systems. It also supports multi-agent RL, which opens up new possibilities like:
With both RLib and Stable Baselines3, you can import and use environments from OpenAI Gymnasium.
Environments packaged with Gymnasium are the right choice for testing new RL strategies and training policies. However, for most practical applications, you need to create and use an environment that accurately reflects the problem you want to solve. You can use Gymnasium to create a custom environment. The advantage of using Gymnasium custom environments is that many external tools like RLib and Stable Baselines3 are already configured to work with the Gymnasium API structure.
To create a custom environment in Gymnasium, you need to define:
reset() function is called). step() function is called). To learn more, follow the Gymnasium guide on creating custom environments.
The code in this tutorial showed how to implement the policy gradient algorithm in the CartPole environment. This is a simple environment with a discrete action space. To understand RL better, we advise you to apply the same policy gradient algorithm (and other algorithms, like PPO) in other environments.
For example, the Pendulum environment has a continuous action space. It consists of a single input represented as a continuous variable - the (magnitude and direction of the) torque applied to the pendulum in any given state. This torque can take on any value between -2 and +2.
Experimenting with different algorithms in various environments helps you better understand different kinds of RL solutions and their challenges.
RL environments often consist of robots, pendulums, mountain cars, video games, etc. Visualizing the agent’s actions within the environment gives a better intuitive understanding of the policy’s performance.
In Gymnasium, the env.render() method visualizes the agent’s interactions with the environment. It graphically displays the current state of the environment—game screens, the position of the pendulum or cart pole, etc. Visual feedback of the agent’s actions and the environment’s responses helps monitor the agent's performance and progress through the training process.
There are four render modes: “human”, “rgb_array”, “ansi”, and “rgb_array_list”. To visualize the agent’s performance, use the “human” render mode. The render mode is specified when the environment is initialized. For example:
env = gym.make(‘CartPole-v1’, render_mode=’human’)
To perform the rendering, involve the .render() method after each action performed by the agent (via calling the .step() method). The pseudo-code below illustrates how to do this:
while not done:
…
step, reward, terminated, truncated, info = env.step(action.item())
env.render()
…
Gymnasium makes it easy to interface with complex RL environments. However, is a continuously updated software with many dependencies. So, watching out for a few common types of errors is essential.
CartPole-v1 and CartPole-v0. Although the behavior of the environment is the same across both versions, some of the parameters, like the episode length, reward threshold, etc., can be different. A policy trained on one version might not perform as well on another version of the same environment. You need to update the training parameters and retrain the policy for each environment version. numpy 2.1.3 and torch 2.5.1. However, Gymnasium works best with torch 1.13.0 and numpy 1.23.3. You might encounter issues if you install Gymnasium into an environment with these dependencies pre-installed. We recommend installing and working with Gymnasium in a fresh Conda environment. In this tutorial, we explored the basic principles of RL, discussed Gymnasium as a software package with a clean API to interface with various RL environments, and showed how to write a Python program to implement a simple RL algorithm and apply it in a Gymnasium environment.
After understanding the basics in this tutorial, I recommend using Gymnasium environments to apply the concepts of RL to solve practical problems such as taxi route optimization and stock trading simulations.
Gymnasium comes with many pre-built environments for testing and developing RL algorithms. It standardizes the interface for these environments and new custom environments. This makes it easier to implement and evaluate RL algorithms.
No. In addition to over 60 pre-built environments, Gymnasium allows you to create new environments by declaring their state and action spaces, and by defining how the environment to respond to various actions.
In a nutshell, Gymnasium, maintained by the Farama Foundation, is the new and upgraded version of Gym, which OpenAI no longer maintains. Gymnasium is expected to work as a drop-in replacement for Gym 0.26.2.
In policy-based methods, like policy gradients, the agent directly modifies the policy parameters to find which policy leads to the maximum expected return.
In value-based methods, the agent is trained on the value function. Learning the optimal value function helps to derive the optimal policy.
Writing code from scratch is the best way to understand different RL algorithms. For production use, there are tools like Stable Baselines3, RLib, and CleanRL. These tools include pre-built deployments of common RL algorithms, which you must configure and finetune depending on the environment.

Arun is a former startup founder who enjoys building new things. He is currently exploring the technical and mathematical foundations of Artificial Intelligence. He loves sharing what he has learned, so he writes about it.
In addition to DataCamp, you can read his publications on Medium, Airbyte, and Vultr.

Tutorial
Tutorial
Tutorial
Learn more about reinforcement learning with these courses!
Curs
Începe-ți călătoria în reinforcement learning! Învață cum agenții pot învăța să rezolve medii prin interacțiuni.
Curs
Învață și folosește algoritmi puternici de Deep Reinforcement Learning, inclusiv tehnici de rafinare și optimizare.
Curs
Învață cum să faci modelele GenAI să reflecte cu adevărat valorile umane, dobândind experiență practică cu LLM-uri avansate.
Reinforcement Learning (RL) is one of the three main machine learning paradigms, the other two being supervised and unsupervised learning. In RL, an agent learns to interact with its environment to maximize the cumulative rewards. It learns the optimal action under different environmental conditions through trial and error. Reinforcement Learning with Human Feedback (RLHF) allows the agent to modify behavior based on human inputs at each step.
RL solves problems like self-driving cars, automated trading, computer players in video games, training robots, and more. When deep neural networks are used to apply RL algorithms, it is called Deep Reinforcement Learning.
In this tutorial, I’ll show you how to get started with Gymnasium, an open-source Python library for developing and comparing reinforcement learning algorithms. I'll demonstrate how to set it up, explore various RL environments, and use Python to build a simple agent to implement an RL algorithm.
What is Gymnasium?
Gymnasium is an open-source Python library designed to support the development of RL algorithms. To facilitate research and development in RL, Gymnasium provides:
Developers can build RL algorithms and use API calls for tasks like:
OpenAI’s Gym versus Farama’s Gymnasium
OpenAI hasn’t committed significant resources to developing Gym because it was not a business priority for the company. The Farama Foundation was created to standardize and maintain RL libraries over the long term. Gymnasium is the Farama Foundation’s fork of OpenAI’s Gym. Gymnasium 0.26.2 is a drop-in replacement for Gym 0.26.2. With the fork, Farama aims to add functional (in addition to class-based) methods for all API calls, support vector environments, and improve the wrappers. The overall goal is to make the framework cleaner and more efficient.