Skip to main content
HomeTutorialsPython

Getting Started with Python HTTP Requests for REST APIs

Learn how to use Python HTTP requests to interact with REST APIs. This guide covers GET and POST requests, examples, and best practices for API integration.
Updated Jul 23, 2024  · 15 min read

Practice performing an HTTP request in Python with this hands-on exercise.

Application Programming Interfaces (APIs) are software mediators; their job is to permit applications to communicate with each other. These subtle mediators appear in everyday life whether you know it or not. For example, if you’ve sent an instant message today, you’ve used an API. 

More specifically, APIs allow people to send and retrieve data using code. However, It’s more common to use APIs to retrieve data; for instance, you can read this blog post because your web browser retrieved the data that makes up this page from the DataCamp server. 

But web servers don’t randomly send data. That would be like going to a restaurant, and the waiter randomly brings you a meal. A request must be made to the server to retrieve data before it responds with data. This is true for the waiter in a restaurant, and if you’d like to retrieve some data from an API, you make an API request to a server, and it will respond with the appropriate data.  

The requests library is the de facto industry standard for sending HTTP requests in Python. There is also Python’s built-in urllib, but Pythonistas tend to prefer the Python requests API due to its readability and the fact that it supports fully restful APIs—something we will touch on a little later. 

The requests library isolates all the challenges of making requests behind a straightforward API, allowing you to concentrate on communicating with services and consuming data in your application.   

In this article, we will walk through some of the core components of the requests library and provide some code examples to help you get started.

Get certified in your dream Data Scientist role

Our certification programs help you stand out and prove your skills are job-ready to potential employers.

Get Your Certification
Timeline mobile.png

Run and edit the code from this tutorial online

Run code

Quick Answer: Making GET and POST requests in Python

In a rush? Here's the Python syntax for making a simple GET and POST request:

1. GET request

import requests

# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts/1"

# A GET request to the API
response = requests.get(url)

# Print the response
print(response.json())

2. POST request

import requests

# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts"

# Data to be sent
data = {
    "userID": 1,
    "title": "Making a POST request",
    "body": "This is the data we created."
}

# A POST request to the API
response = requests.post(url, json=data)

# Print the response
print(response.json())

If you'd like to dive deeper into making HTTP requests with Python and learn more about REST APIs, keep reading!

What Is a REST API?

We’ve established APIs are software mediators. Another way to think of them is as a type of software interface that grants other applications access to specific data and methods. 

One of the most popular architectures used to build APIs is the REpresentational State Transfer (REST) pattern. The REST architectural design enables the client and server to be implemented independently of one another without being aware of each other. This means code on either side can be changed without worrying about how the change will affect the other. 

REST APIs, therefore, follow a set of guidelines designed to simplify communications between software, thereby making the process of accessing data more straightforward and logical. Don’t worry if you don’t know these guidelines; you don’t need to know them to get started – what you do need to know is how data is exposed from REST services. 

Data from REST web services are exposed to the internet through a public URL, which can be accessed by sending an HTTP request.  

HTTP Request Methods Overview

Let’s rewind to our restaurant analogy; to order food at a restaurant, the waiter will approach you, and you say what you want. The waiter then passes your request to the chef, who makes the meal and passes it to the waiter to return to you. In other words, the chef wouldn’t cook your meal until your request has been sent. 

REST APIs are the same: they listen for HTTP request methods before taking action. HTTP defines a set of request methods that tell the API what operations to perform for a given resource. It specifies how to interact with the resources located at the provided endpoint. 

There are several HTTP methods, but five are commonly used with REST APIs: 

HTTP Method

Description

GET

Retrieve data

POST

Create data

PUT

Update existing data

PATCH

Partially update existing data

DELETE

Delete data

It’s highly likely you will be performing GET requests more than any other method in data analysis and data science. This is because it’s the necessary method to access certain datasets. Learn more about this with DataCamp’s Intermediate Importing Data in Python course.

When you make a request to a web server, the API returns a response. Attached to the response is an HTTP status code. The purpose of the status code is to provide additional information about the response so the client knows the type of request being received.

What Are API Endpoints?

A URL defines the data you interact with on a web server. Much like a web page URL is connected to a single page, an endpoint URL is linked to particular resources within an API. Therefore, an endpoint may be described as a digital location where an API receives inquiries about a particular resource on its server—think of it as the other end of a communication channel. 

To add more context, REST APIs expose a set of public URLs that may be requested by client applications to access the resources of the web service. The public URLs exposed by the REST API are known as “endpoints.”

Using Python to Make HTTP Requests

The Python requests module enables developers to write code to interact with REST APIs. It allows them to send HTTP requests using Python without having to worry about the complexities that typically come with carrying out such tasks (i.e., manually adding query strings to URLs, form-encoding PUT and POST data, etc.). 

Despite being considered the de facto standard for making HTTP requests in Python, the requests module is not part of Python’s standard library – it must be installed. 

The most straightforward way to install the requests module is with pip

python -m pip install requests

It’s always recommended that the Python packages required for different projects be managed using virtual environments; this way, the packages for one project will not interfere and break system tools in other projects because they are isolated – instead of being installed globally. 

Now we’ve installed the requests module, so let’s see how it works. Follow along with the code in this DataLab workbook

Making a GET request in Python

We’ve already established GET is one of the most common HTTP request methods you’ll encounter when working with REST APIs. It allows you (the client) to retrieve data from web servers. 

It's important to note that GET is a read-only operation, meaning it’s only suitable for accessing existing resources and should not be used to modify them. 

To demonstrate how the request module works, we will use JSONPlaceholder, which is a freely available fake API used for testing and prototyping. 

import requests

# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts/1"

# A GET request to the API
response = requests.get(url)

# Print the response
response_json = response.json()
print(response_json)

"""
{'userId': 1, 'id': 1, 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}
"""

In the code above, we carried out the following: 

  1. Defined the API endpoint to retrieve data from.
  2. Used the requests.get(url) method to retrieve the data from the defined endpoint. 
  3. We used the response.json() method to store the response data in a dictionary object; note that this only works because the result is written in JSON format – an error would have been raised otherwise.   
  4. The last step is to print the JSON response data. 

We can also check the status code returned from the API like this: 

# Print status code from original response (not JSON)
print(response.status_code)

"""
200
"""

You can also pass arguments to a Python GET request. To do this, we must slightly alter the code above. Here’s how the new code looks:

# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts/"

# Adding a payload
payload = {"id": [1, 2, 3], "userId":1}

# A get request to the API
response = requests.get(url, params=payload)

# Print the response
response_json = response.json()

for i in response_json:
    print(i, "\n")

"""
{'userId': 1, 'id': 1, 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}

{'userId': 1, 'id': 2, 'title': 'qui est esse', 'body': 'est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla'}

{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
"""

Here’s what we did differently: 

  1. Changed the API endpoint. Notice it no longer has a ‘1’ at the end. 
  2. Defined the payload in a dictionary.  
  3. Passed the payload to the param argument of the requests.get() method. 
  4. This returned a list object, so we looped through the list and printed each item on a new line.  

Making a POST request in Python

GET requests allow you to retrieve data; POST requests will enable you to create new data. Let’s take a look at how we can create new data on the JSONPlaceholder server.

# Define new data to create
new_data = {
    "userID": 1,
    "id": 1,
    "title": "Making a POST request",
    "body": "This is the data we created."
}

# The API endpoint to communicate with
url_post = "https://jsonplaceholder.typicode.com/posts"

# A POST request to tthe API
post_response = requests.post(url_post, json=new_data)

# Print the response
post_response_json = post_response.json()
print(post_response_json)

"""
{'userID': 1, 'id': 101, 'title': 'Making a POST request', 'body': 'This is the data we created.'}
"""

In the code above, we performed the following: 

  1. Created a new resource we wanted to add to the JSONPlaceholder API.
  2. Defined the endpoint to POST the new data.
  3. Sent a POST request using the requests.post() method. Note that the json parameter was set in the post() method; we do this to tell the API we are explicitly sending a JSON object to the specified URL. 
  4. Used the response.json() method to store the response data in a dictionary object.
  5. The last step is to print the JSON response data. 

But, wait!

Before you read the next bit of code, take 20 seconds to consider what status code the API will return. 

Remember: this time, we created a new resource instead of simply retrieving it. 

Okay, here it goes… 

# Print status code from original response (not JSON)
print(post_response.status_code)

"""
201
"""

Did you get it right? 

Python HTTP Requests Advanced Topics

Making HTTP requests with Python is generally straightforward; however, requiring a more advanced setup or facing problems is sometimes inevitable. Here are some challenges you may encounter and how to solve them. 

Authenticating HTTP requests

Until now, our interactions with the REST API have been pretty straightforward. The JSONPlaceholder API does not require any authentication for you to start interacting with it. However, there are several instances where a REST API may require authentication before access is granted to specific endpoints – especially when you’re dealing with sensitive data. 

For example, if you want to create integrations, retrieve data, and automate your workflows on GitHub, you can do so with the GitHub REST API. However, there are many operations on the GitHub REST API that require authentication, such as retrieving public and private information about authenticated users. 

Here’s a simple workaround using the Python requests module: 

from requests.auth import HTTPBasicAuth

private_url = "https://api.github.com/user"
github_username = "username"
token = "token"

private_url_response = requests.get(
    url=private_url,
    auth=HTTPBasicAuth(github_username, token)
)

private_url_response.status_code

"""
200
"""

In the code above we:

  1. Imported the HTTPBasicAuth object from requests.auth. This object attaches HTTP basic authentication to the given request object—it’s essentially the same as typing your username and password into a website. 
  2. Defined the private URL endpoint to access.
  3. Instantiated a variable with a GitHub username – we anonymized the username for privacy.
  4. Instantiated a variable GitHub with a personal access token for authentication.  
  5. Retrieved data from our endpoint and stored it in the private_url_response variable. 
  6. Displayed the status code.  

Handling HTTP request errors

There are instances where requests made to an API do not go as expected. Several factors on either the client or server side could be at play. Regardless of the cause, the outcome is always the same: the request fails. 

When using REST APIs, it’s always a good idea to make your code resilient. However, before you can write robust code, you must understand how to manage the reported errors when things do not go to plan.  

For this demonstration, let’s return to the JSONPlaceholder API. We will start by writing some code and then explain what is happening.  

# A deliberate typo is made in the endpoint "postz" instead of "posts"
url = "https://jsonplaceholder.typicode.com/postz"

# Attempt to GET data from provided endpoint
try:
    response = requests.get(url)
    response.raise_for_status()
# If the request fails (404) then print the error.
except requests.exceptions.HTTPError as error:
  print(error)

"""
404 Client Error: Not Found for url: https://jsonplaceholder.typicode.com/postz
"""

In the code above: 

  1. We defined the JSONPlace holder endpoint from which to retrieve data, but we made a deliberate typo when constructing the URL – this will raise a 404 error. 
  2. We used Python’s built-in exception handling, to try and except (catch) any errors that occur when attempting to visit the JSONPlaceholder endpoint. Note the raise_for_status() method is what is used to return an HTTPError object when an error occurs during the process.
  3. And lastly, we printed the error that was raised. 

Although we demonstrated how to handle 404 error status codes in this instance, the same format can be used to handle any HTTP status code. 

Dealing with too many redirects

HTTP status codes with the 3xx format indicate that the client was redirected and must perform some additional actions to complete the request. However, this can occasionally lead to an infinite redirect loop.

Python’s requests module provides the TooManyRedirects object to handle this problem, as follows: 

"""
Note: The code here will not raise an error
but the structure is how you would hand a case where there
are multiple redirects
"""

url = "https://jsonplaceholder.typicode.com/posts"

try:
  response = requests.get(url)
  response.raise_for_status()
except requests.exceptions.TooManyRedirects as error:
  print(error)

You can also set the maximum number of redirects as a parameter of your HTTP request method: 

# Solution 2
url = "https://jsonplaceholder.typicode.com/posts"
session = requests.Session()
session.max_redirects = 3
response = session.get(url)

Another option is to disable redirects completely: 

# Solution 3
url = "https://jsonplaceholder.typicode.com/posts"
session = requests.Session()
session.allow_redirects = False
response = session.get(url)

Handling HTTP requests connection errors

These are other sorts of errors you may face when attempting to send requests to a server. There are several reasons you may not receive a response from the server (i.e., DNS failure, refused connection, internet connection issues, etc.), but the outcome is consistent: a connection error is raised. 

You can use the requests modules ConnectionError exception object to catch these issues and handle them accordingly. 

Here’s how the code would look:  

"""
Note: The code here will not raise an error
but the structure is how you would hand a case where there
is a connection error.
"""

url = "https://jsonplaceholder.typicode.com/posts"

try:
  response = requests.get(url)
except requests.ConnectionError as error:
  print(error)

Handling HTTP requests timeout

When the API server accepts your connection but cannot finish your request within the allowed time, you will get a “timeout error.”

We will demonstrate how to handle this case by setting the timeout parameter in the requests.get() method to an extremely small number; this will raise an error, and we will handle that error using the requests.Timeout object. 

url = "https://jsonplaceholder.typicode.com/posts"

try:
  response = requests.get(url, timeout=0.0001)
except requests.Timeout as error:
  print(error)

The most straightforward workaround for timeout errors is to set longer timeouts. Other solutions may include optimizing your requests, incorporating a retry loop into your scripts, or performing asynchronous API calls – a technique that allows your software to begin a potentially long-running activity while being responsive to other events rather than waiting until that task is completed.

Wrap up

In this tutorial, we covered what APIs are and explored a common API architecture called REST. We also looked at HTTP methods and how we can use the Python requests library to interact with web services. 

Check out the following courses to develop your data science skills: 

Become a Python Developer

Gain the programming skills all Python Developers need.

FAQs

How can you handle custom headers in Python HTTP requests?

You can add custom headers to your HTTP requests by passing a dictionary of headers to the headers parameter in your request. For example:

import requests

url = "https://jsonplaceholder.typicode.com/posts"
headers = {
    "Authorization": "Bearer YOUR_ACCESS_TOKEN",
    "Content-Type": "application/json"
}

response = requests.get(url, headers=headers)
print(response.json())

What are the benefits of using a session object in the requests module?

Using a Session object in the requests module provides several benefits, including:

  • Persistent Connections: Maintains a connection across multiple requests, improving performance.
  • Session-wide Settings: Allows you to set headers, cookies, and parameters once for all requests made through the session.
  • Cookie Persistence: Automatically handles cookies, persisting them across requests.

Here's an example:

import requests

session = requests.Session()
session.headers.update({"Authorization": "Bearer YOUR_ACCESS_TOKEN"})

response = session.get("https://jsonplaceholder.typicode.com/posts")
print(response.json())

How do you send a file with a POST request in Python?

You can send a file with a POST request by passing a dictionary to the files parameter. For example:

import requests

url = "https://example.com/upload"
file_path = "/path/to/your/file.txt"

with open(file_path, 'rb') as file:
    files = {'file': file}
    response = requests.post(url, files=files)

print(response.status_code)

How can you configure the requests module to use proxies?

You can configure proxies in the requests module by passing a dictionary to the proxies parameter. For example:

import requests

url = "https://jsonplaceholder.typicode.com/posts"
proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

response = requests.get(url, proxies=proxies)
print(response.json())

How can you handle timeouts when making HTTP requests with the Python requests module?

You can handle timeouts by specifying the timeout parameter in your request. This parameter takes a value in seconds. For example:

import requests

url = "https://jsonplaceholder.typicode.com/posts"

try:
    response = requests.get(url, timeout=5)  # Timeout after 5 seconds
    print(response.json())
except requests.Timeout:
    print("The request timed out")

Photo of Kurtis Pykes
Author
Kurtis Pykes
LinkedIn
Topics

Learn more about Python with these courses!

Course

Introduction to Python

4 hr
5.8M
Master the basics of data analysis with Python in just four hours. This online course will introduce the Python interface and explore popular packages.
See DetailsRight Arrow
Start Course
See MoreRight Arrow
Related

tutorial

FastAPI Tutorial: An Introduction to Using FastAPI

Explore the FastAPI framework and discover how you can use it to create APIs in Python
Moez Ali's photo

Moez Ali

13 min

tutorial

Scraping Reddit with Python and BeautifulSoup 4

In this tutorial, you'll learn how to get web pages using requests, analyze web pages in the browser, and extract information from raw HTML with BeautifulSoup.

Abhishek Kasireddy

16 min

tutorial

Python Tutorial for Beginners

Get a step-by-step guide on how to install Python and use it for basic data science functions.
Matthew Przybyla's photo

Matthew Przybyla

12 min

tutorial

MeetUp API

In this tutorial, you’ll learn how to pull data directly from MeetUp’s API using Python and write it into a JSON.
Keith Singleton's photo

Keith Singleton

8 min

tutorial

GraphQL vs. REST: A Complete Guide

Discover the distinct advantages of REST and GraphQL for modern API design. Learn when to leverage REST for simplicity and caching. Master the flexibility of GraphQL for complex, real-time data needs.
Marie Fayard's photo

Marie Fayard

7 min

tutorial

Python Backend Development: A Complete Guide for Beginners

This complete guide teaches you the fundamentals of Python backend development. Learn basic concepts, frameworks, and best practices to start building web applications.
Oluseye Jeremiah's photo

Oluseye Jeremiah

26 min

See MoreSee More