Course
Practice performing an HTTP request in Python with this hands-on exercise.
Application Programming Interfaces (APIs) are software mediators; their job is to permit applications to communicate with each other. These subtle mediators appear in everyday life whether you know it or not. For example, if you’ve sent an instant message today, you’ve used an API.
More specifically, APIs allow people to send and retrieve data using code. However, It’s more common to use APIs to retrieve data; for instance, you can read this blog post because your web browser retrieved the data that makes up this page from the DataCamp server.
But web servers don’t randomly send data. That would be like going to a restaurant, and the waiter randomly brings you a meal. A request must be made to the server to retrieve data before it responds with data. This is true for the waiter in a restaurant, and if you’d like to retrieve some data from an API, you make an API request to a server, and it will respond with the appropriate data.
The requests library is the de facto industry standard for sending HTTP requests in Python. There is also Python’s built-in urllib, but Pythonistas tend to prefer the Python requests API due to its readability and the fact that it supports fully restful APIs—something we will touch on a little later.
The requests library isolates all the challenges of making requests behind a straightforward API, allowing you to concentrate on communicating with services and consuming data in your application.
In this article, we will walk through some of the core components of the requests library and provide some code examples to help you get started.
Get certified in your dream Data Scientist role
Our certification programs help you stand out and prove your skills are job-ready to potential employers.
Run and edit the code from this tutorial online
Run codeQuick Answer: Making GET and POST requests in Python
In a rush? Here's the Python syntax for making a simple GET
and POST
request:
1. GET request
import requests
# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts/1"
# A GET request to the API
response = requests.get(url)
# Print the response
print(response.json())
2. POST request
import requests
# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts"
# Data to be sent
data = {
"userID": 1,
"title": "Making a POST request",
"body": "This is the data we created."
}
# A POST request to the API
response = requests.post(url, json=data)
# Print the response
print(response.json())
If you'd like to dive deeper into making HTTP requests with Python and learn more about REST APIs, keep reading!
What Is a REST API?
We’ve established APIs are software mediators. Another way to think of them is as a type of software interface that grants other applications access to specific data and methods.
One of the most popular architectures used to build APIs is the REpresentational State Transfer (REST) pattern. The REST architectural design enables the client and server to be implemented independently of one another without being aware of each other. This means code on either side can be changed without worrying about how the change will affect the other.
REST APIs, therefore, follow a set of guidelines designed to simplify communications between software, thereby making the process of accessing data more straightforward and logical. Don’t worry if you don’t know these guidelines; you don’t need to know them to get started – what you do need to know is how data is exposed from REST services.
Data from REST web services are exposed to the internet through a public URL, which can be accessed by sending an HTTP request.
HTTP Request Methods Overview
Let’s rewind to our restaurant analogy; to order food at a restaurant, the waiter will approach you, and you say what you want. The waiter then passes your request to the chef, who makes the meal and passes it to the waiter to return to you. In other words, the chef wouldn’t cook your meal until your request has been sent.
REST APIs are the same: they listen for HTTP request methods before taking action. HTTP defines a set of request methods that tell the API what operations to perform for a given resource. It specifies how to interact with the resources located at the provided endpoint.
There are several HTTP methods, but five are commonly used with REST APIs:
HTTP Method |
Description |
GET |
Retrieve data |
POST |
Create data |
PUT |
Update existing data |
PATCH |
Partially update existing data |
DELETE |
Delete data |
It’s highly likely you will be performing GET requests more than any other method in data analysis and data science. This is because it’s the necessary method to access certain datasets. Learn more about this with DataCamp’s Intermediate Importing Data in Python course.
When you make a request to a web server, the API returns a response. Attached to the response is an HTTP status code. The purpose of the status code is to provide additional information about the response so the client knows the type of request being received.
What Are API Endpoints?
A URL defines the data you interact with on a web server. Much like a web page URL is connected to a single page, an endpoint URL is linked to particular resources within an API. Therefore, an endpoint may be described as a digital location where an API receives inquiries about a particular resource on its server—think of it as the other end of a communication channel.
To add more context, REST APIs expose a set of public URLs that may be requested by client applications to access the resources of the web service. The public URLs exposed by the REST API are known as “endpoints.”
Using Python to Make HTTP Requests
The Python requests
module enables developers to write code to interact with REST APIs. It allows them to send HTTP requests using Python without having to worry about the complexities that typically come with carrying out such tasks (i.e., manually adding query strings to URLs, form-encoding PUT
and POST
data, etc.).
Despite being considered the de facto standard for making HTTP requests in Python, the requests
module is not part of Python’s standard library – it must be installed.
The most straightforward way to install the requests module is with pip:
python -m pip install requests
It’s always recommended that the Python packages required for different projects be managed using virtual environments; this way, the packages for one project will not interfere and break system tools in other projects because they are isolated – instead of being installed globally.
Now we’ve installed the requests module, so let’s see how it works. Follow along with the code in this DataLab workbook.
Making a GET request in Python
We’ve already established GET
is one of the most common HTTP request methods you’ll encounter when working with REST APIs. It allows you (the client) to retrieve data from web servers.
It's important to note that GET
is a read-only operation, meaning it’s only suitable for accessing existing resources and should not be used to modify them.
To demonstrate how the request module works, we will use JSONPlaceholder, which is a freely available fake API used for testing and prototyping.
import requests
# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts/1"
# A GET request to the API
response = requests.get(url)
# Print the response
response_json = response.json()
print(response_json)
"""
{'userId': 1, 'id': 1, 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}
"""
In the code above, we carried out the following:
- Defined the API endpoint to retrieve data from.
- Used the
requests.get(url)
method to retrieve the data from the defined endpoint. - We used the
response.json()
method to store the response data in a dictionary object; note that this only works because the result is written in JSON format – an error would have been raised otherwise. - The last step is to print the JSON response data.
We can also check the status code returned from the API like this:
# Print status code from original response (not JSON)
print(response.status_code)
"""
200
"""
You can also pass arguments to a Python GET request. To do this, we must slightly alter the code above. Here’s how the new code looks:
# The API endpoint
url = "https://jsonplaceholder.typicode.com/posts/"
# Adding a payload
payload = {"id": [1, 2, 3], "userId":1}
# A get request to the API
response = requests.get(url, params=payload)
# Print the response
response_json = response.json()
for i in response_json:
print(i, "\n")
"""
{'userId': 1, 'id': 1, 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}
{'userId': 1, 'id': 2, 'title': 'qui est esse', 'body': 'est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla'}
{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
"""
Here’s what we did differently:
- Changed the API endpoint. Notice it no longer has a ‘1’ at the end.
- Defined the payload in a dictionary.
- Passed the payload to the
param
argument of therequests.get()
method. - This returned a list object, so we looped through the list and printed each item on a new line.
Making a POST request in Python
GET
requests allow you to retrieve data; POST
requests will enable you to create new data. Let’s take a look at how we can create new data on the JSONPlaceholder server.
# Define new data to create
new_data = {
"userID": 1,
"id": 1,
"title": "Making a POST request",
"body": "This is the data we created."
}
# The API endpoint to communicate with
url_post = "https://jsonplaceholder.typicode.com/posts"
# A POST request to tthe API
post_response = requests.post(url_post, json=new_data)
# Print the response
post_response_json = post_response.json()
print(post_response_json)
"""
{'userID': 1, 'id': 101, 'title': 'Making a POST request', 'body': 'This is the data we created.'}
"""
In the code above, we performed the following:
- Created a new resource we wanted to add to the JSONPlaceholder API.
- Defined the endpoint to
POST
the new data. - Sent a
POST
request using therequests.post()
method. Note that thejson
parameter was set in thepost()
method; we do this to tell the API we are explicitly sending a JSON object to the specified URL. - Used the
response.json()
method to store the response data in a dictionary object. - The last step is to print the JSON response data.
But, wait!
Before you read the next bit of code, take 20 seconds to consider what status code the API will return.
Remember: this time, we created a new resource instead of simply retrieving it.
Okay, here it goes…
# Print status code from original response (not JSON)
print(post_response.status_code)
"""
201
"""
Did you get it right?
Python HTTP Requests Advanced Topics
Making HTTP requests with Python is generally straightforward; however, requiring a more advanced setup or facing problems is sometimes inevitable. Here are some challenges you may encounter and how to solve them.
Authenticating HTTP requests
Until now, our interactions with the REST API have been pretty straightforward. The JSONPlaceholder API does not require any authentication for you to start interacting with it. However, there are several instances where a REST API may require authentication before access is granted to specific endpoints – especially when you’re dealing with sensitive data.
For example, if you want to create integrations, retrieve data, and automate your workflows on GitHub, you can do so with the GitHub REST API. However, there are many operations on the GitHub REST API that require authentication, such as retrieving public and private information about authenticated users.
Here’s a simple workaround using the Python requests module:
from requests.auth import HTTPBasicAuth
private_url = "https://api.github.com/user"
github_username = "username"
token = "token"
private_url_response = requests.get(
url=private_url,
auth=HTTPBasicAuth(github_username, token)
)
private_url_response.status_code
"""
200
"""
In the code above we:
- Imported the
HTTPBasicAuth
object fromrequests.auth
. This object attaches HTTP basic authentication to the given request object—it’s essentially the same as typing your username and password into a website. - Defined the private URL endpoint to access.
- Instantiated a variable with a GitHub username – we anonymized the username for privacy.
- Instantiated a variable GitHub with a personal access token for authentication.
- Retrieved data from our endpoint and stored it in the
private_url_response
variable. - Displayed the status code.
Handling HTTP request errors
There are instances where requests made to an API do not go as expected. Several factors on either the client or server side could be at play. Regardless of the cause, the outcome is always the same: the request fails.
When using REST APIs, it’s always a good idea to make your code resilient. However, before you can write robust code, you must understand how to manage the reported errors when things do not go to plan.
For this demonstration, let’s return to the JSONPlaceholder API. We will start by writing some code and then explain what is happening.
# A deliberate typo is made in the endpoint "postz" instead of "posts"
url = "https://jsonplaceholder.typicode.com/postz"
# Attempt to GET data from provided endpoint
try:
response = requests.get(url)
response.raise_for_status()
# If the request fails (404) then print the error.
except requests.exceptions.HTTPError as error:
print(error)
"""
404 Client Error: Not Found for url: https://jsonplaceholder.typicode.com/postz
"""
In the code above:
- We defined the JSONPlace holder endpoint from which to retrieve data, but we made a deliberate typo when constructing the URL – this will raise a 404 error.
- We used Python’s built-in exception handling, to
try
andexcept
(catch) any errors that occur when attempting to visit the JSONPlaceholder endpoint. Note theraise_for_status()
method is what is used to return anHTTPError
object when an error occurs during the process. - And lastly, we printed the error that was raised.
Although we demonstrated how to handle 404 error status codes in this instance, the same format can be used to handle any HTTP status code.
Dealing with too many redirects
HTTP status codes with the 3xx format indicate that the client was redirected and must perform some additional actions to complete the request. However, this can occasionally lead to an infinite redirect loop.
Python’s requests module provides the TooManyRedirects
object to handle this problem, as follows:
"""
Note: The code here will not raise an error
but the structure is how you would hand a case where there
are multiple redirects
"""
url = "https://jsonplaceholder.typicode.com/posts"
try:
response = requests.get(url)
response.raise_for_status()
except requests.exceptions.TooManyRedirects as error:
print(error)
You can also set the maximum number of redirects as a parameter of your HTTP request method:
# Solution 2
url = "https://jsonplaceholder.typicode.com/posts"
session = requests.Session()
session.max_redirects = 3
response = session.get(url)
Another option is to disable redirects completely:
# Solution 3
url = "https://jsonplaceholder.typicode.com/posts"
session = requests.Session()
session.allow_redirects = False
response = session.get(url)
Handling HTTP requests connection errors
These are other sorts of errors you may face when attempting to send requests to a server. There are several reasons you may not receive a response from the server (i.e., DNS failure, refused connection, internet connection issues, etc.), but the outcome is consistent: a connection error is raised.
You can use the requests
modules ConnectionError
exception object to catch these issues and handle them accordingly.
Here’s how the code would look:
"""
Note: The code here will not raise an error
but the structure is how you would hand a case where there
is a connection error.
"""
url = "https://jsonplaceholder.typicode.com/posts"
try:
response = requests.get(url)
except requests.ConnectionError as error:
print(error)
Handling HTTP requests timeout
When the API server accepts your connection but cannot finish your request within the allowed time, you will get a “timeout error.”
We will demonstrate how to handle this case by setting the timeout
parameter in the requests.get()
method to an extremely small number; this will raise an error, and we will handle that error using the requests.Timeout
object.
url = "https://jsonplaceholder.typicode.com/posts"
try:
response = requests.get(url, timeout=0.0001)
except requests.Timeout as error:
print(error)
The most straightforward workaround for timeout errors is to set longer timeouts. Other solutions may include optimizing your requests, incorporating a retry loop into your scripts, or performing asynchronous API calls – a technique that allows your software to begin a potentially long-running activity while being responsive to other events rather than waiting until that task is completed.
Wrap up
In this tutorial, we covered what APIs are and explored a common API architecture called REST. We also looked at HTTP methods and how we can use the Python requests
library to interact with web services.
Check out the following courses to develop your data science skills:
Become a Python Developer
FAQs
How can you handle custom headers in Python HTTP requests?
You can add custom headers to your HTTP requests by passing a dictionary of headers to the headers
parameter in your request. For example:
import requests
url = "https://jsonplaceholder.typicode.com/posts"
headers = {
"Authorization": "Bearer YOUR_ACCESS_TOKEN",
"Content-Type": "application/json"
}
response = requests.get(url, headers=headers)
print(response.json())
What are the benefits of using a session object in the requests module?
Using a Session
object in the requests
module provides several benefits, including:
- Persistent Connections: Maintains a connection across multiple requests, improving performance.
- Session-wide Settings: Allows you to set headers, cookies, and parameters once for all requests made through the session.
- Cookie Persistence: Automatically handles cookies, persisting them across requests.
Here's an example:
import requests
session = requests.Session()
session.headers.update({"Authorization": "Bearer YOUR_ACCESS_TOKEN"})
response = session.get("https://jsonplaceholder.typicode.com/posts")
print(response.json())
How do you send a file with a POST request in Python?
You can send a file with a POST request by passing a dictionary to the files
parameter. For example:
import requests
url = "https://example.com/upload"
file_path = "/path/to/your/file.txt"
with open(file_path, 'rb') as file:
files = {'file': file}
response = requests.post(url, files=files)
print(response.status_code)
How can you configure the requests module to use proxies?
You can configure proxies in the requests
module by passing a dictionary to the proxies
parameter. For example:
import requests
url = "https://jsonplaceholder.typicode.com/posts"
proxies = {
"http": "http://10.10.1.10:3128",
"https": "http://10.10.1.10:1080",
}
response = requests.get(url, proxies=proxies)
print(response.json())
How can you handle timeouts when making HTTP requests with the Python requests module?
You can handle timeouts by specifying the timeout
parameter in your request. This parameter takes a value in seconds. For example:
import requests
url = "https://jsonplaceholder.typicode.com/posts"
try:
response = requests.get(url, timeout=5) # Timeout after 5 seconds
print(response.json())
except requests.Timeout:
print("The request timed out")
Learn more about Python with these courses!
Course
Introduction to Data Science in Python
Course
Intermediate Python
tutorial
FastAPI Tutorial: An Introduction to Using FastAPI
tutorial
Scraping Reddit with Python and BeautifulSoup 4
Abhishek Kasireddy
16 min
tutorial
Python Tutorial for Beginners
tutorial
MeetUp API
tutorial
GraphQL vs. REST: A Complete Guide
Marie Fayard
7 min
tutorial
Python Backend Development: A Complete Guide for Beginners
Oluseye Jeremiah
26 min