Skip to main content

An Introduction to Python Subprocess: Basics and Examples

Explore our step-by-step guide to running external commands using Python's subprocess module, complete with examples.
Updated Oct 1, 2024  · 15 min read

You can use the Python subprocess module to create new processes, connect to their input and output, and retrieve their return codes and/or output of the process. In this article, we'll explore the basics of the subprocess module, including how to run external commands, redirect input and output, and handle errors. Whether you're a beginner or an experienced Python developer, this tutorial will provide you with the knowledge you need to use the subprocess module effectively in your projects.

What is Python Subprocess?

The Python subprocess module is a tool that allows you to run other programs or commands from your Python code. It can be used to open new programs, send them data and get results back.

Using the Python subprocess module is like giving commands to your computer using Python instead of typing them directly into the command prompt. This module makes it easy to automate tasks and integrate other programs with your Python code. For example, you can use the subprocess module to run a shell command, like ls or ping, and get the output of that command in your Python code. You can also use it to run other Python scripts or executables, like .exe files on Windows.

Additionally, the subprocess module can redirect the input and output of the process, meaning you can control what data is sent to the process and what data is received from it.

One of the most useful capabilities of the subprocess module is that it enables the user to handle the inputs, outputs, and even the errors that are generated by the child process from within the Python code. This feature is considered to be one of the module's most powerful aspects. Because of this feature, it is now possible to make the process of calling subprocesses more powerful and versatile. For example, it is now possible to use the output of the subprocess as a variable throughout the rest of the Python script.

Become a Python Developer

Gain the programming skills all Python Developers need.

When to use Python Subprocess

Let's take a look at when to use the Python subprocess module.

Automating system tasks

The subprocess module can be used to automate various system tasks, such as running backups, starting and stopping services, and scheduling cron jobs. For example, you can use the subprocess module to run the cp command to create a backup of a file or the service command to start and stop services on Linux systems. You can also use the subprocess module to schedule tasks to run at specific intervals using cron or other scheduling tools.

Running command-line tools

The subprocess module can be used to run command-line tools such as grep, sed, and awk, and process their output in your Python code. For example, you can use the subprocess module to run the grep command to search for a specific pattern in a file and then process the output in your Python code. This can be useful for tasks such as log analysis, data processing, and text manipulation.

Running external executables

The subprocess module can run other executables, such as .exe files on Windows, and control their behavior. For example, you can use the subprocess module to run an executable that performs a specific task and then use the output of that executable in your own code. This can be useful for image processing, data analysis, and machine learning tasks.

Running scripts as background processes

You can use the subprocess module to run scripts as background processes so that they continue running after the main program exits. For example, you can use the subprocess module to run a script that performs a specific task and exit the main program without waiting for the script to complete. This can be useful for monitoring, logging, and data collection.

Running scripts with non-Python interpreter

The subprocess module can help you run scripts written in other languages like Perl, Ruby, and Bash. For example, you can use the subprocess module to run a Perl script that performs a specific task and then use the output of that script in your own code. This can be useful for data processing, text manipulation, and system administration.

Parallel processing

The subprocess module can be used to run multiple processes in parallel, which can be useful for tasks such as image processing, data analysis, and machine learning. For example, you can use the subprocess module to run multiple instances of the same script, each processing a different part of the data and then combine the results.

Run and edit the code from this tutorial online

Run code

Python Subprocess Examples 

Let's now take a look at some Python subprocess examples.

python subprocess run

The subprocess.run() method is a convenient way to run a subprocess and wait for it to complete. It lets you choose the command to run and add options like arguments, environment variables, and input/output redirections. Once the subprocess is started, the run() method blocks until the subprocess completes and returns a CompletedProcess object, which contains the return code and output of the subprocess.

The subprocess.run() method takes several arguments, some of which are:

  • args: The command to run and its arguments, passed as a list of strings.

  • capture_output: When set to True, will capture the standard output and standard error.

  • text: When set to True, will return the stdout and stderr as string, otherwise as bytes.

  • check: a boolean value that indicates whether to check the return code of the subprocess, if check is true and the return code is non-zero, then subprocess CalledProcessError is raised.

  • timeout: A value in seconds that specifies how long to wait for the subprocess to complete before timing out.

  • shell: A boolean value that indicates whether to run the command in a shell. This means that the command is passed as a string, and shell-specific features, such as wildcard expansion and variable substitution, can be used.

The subprocess.run() method also returns a CompletedProcess object, which contains the following attributes:

  • args: The command and arguments that were run.

  • returncode: The return code of the subprocess.

  • stdout: The standard output of the subprocess, as a bytes object.

  • stderr: The standard error of the subprocess, as a bytes object.

Example 1: Running shell commands:

import subprocess

result = subprocess.run(["dir"], shell=True, capture_output=True, text=True)

print(result.stdout)

Output: 

Volume in drive C has no label.

Volume Serial Number is E414-A41C

 Directory of C:\Users\owner

01/25/2023  10:56 AM    <DIR>          .

01/25/2023  10:56 AM    <DIR>          ..

07/19/2021  01:19 PM    <DIR>          .anaconda

07/19/2021  01:19 PM    <DIR>          .astropy

07/19/2021  01:19 PM    <DIR>          .aws

09/12/2022  08:48 AM               496 .bash_history

03/27/2022  03:08 PM    <DIR>          .cache

09/26/2021  06:58 AM    <DIR>          .conda

09/26/2021  06:59 AM                25 .condarc

…

Linux or mac users replace dir to ls and get rid of the shell argument.

Example 2: Running Python scripts:

You can also run a python script using the subprocess.run() method. Let’s start by creating a simple Python script in .py file

print(“This is the output from subprocess module”)

Save this file as my_python_file.py. Now, you can use the subprocess module to run this file:

import subprocess

result = subprocess.run(["python", "my_python_file.py"], capture_output=True, text=True)

print(result.stdout)

Output:

This is the output from subprocess module

Example 3: Running Python code directly from a function:

For simple use-cases, you can directly pass a python command in the subprocess.run() function. Here is how:

result = subprocess.run(["C:/Users/owner/anaconda3/python", "-c", "print('This is directly from a subprocess.run() function')"], capture_output = True, text = True)

print(result.stdout)

Output:

This is directly from a subprocess.run() function

In the args list, the first element C:/Users/owner/anaconda3/python is a path to executable Python (your path may be different). The second element, -c is a Python tag that allows the user to write Python code as text to the command line. The third element, print(...), is the Python command itself.

Example 4: Using the check argument

The check argument is an optional argument of the subprocess.run() function in the Python subprocess module. It is a boolean value that controls whether the function should check the return code of the command being run.

When check is set to True, the function will check the return code of the command and raise a CalledProcessError exception if the return code is non-zero. The exception will have the return code, stdout, stderr, and command as attributes.

When check is set to False (default), the function will not check the return code and will not raise an exception, even if the command fails.

import subprocess

result = subprocess.run(["python", "file_donot_exist.py"], capture_output=True, text=True, check=True)

print(result.stdout)

print(result.stderr)

Output:

---------------------------------------------------------------------------

CalledProcessError                        Traceback (most recent call last)

<ipython-input-81-503b60184db8> in <module>

      1 import subprocess

----> 2 result = subprocess.run(["python", "file_donot_exist.py"], capture_output=True, text=True, check=True)

      3 print(result.stdout)

      4 print(result.stderr)

~\anaconda3\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)

    514         retcode = process.poll()

    515         if check and retcode:

--> 516             raise CalledProcessError(retcode, process.args,

    517                                      output=stdout, stderr=stderr)

    518     return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['python', 'file_donot_exist.py']' returned non-zero exit status 2.

Notice that the command failed because file_donot_exist.py does not exist. As opposed to when you set check=True, your process won’t fail; instead, you will get the error message in stdout

import subprocess

result = subprocess.run(["python", "my_python_file2.py"], capture_output=True, text=True, check=False)

print(result.stdout)

print(result.stderr)

Output:

python: can't open file 'my_python_file2.py': [Errno 2] No such file or directory

python subprocess Popen

subprocess.Popen() is a lower-level interface to running subprocesses, while subprocess.run is a higher-level wrapper around Popen that is intended to be more convenient to use.

Popen allows you to start a new process and interact with its standard input, output, and error streams. It returns a handle to the running process that can be used to wait for the process to complete, check its return code, or terminate it.

run() is a more convenient function that allows you to run a command and capture its output in a single call, without having to create a Popen object and manage the streams yourself. It also allows you to specify various options for running the command, such as whether to raise an exception if the command fails.

In general, you should use run() if you just need to run a command and capture its output and Popen if you need more control over the process, such as interacting with its input and output streams.

The Popen class takes the same arguments as run(), including the args that specify the command to be run and other optional arguments such as stdin, stdout, stderr, shell, cwd, and env. Also, the Popen class has several methods that allow you to interact with the process, such as communicate(), poll(), wait(), terminate(), and kill().

import subprocess

p = subprocess.Popen(["python", "--help"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

output, errors = p.communicate()

print(output)

Output:

usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...

Options and arguments (and corresponding environment variables):

-b     : issue warnings about str(bytes_instance), str(bytearray_instance)

         and comparing bytes/bytearray with str. (-bb: issue errors)

-B     : don't write .pyc files on import; also PYTHONDONTWRITEBYTECODE=x

-c cmd : program passed in as string (terminates option list)

-d     : debug output from parser; also PYTHONDEBUG=x

-E     : ignore PYTHON* environment variables (such as PYTHONPATH)

…

This will run the command python –help and create a new Popen object, which is stored in the variable p. The standard output and error of the command are captured using the communicate() method and stored in the variables output and errors, respectively.

subprocess.Popen is useful when you want more control over the process, such as sending input to it, receiving output from it, or waiting for it to complete.

python subprocess call

subprocess.call() is a function in the Python subprocess module that is used to run a command in a separate process and wait for it to complete. It returns the return code of the command, which is zero if the command was successful, and non-zero if it failed.

The call() function takes the same arguments as run(), including the args which specify the command to be run, and other optional arguments, such as stdin, stdout, stderr, shell, cwd, and env.

The standard output and error of the command are sent to the same stdout and stderr as the parent process unless you redirect them using stdout and stderr arguments.

import subprocess

return_code = subprocess.call(["python", "--version"])

if return_code == 0:

    print("Command executed successfully.")

else:

    print("Command failed with return code", return_code)

Output:

Command executed successfully.

This will run the command python –version in a separate process and wait for it to complete. The command's return code will be stored in the return_code variable, which will be zero if the command was successful, and non-zero if it failed.

subprocess.call() is useful when you want to run a command and check the return code, but do not need to capture the output.

python subprocess check_output

check_output is a function in the subprocess module that is similar to run(), but it only returns the standard output of the command, and raises a CalledProcessError exception if the return code is non-zero.

The check_output() function takes the same arguments as run(), including the args which specify the command to be run, and other optional arguments, such as stdin, stderr, shell, cwd, and env.

The check_output() function returns the standard output of the command as a bytes object or string, if text=True is passed.

import subprocess

try:

    output = subprocess.check_output(["python", "--version"], text=True)

    print(output)

except subprocess.CalledProcessError as e:

    print(f"Command failed with return code {e.returncode}")

Output:

Python 3.8.8

Python Subprocess Pipe

Python subprocess module provides a way to create and interact with child processes, which can be used to run other programs or commands. One of the features of the subprocess module is the ability to create pipes, which allow communication between the parent and child processes.

A pipe is a unidirectional communication channel that connects one process's standard output to another's standard input. A pipe can connect the output of one command to the input of another, allowing the output of the first command to be used as input to the second command.

Pipes can be created using the subprocess module with the Popen class by specifying the stdout or stdin argument as subprocess.PIPE.

For example, the following code creates a pipe that connects the output of the ls command to the input of the grep command, which filters the output to show only the lines that contain the word "file":

import subprocess

ls_process = subprocess.Popen(["ls"], stdout=subprocess.PIPE, text=True)

grep_process = subprocess.Popen(["grep", "sample"], stdin=ls_process.stdout, stdout=subprocess.PIPE, text=True)

output, error = grep_process.communicate()

print(output)

print(error)

Output:

sample_data

In this example, the Popen class is used to create two child processes, one for the ls command and one for the grep command. The stdout of the ls command is connected to the stdin of the grep command using subprocess.PIPE, which creates a pipe between the two processes. The communicate() method is used to send the output of the ls command to the grep command and retrieve the filtered output.

Conclusion

The Python subprocess module provides a powerful and flexible way to create and interact with child processes, allowing you to run other programs or issue commands from within your Python script. From simple commands like subprocess.call() to more advanced features like pipes, redirecting input and output, and passing environment variables, the subprocess module has something to offer for almost every use case. It is a great way to automate repetitive tasks, run system commands, and even interact with other programming languages and platforms.

While working with the subprocess module, it's important to remember that running external commands poses a security risk, especially when using the shell=True parameter or passing unsanitized input. It's always a good practice to use the subprocess.run() function that allows you to specify various options for how the command should be run, such as whether to raise an exception if the command fails. 

If you are interested in doing a deep dive into endless possibilities for command line automation through Python, check out our Command Line Automation in Python course. In this course, you will learn to write an automation code that will browse a filesystem, look for files that follow a pattern, and then determine whether files are duplicates in one of the numerous cases. After finishing the course, you'll be able to manage and interact with Unix processes as well as automate a variety of routine file system activities. 

Boost Your Team’s Python Skills with DataCamp for Business

If you or your team is looking to enhance your skills in Python and command line automation, consider exploring DataCamp for Business. DataCamp offers tailored learning solutions for teams of all sizes, helping businesses stay ahead in the rapidly evolving tech landscape. With DataCamp for Business, you can upskill your team with courses and custom learning tracks designed to build expertise in Python, automation, and other essential data science tools. Whether you're a startup or an enterprise, DataCamp for Business provides the resources and flexibility to achieve your team's learning goals. Request a demo today to learn more

Get certified in your dream Data Scientist role

Our certification programs help you stand out and prove your skills are job-ready to potential employers.

Get Your Certification
Timeline mobile.png

Python Subprocess FAQs

What is the difference between subprocess.call() and subprocess.run()?

subprocess.call() is a function that runs a command and waits for it to complete, returning the return code of the command. subprocess.run() is a more powerful function that allows you to run a command, capture its output, and specify various options for how the command should be run, such as whether to raise an exception if the command fails.

How can I run a command and capture its output in Python using subprocess?

You can use the subprocess.run() function to run a command and capture its output in a single call. For example, the following code runs the ls command and captures its output in the result variable:

import subprocess

result = subprocess.run(["ls"], stdout=subprocess.PIPE)

print(result.stdout.decode())

How can I pass variables as arguments to a command in a subprocess?

You can pass variables as arguments to a command in the subprocess by including them in the list passed to the subprocess.run() or subprocess.Popen() functions. For example, the following code runs the echo command and passes the value of the message variable as an argument:

import subprocess

message = "Hello, World!"

subprocess.run(["echo", message])

How can I redirect the output of a command to a file using subprocess?

You can redirect the output of a command to a file using the > operator in the command. For example, the following code runs the ls command and redirects its output to a file named ls.txt:

import subprocess

subprocess.run("ls > ls.txt", shell=True)

How can I run a command in the background using subprocess?

You can run a command in the background by setting the start_new_session parameter to True when creating a new process using subprocess.Popen(). This will create a new process group, allowing you to run the command in the background.

import subprocess

subprocess.Popen(["ls"], start_new_session=True)

How can I check the return code of a command run using subprocess?

You can check the return code of a command by accessing the returncode attribute of the subprocess.CompletedProcess object returned by subprocess.run(). For example, the following code runs the ls command and checks its return code:

import subprocess

result = subprocess.run(["ls"])

if result.returncode == 0:

    print("Command ran successfully")

else:

    print("Command failed with error code", result.returncode)

How can I pass environment variables to a command run using subprocess?

You can pass environment variables to a command by including them in a dictionary and passing that dictionary as the env parameter when creating a new process using subprocess.Popen() or subprocess.run(). For example, the following code sets the TEST_VAR environment variable and runs the printenv command:

import subprocess

env = {'TEST_VAR': 'test_value'}

subprocess.run(["printenv", "TEST_VAR"], env=env)

Photo of Moez Ali
Author
Moez Ali
LinkedIn
Twitter

Data Scientist, Founder & Creator of PyCaret

Topics
Related

tutorial

Python Multiprocessing Tutorial

Discover the basics of multiprocessing in Python and the benefits it can bring to your workflows.
Kurtis Pykes 's photo

Kurtis Pykes

6 min

tutorial

Python Tutorial for Beginners

Get a step-by-step guide on how to install Python and use it for basic data science functions.
Matthew Przybyla's photo

Matthew Przybyla

12 min

tutorial

How to Run Python Scripts Tutorial

Learn how you can execute a Python script from the command line, and also how you can provide command line arguments to your script.
Aditya Sharma's photo

Aditya Sharma

10 min

tutorial

Python Count Tutorial

Learn how to use the counter tool with an interactive example.
DataCamp Team's photo

DataCamp Team

3 min

tutorial

Python Setup: The Definitive Guide

In this tutorial, you'll learn how to set up your computer for Python development, and explain the basics for having the best application lifecycle.

J. Andrés Pizarro

15 min

tutorial

Argument Parsing in Python

In this tutorial, learn how to parse one or more arguments from the command-line or terminal using the getopt, sys, and argparse modules.
Sayak Paul's photo

Sayak Paul

10 min

See MoreSee More