On this page

Subprocess and system interaction

Enrico Hochkoeppler Profile

By Enrico Hochkoeppler

Web Developer & TutoriaLibre Founder

Purpose of this lesson​

Python’s subprocess module is the bridge between your Python program and the operating system’s process layer.
It allows Python to start other programs, send data to them, read their output, and react to their success or failure.

The goal of this lesson is to teach you how to think about processes, not just how to call commands.
You will learn:

By the end, you should understand why each API exists and when to use it.


Why subprocess exists​

The real problem it solves​

Python is powerful, but it cannot replace every system tool.
Sometimes the best solution is to call an external program:

Before subprocess, Python had multiple inconsistent ways to run commands (os.system, popen2, etc.).
subprocess unifies everything into a safe, consistent, cross‑platform interface.

The mental model​

Think of subprocess as:

“Start another program, treat it as a separate process, and control how Python communicates with it.”

This means you must think about:

Once you understand these concepts, the API becomes intuitive.


Running simple commands​

How Python executes a command​

The simplest way to run a command is subprocess.run().
It starts a process, waits for it to finish, and returns an object describing what happened.


import subprocess



subprocess.run(["ls", "-l"])

Here’s what actually happens:

On Windows, some commands are shell built‑ins, so you may need:


subprocess.run(["dir"], shell=True)

But shell=True should be used sparingly because it exposes you to shell injection risks.

The key idea​

“If I just need to run a command and don’t care about its output, run() is enough.”


Capturing output​

Why capturing output matters​

Often you don’t want the command to print to the terminal—you want Python to use the output:

To do that, you must tell Python to capture it:


result = subprocess.run(

    ["ls", "-l"],

    capture_output=True,

    text=True

)



print(result.stdout)

Here’s what’s happening conceptually:

Capturing errors​

Errors are just another output stream:


result = subprocess.run(

    ["ls", "nonexistent"],

    capture_output=True,

    text=True

)



print(result.stderr)

This lets you build scripts that react intelligently to failures.


Checking return codes​

Why return codes matter​

Every process ends with a return code:

Python exposes this through:


result = subprocess.run(["ls"])

print(result.returncode)

This is how external programs communicate success or failure to Python.

Making failures explicit​

If you want Python to treat failures as exceptions:


subprocess.run(["false"], check=True)

This is useful in automation scripts where silent failure is unacceptable.


Passing input to a subprocess​

Why stdin matters​

Some programs expect input from the user or from another program.
You can simulate this by sending data through stdin:


result = subprocess.run(

    ["grep", "hello"],

    input="hello world\nbye",

    text=True,

    capture_output=True

)



print(result.stdout)

This lets Python participate in pipelines, even without using a shell.

The conceptual shift​

Instead of thinking:

“I need to run grep.”

You think:

“I need to send data to a process and read what it returns.”


Using Popen for advanced control​

Why Popen exists​

subprocess.run() is a convenience wrapper.
It waits for the process to finish.

But some processes:

Popen gives you full control over the process lifecycle.

Streaming output line by line​


from subprocess import Popen, PIPE



with Popen(["ping", "-c", "3", "google.com"], stdout=PIPE, text=True) as p:

    for line in p.stdout:

        print(line.strip())

This pattern is essential when:

The mental model​

“Use run() when you want the result.
Use Popen when you want control.”


Running commands with shell features​

Why shell=True is dangerous​

shell=True runs your command through the system shell (bash, cmd.exe, etc.).
This enables:

But it also enables shell injection, which is catastrophic if any part of the command comes from user input.

Example:


subprocess.run("echo $HOME", shell=True)

Use it only when shell features are truly required.


Cross‑platform considerations​

Why cross‑platform thinking matters​

Commands differ across operating systems:

To write portable scripts:

The goal is to make your automation predictable everywhere.


Practical examples​

Checking if a program exists​


result = subprocess.run(

    ["which", "python3"],

    capture_output=True,

    text=True

)

print(result.stdout)

Running git commands​


subprocess.run(["git", "status"])

Converting a video with ffmpeg​


subprocess.run(["ffmpeg", "-i", "input.mp4", "output.mp4"])

These examples show how Python becomes a controller for external tools.


Best practices​

The principles behind safe subprocess usage​

These practices prevent subtle bugs, security vulnerabilities, and cross‑platform failures.


Summary​

The subprocess module is Python’s interface to the operating system’s process layer.
You learned how to:

With these skills, Python becomes a powerful automation and orchestration tool capable of integrating with any system utility or external program.

Found this helpful? Share the knowledge...

... or support this project to keep it alive!

Buy Me a Coffee

Previous

Collections

Next

Guided Exercise: Build a System Utility Suite