- Courses
- đ»Programming Languages
- đPython
- Standard Library Deep Dive
- Subprocess & System Interaction
On this page
Subprocess and system interaction
By Enrico Hochkoeppler
Web Developer & TutoriaLibre Founder
Purpose of this lessonâ
Pythonâs subprocess module is the bridge between your Python program and the operating systemâs process layer.
It allows Python to start other programs, send data to them, read their output, and react to their success or failure.
The goal of this lesson is to teach you how to think about processes, not just how to call commands.
You will learn:
- What a process is and how Python interacts with it
- How to run commands safely and predictably
- How to capture output and errors
- How to communicate with longârunning or interactive programs
- How to avoid security pitfalls
- How to write crossâplatform automation scripts
By the end, you should understand why each API exists and when to use it.
Why subprocess existsâ
The real problem it solvesâ
Python is powerful, but it cannot replace every system tool.
Sometimes the best solution is to call an external program:
- A system utility (
ls,dir,which) - A developer tool (
git,docker,ffmpeg) - A legacy binary your company still depends on
- A shell pipeline that already solves the problem
Before subprocess, Python had multiple inconsistent ways to run commands (os.system, popen2, etc.).
subprocess unifies everything into a safe, consistent, crossâplatform interface.
The mental modelâ
Think of subprocess as:
âStart another program, treat it as a separate process, and control how Python communicates with it.â
This means you must think about:
- Arguments (what command to run)
- Input (what you send to it)
- Output (what it prints)
- Errors (what happens when it fails)
- Return codes (how it reports success/failure)
- Lifecycles (shortâlived vs longârunning processes)
Once you understand these concepts, the API becomes intuitive.
Running simple commandsâ
How Python executes a commandâ
The simplest way to run a command is subprocess.run().
It starts a process, waits for it to finish, and returns an object describing what happened.
import subprocess
subprocess.run(["ls", "-l"])
Hereâs what actually happens:
- Python creates a new process
- That process runs
ls -l - Its output goes directly to your terminal
- Python waits until it finishes
On Windows, some commands are shell builtâins, so you may need:
subprocess.run(["dir"], shell=True)
But shell=True should be used sparingly because it exposes you to shell injection risks.
The key ideaâ
âIf I just need to run a command and donât care about its output,
run()is enough.â
Capturing outputâ
Why capturing output mattersâ
Often you donât want the command to print to the terminalâyou want Python to use the output:
- Parse it
- Log it
- Compare it
- Transform it
To do that, you must tell Python to capture it:
result = subprocess.run(
["ls", "-l"],
capture_output=True,
text=True
)
print(result.stdout)
Hereâs whatâs happening conceptually:
- The process writes to stdout
- Python intercepts that stream
- Python stores it in
result.stdout - You decide what to do with it
Capturing errorsâ
Errors are just another output stream:
result = subprocess.run(
["ls", "nonexistent"],
capture_output=True,
text=True
)
print(result.stderr)
This lets you build scripts that react intelligently to failures.
Checking return codesâ
Why return codes matterâ
Every process ends with a return code:
0â success- nonâzero â failure
Python exposes this through:
result = subprocess.run(["ls"])
print(result.returncode)
This is how external programs communicate success or failure to Python.
Making failures explicitâ
If you want Python to treat failures as exceptions:
subprocess.run(["false"], check=True)
This is useful in automation scripts where silent failure is unacceptable.
Passing input to a subprocessâ
Why stdin mattersâ
Some programs expect input from the user or from another program.
You can simulate this by sending data through stdin:
result = subprocess.run(
["grep", "hello"],
input="hello world\nbye",
text=True,
capture_output=True
)
print(result.stdout)
This lets Python participate in pipelines, even without using a shell.
The conceptual shiftâ
Instead of thinking:
âI need to run grep.â
You think:
âI need to send data to a process and read what it returns.â
Using Popen for advanced controlâ
Why Popen existsâ
subprocess.run() is a convenience wrapper.
It waits for the process to finish.
But some processes:
- Run for a long time
- Produce output continuously
- Need interactive communication
- Should be monitored while running
Popen gives you full control over the process lifecycle.
Streaming output line by lineâ
from subprocess import Popen, PIPE
with Popen(["ping", "-c", "3", "google.com"], stdout=PIPE, text=True) as p:
for line in p.stdout:
print(line.strip())
This pattern is essential when:
- Watching logs
- Monitoring longârunning tasks
- Building realâtime dashboards
- Processing streaming data
The mental modelâ
âUse
run()when you want the result.
UsePopenwhen you want control.â
Running commands with shell featuresâ
Why shell=True is dangerousâ
shell=True runs your command through the system shell (bash, cmd.exe, etc.).
This enables:
- Pipes
- Redirects
- Globbing
- Environment variable expansion
But it also enables shell injection, which is catastrophic if any part of the command comes from user input.
Example:
subprocess.run("echo $HOME", shell=True)
Use it only when shell features are truly required.
Crossâplatform considerationsâ
Why crossâplatform thinking mattersâ
Commands differ across operating systems:
lsvsdir/usr/binpaths vs Windows paths- Shell syntax differences
To write portable scripts:
- Prefer argument lists (
["ls", "-l"]) - Avoid shellâspecific syntax
- Use absolute paths when possible
- Test on all target platforms
The goal is to make your automation predictable everywhere.
Practical examplesâ
Checking if a program existsâ
result = subprocess.run(
["which", "python3"],
capture_output=True,
text=True
)
print(result.stdout)
Running git commandsâ
subprocess.run(["git", "status"])
Converting a video with ffmpegâ
subprocess.run(["ffmpeg", "-i", "input.mp4", "output.mp4"])
These examples show how Python becomes a controller for external tools.
Best practicesâ
The principles behind safe subprocess usageâ
- Prefer
run()for simple tasks - Use
Popenwhen you need streaming or interaction - Avoid
shell=Trueunless absolutely required - Always check return codes or use
check=True - Capture output explicitly when you need to process it
- Never pass untrusted input into shell commands
- Use absolute paths for reliability
These practices prevent subtle bugs, security vulnerabilities, and crossâplatform failures.
Summaryâ
The subprocess module is Pythonâs interface to the operating systemâs process layer.
You learned how to:
- Run external commands safely
- Capture output and errors
- Interpret return codes
- Communicate with processes through stdin
- Stream output from longârunning programs
- Use
Popenfor full control - Avoid security pitfalls
- Write crossâplatform automation scripts
With these skills, Python becomes a powerful automation and orchestration tool capable of integrating with any system utility or external program.
Found this helpful? Share the knowledge...
... or support this project to keep it alive!
Previous
Collections
Next
Guided Exercise: Build a System Utility Suite
-
Purpose of this lesson
-
Why subprocess exists
- The real problem it solves
- The mental model
-
Running simple commands
- How Python executes a command
- The key idea
-
Capturing output
- Why capturing output matters
- Capturing errors
-
Checking return codes
- Why return codes matter
- Making failures explicit
-
Passing input to a subprocess
- Why stdin matters
- The conceptual shift
-
Using Popen for advanced control
- Why Popen exists
- Streaming output line by line
- The mental model
-
Running commands with shell features
- Why shell=True is dangerous
-
Crossâplatform considerations
- Why crossâplatform thinking matters
-
Practical examples
- Checking if a program exists
- Running git commands
- Converting a video with ffmpeg
-
Best practices
- The principles behind safe subprocess usage
-
Summary
