- Courses
- 💻Programming Languages
- 🐍Python
- Advanced Python
- Multiprocessing
On this page
Python Multiprocessing: Complete Guide to Parallel Processing
Published on June 25, 2026
By Enrico Hochkoeppler
Web Developer & TutoriaLibre Founder
Python's multiprocessing module unlocks true parallel execution by spawning separate processes, each with its own Python interpreter and memory space. This bypasses the Global Interpreter Lock (GIL), allowing CPU-bound code to fully leverage multiple processor cores. The module provides an API similar to threading but uses processes instead of threads, making it the go-to solution for computationally intensive tasks. This guide covers everything from basic process creation to advanced patterns like process pools, shared memory, and inter-process communication, with practical examples and battle-tested best practices.
Why Multiprocessing Matters
Python's Global Interpreter Lock (GIL) prevents multiple threads from executing Python bytecode simultaneously. For CPU-bound tasks, threads provide no performance benefit. The multiprocessing module solves this by creating separate processes, each with its own GIL. This allows true parallelism across multiple CPU cores.
Unlike threads, which share memory and require careful locking, processes are independent execution units with isolated memory spaces. This isolation makes multiprocessing programs safer and more predictable, though it introduces challenges in sharing data between processes.
When to Use Multiprocessing
- CPU-bound computations — mathematical operations, data processing, image manipulation, machine learning
- Parallelizing independent tasks — processing many files, running multiple simulations
- Leveraging multi-core systems — fully utilizing available CPU resources
- Bypassing the GIL — when threads are blocked by the GIL
When Not to Use Multiprocessing
- I/O-bound tasks — use threads or asyncio instead
- Tasks requiring frequent communication — inter-process communication has overhead
- Small workloads — process creation overhead may outweigh benefits
- Memory-constrained environments — each process has its own memory space
Getting Started: The Basics
The Guard Pattern: if __name__ == "__main__"
Before writing any multiprocessing code, understand this essential pattern:
from multiprocessing import Process
def worker(name):
print(f"Hello from {name}")
if __name__ == "__main__":
p = Process(target=worker, args=("bob",))
p.start()
p.join()
Code explanation: The if __name__ == "__main__" guard is mandatory on Windows and recommended on all platforms. Without it, child processes will attempt to re-execute the module's top-level code, potentially creating an infinite loop of process creation. Child processes import the main module to access the target function, so the guard prevents recursive spawning.
Creating and Managing Processes
The Process class follows the same API as threading.Thread:
from multiprocessing import Process, current_process
import os
def info(title):
print(title)
print(f"Module name: {__name__}")
print(f"Parent process: {os.getppid()}")
print(f"Process ID: {os.getpid()}")
def worker(name):
info("Child process")
print(f"Hello, {name}")
if __name__ == "__main__":
info("Main process")
p = Process(target=worker, args=("bob",))
p.start()
p.join()
Code explanation: os.getpid() returns the current process ID. os.getppid() returns the parent process ID. Each process has a unique ID that the operating system uses for management. The join() method blocks until the child process finishes.
Process Lifecycle Methods
MethodDescriptionstart()Begins process executionjoin(timeout)Waits for process to finishis_alive()Returns whether the process is runningterminate()Terminates the process forcefullykill()Kills the process (Unix only)close()Closes the process (cleans up resources)exitcodeExit code of the process (if finished)
The multiprocessing module also introduces APIs not available in threading, including the ability to terminate, interrupt, or kill a running process.
The Process Pool Pattern
Creating a new process for each task is expensive. The Pool object manages a fixed number of worker processes that execute tasks from a queue.
Basic Pool Usage
from multiprocessing import Pool
import time
def cpu_intensive(x):
# Simulate CPU-bound work
result = 0
for i in range(10_000_000):
result += i * x
return result
if __name__ == "__main__":
# Create a pool with the number of available CPU cores
with Pool() as pool:
results = pool.map(cpu_intensive, range(8))
print(results)
Code explanation: Pool() without arguments creates a pool with the number of CPU cores available on the system. The map() method distributes the work across the pool, blocking until all results are ready. The with statement ensures proper cleanup.
Different Pool Methods
from multiprocessing import Pool
def square(x):
return x * x
if __name__ == "__main__":
with Pool(4) as pool:
# map: applies function to each item, returns list in order
results = pool.map(square, range(10))
print(results)
# map_async: non-blocking version
async_result = pool.map_async(square, range(10))
results = async_result.get()
# imap: lazy iterator (yields results as they complete)
for result in pool.imap(square, range(10)):
print(result)
# imap_unordered: yields results as they complete, any order
for result in pool.imap_unordered(square, range(10)):
print(result)
# apply: single function call
result = pool.apply(square, (5,))
# apply_async: non-blocking single call
async_result = pool.apply_async(square, (5,))
result = async_result.get(timeout=1)
Code explanation: map returns results in the original order. imap returns an iterator that yields results as they complete. imap_unordered yields results as they complete without preserving order. apply executes a single function with arguments.
Setting the Number of Workers
import multiprocessing
# Get the number of available CPU cores
cpu_count = multiprocessing.cpu_count()
# Best practice: use cpu_count() as a starting point
with Pool(processes=cpu_count) as pool:
results = pool.map(process_item, items)
Code explanation: cpu_count() returns the number of logical CPU cores. Adding more processes than cores can cause contention and reduce performance. For I/O-bound tasks, you might increase the count; for CPU-bound tasks, match the number of cores.
Inter-Process Communication (IPC)
Processes have separate memory spaces, so sharing data requires special mechanisms.
Queues: Thread-Safe Process Communication
The multiprocessing.Queue class is a near clone of queue.Queue, designed for process-safe communication:
from multiprocessing import Process, Queue
def producer(q):
for i in range(5):
q.put(f"Item {i}")
q.put(None) # Sentinel: signal completion
def consumer(q):
while True:
item = q.get()
if item is None:
break
print(f"Consumed: {item}")
if __name__ == "__main__":
q = Queue()
p1 = Process(target=producer, args=(q,))
p2 = Process(target=consumer, args=(q,))
p1.start()
p2.start()
p1.join()
p2.join()
Code explanation: Queue provides a first-in, first-out data structure that prevents race conditions. The put() method adds data, and get() retrieves it. Using a sentinel value (None) signals completion.
Pipes: Two-Way Communication
from multiprocessing import Process, Pipe
def sender(conn):
conn.send("Hello from sender")
conn.send({"key": "value"})
conn.close()
def receiver(conn):
while True:
try:
msg = conn.recv()
print(f"Received: {msg}")
except EOFError:
break
if __name__ == "__main__":
parent_conn, child_conn = Pipe()
p = Process(target=sender, args=(child_conn,))
p.start()
receiver(parent_conn)
p.join()
Code explanation: Pipe() returns a pair of connection objects representing the two ends of a pipe. Data sent on one end is received on the other. recv() blocks until data is available. EOFError is raised when the other end is closed.
Sharing Data with Shared Memory
For simple shared values, use Value and Array:
from multiprocessing import Process, Value, Array
def increment(shared_value, shared_array):
shared_value.value += 1
for i in range(len(shared_array)):
shared_array[i] += 1
if __name__ == "__main__":
counter = Value('i', 0) # 'i' = signed integer
numbers = Array('i', [1, 2, 3, 4, 5])
p = Process(target=increment, args=(counter, numbers))
p.start()
p.join()
print(f"Counter: {counter.value}")
print(f"Array: {numbers[:]}")
Code explanation: Value creates a shared memory object for a single value. Array creates a shared array. The type codes ('i' for integer) follow the same format as the array module. Access to these objects is synchronized automatically.
Managers: Sharing Complex Objects
The Manager class creates a server process that manages shared objects:
from multiprocessing import Process, Manager
def worker(shared_list, shared_dict, key, value):
shared_list.append(value)
shared_dict[key] = value
if __name__ == "__main__":
with Manager() as manager:
shared_list = manager.list()
shared_dict = manager.dict()
processes = []
for i in range(5):
p = Process(target=worker, args=(shared_list, shared_dict, f"key_{i}", i))
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"List: {shared_list}")
print(f"Dict: {shared_dict}")
Code explanation: Manager provides a server process that manages shared data structures. It supports lists, dicts, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value, and Array. Managers are flexible but have higher overhead than shared memory.
Synchronization Primitives
When processes access shared resources, synchronization prevents race conditions.
Locks
from multiprocessing import Process, Lock, Value
def increment(counter, lock):
for _ in range(1000):
with lock:
counter.value += 1
if __name__ == "__main__":
counter = Value('i', 0)
lock = Lock()
processes = [Process(target=increment, args=(counter, lock)) for _ in range(10)]
for p in processes:
p.start()
for p in processes:
p.join()
print(f"Counter: {counter.value}") # Should be 10000
Code explanation: Lock ensures only one process can execute the protected block at a time. The with lock: statement automatically acquires and releases the lock. Without synchronization, the counter would have an inconsistent value due to race conditions.
Other Synchronization Primitives
PrimitiveDescriptionRLockReentrant lock (can be acquired multiple times by same process)SemaphoreAllows a fixed number of processes to access a resourceEventSignaling between processesConditionAdvanced synchronization with wait/notifyBarrierSynchronizes a fixed number of processes at a point
Process Start Methods
Python supports three process start methods, each with different behavior:
MethodDescriptionAvailabilityspawnStarts a fresh Python interpreter processWindows, macOS (default), UnixforkCopies the parent process (fast but can cause issues)Unix onlyforkserverServer process that forks new processesUnix only
import multiprocessing as mp
# Check the current start method
print(mp.get_start_method())
# Set the start method (must be done before any Process creation)
mp.set_start_method('spawn', force=True)
Code explanation: Windows only supports spawn. On Unix, fork is faster but can cause problems with threading and file descriptors. forkserver provides a safer alternative to fork. The force=True parameter allows changing the method after it has been implicitly set.
Common Pitfalls and Solutions
1. Pickling Errors
The multiprocessing module uses pickle to serialize data sent between processes. Not all objects can be pickled.
Solution: Ensure all arguments and return values are picklable. Use global functions (not lambdas). For complex objects, implement __getstate__ and __setstate__ methods.
2. Deadlocks
Deadlocks occur when processes wait for each other to release resources.
Solution: Use timeouts on operations. Acquire locks in a consistent order. Use Queue with timeouts.
3. Resource Exhaustion
Creating too many processes can exhaust system resources.
Solution: Use Pool to limit the number of concurrent processes. Use cpu_count() as a guideline.
4. Zombie Processes
Processes that have terminated but have not been waited for become zombies.
Solution: Always call join() on processes. Use Pool which handles cleanup automatically.
5. Overhead
Process creation has significant overhead.
Solution: Use Pool for many small tasks. Consider concurrent.futures.ProcessPoolExecutor for a higher-level interface.
Advanced Patterns
Worker Pool with Queues
from multiprocessing import Process, Queue, cpu_count
import time
def worker(task_queue, result_queue):
while True:
try:
task = task_queue.get(timeout=1)
except:
break
result = process_task(task)
result_queue.put(result)
def process_task(task):
# Simulate CPU-bound work
time.sleep(0.1)
return task * 2
if __name__ == "__main__":
task_queue = Queue()
result_queue = Queue()
# Add tasks
for i in range(100):
task_queue.put(i)
# Create worker processes
num_workers = cpu_count()
workers = []
for _ in range(num_workers):
p = Process(target=worker, args=(task_queue, result_queue))
p.start()
workers.append(p)
# Collect results
results = []
for _ in range(100):
results.append(result_queue.get())
# Clean up
for p in workers:
p.join()
print(results)
Code explanation: This pattern creates a fixed number of worker processes that consume tasks from a queue and push results to another queue. It scales well for large workloads and provides fine-grained control.
Using concurrent.futures.ProcessPoolExecutor
For a simpler, more modern interface, use ProcessPoolExecutor:
from concurrent.futures import ProcessPoolExecutor
def process_item(x):
return x * x
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=4) as executor:
results = list(executor.map(process_item, range(10)))
print(results)
Code explanation: ProcessPoolExecutor provides a higher-level interface for process pools. It separates task submission from result collection, making it easier to work with asynchronous workflows.
Performance Optimization Tips
1. Use chunksize with map
with Pool() as pool:
# Larger chunksize reduces overhead but can imbalance work
results = pool.map(process_item, large_list, chunksize=100)
Code explanation: chunksize controls how many items are sent to each worker at once. Larger chunks reduce communication overhead but can cause load imbalance if tasks vary in duration.
2. Avoid Large Data Transfer
Data serialization and transfer between processes is expensive. Use shared memory when possible.
3. Batch Tasks
Group small tasks together to reduce the overhead of individual task submission.
4. Profile First
Use cProfile to identify bottlenecks before applying multiprocessing. Not all code benefits from parallelization.
Real-World Example: Image Processing
from multiprocessing import Pool, cpu_count
import os
from PIL import Image
def process_image(image_path):
"""Apply a computationally expensive transformation to an image."""
with Image.open(image_path) as img:
# Simulate CPU-bound processing
img = img.convert('L') # Grayscale
img = img.resize((img.width // 2, img.height // 2))
output_path = f"processed_{os.path.basename(image_path)}"
img.save(output_path)
return output_path
if __name__ == "__main__":
image_files = ["image1.jpg", "image2.jpg", "image3.jpg", "image4.jpg"]
# Use all available CPU cores
with Pool(cpu_count()) as pool:
results = pool.map(process_image, image_files)
print(f"Processed: {results}")
Code explanation: This example demonstrates parallel image processing. Each image is processed independently, making it an ideal candidate for multiprocessing. The pool distributes the work across available CPU cores.
Summary
Python's multiprocessing module is the definitive solution for CPU-bound parallel processing. It bypasses the GIL by spawning independent processes, each with its own interpreter and memory space. The module provides:
Processfor fine-grained control over individual processesPoolfor efficient management of worker process poolsQueueandPipefor inter-process communicationValue,Array, andManagerfor sharing data between processesLock,Semaphore, andEventfor synchronization
Key takeaways:
- Always use
if __name__ == "__main__"to prevent infinite process spawning - Use
Poolfor many independent tasks; useProcessfor individual control - Match the number of workers to available CPU cores for CPU-bound tasks
- Choose the right inter-process communication mechanism based on your needs
- Handle pickling issues by ensuring all data is serializable
- Consider
ProcessPoolExecutorfor a higher-level interface
By mastering these concepts, you can unlock the full potential of multi-core systems and dramatically accelerate CPU-intensive Python programs.
Found this helpful? Share the knowledge...
... or support this project to keep it alive!
Previous
Concurrency with Threads
Next
Asyncio & Coroutines
-
Why Multiprocessing Matters
- When to Use Multiprocessing
- When Not to Use Multiprocessing
-
Getting Started: The Basics
- The Guard Pattern:
if __name__ == "__main__" - Creating and Managing Processes
- Process Lifecycle Methods
- The Guard Pattern:
-
The Process Pool Pattern
- Basic Pool Usage
- Different Pool Methods
- Setting the Number of Workers
-
Inter-Process Communication (IPC)
- Queues: Thread-Safe Process Communication
- Pipes: Two-Way Communication
- Sharing Data with Shared Memory
- Managers: Sharing Complex Objects
-
Synchronization Primitives
- Locks
- Other Synchronization Primitives
-
Process Start Methods
-
Common Pitfalls and Solutions
- 1. Pickling Errors
- 2. Deadlocks
- 3. Resource Exhaustion
- 4. Zombie Processes
- 5. Overhead
-
Advanced Patterns
- Worker Pool with Queues
- Using concurrent.futures.ProcessPoolExecutor
-
Performance Optimization Tips
- 1. Use
chunksizewithmap - 2. Avoid Large Data Transfer
- 3. Batch Tasks
- 4. Profile First
- 1. Use
-
Real-World Example: Image Processing
-
Summary
