On this page

Python Multiprocessing: Complete Guide to Parallel Processing

Published on June 25, 2026

Enrico Hochkoeppler Profile

By Enrico Hochkoeppler

Web Developer & TutoriaLibre Founder

Python's multiprocessing module unlocks true parallel execution by spawning separate processes, each with its own Python interpreter and memory space. This bypasses the Global Interpreter Lock (GIL), allowing CPU-bound code to fully leverage multiple processor cores. The module provides an API similar to threading but uses processes instead of threads, making it the go-to solution for computationally intensive tasks. This guide covers everything from basic process creation to advanced patterns like process pools, shared memory, and inter-process communication, with practical examples and battle-tested best practices.


Why Multiprocessing Matters​

Python's Global Interpreter Lock (GIL) prevents multiple threads from executing Python bytecode simultaneously. For CPU-bound tasks, threads provide no performance benefit. The multiprocessing module solves this by creating separate processes, each with its own GIL. This allows true parallelism across multiple CPU cores.

Unlike threads, which share memory and require careful locking, processes are independent execution units with isolated memory spaces. This isolation makes multiprocessing programs safer and more predictable, though it introduces challenges in sharing data between processes.

When to Use Multiprocessing​

When Not to Use Multiprocessing​


Getting Started: The Basics​

The Guard Pattern: if __name__ == "__main__"​

Before writing any multiprocessing code, understand this essential pattern:


from multiprocessing import Process



def worker(name):

    print(f"Hello from {name}")



if __name__ == "__main__":

    p = Process(target=worker, args=("bob",))

    p.start()

    p.join()

Code explanation: The if __name__ == "__main__" guard is mandatory on Windows and recommended on all platforms. Without it, child processes will attempt to re-execute the module's top-level code, potentially creating an infinite loop of process creation. Child processes import the main module to access the target function, so the guard prevents recursive spawning.

Creating and Managing Processes​

The Process class follows the same API as threading.Thread:


from multiprocessing import Process, current_process

import os



def info(title):

    print(title)

    print(f"Module name: {__name__}")

    print(f"Parent process: {os.getppid()}")

    print(f"Process ID: {os.getpid()}")



def worker(name):

    info("Child process")

    print(f"Hello, {name}")



if __name__ == "__main__":

    info("Main process")

    p = Process(target=worker, args=("bob",))

    p.start()

    p.join()

Code explanation: os.getpid() returns the current process ID. os.getppid() returns the parent process ID. Each process has a unique ID that the operating system uses for management. The join() method blocks until the child process finishes.

Process Lifecycle Methods​

MethodDescriptionstart()Begins process executionjoin(timeout)Waits for process to finishis_alive()Returns whether the process is runningterminate()Terminates the process forcefullykill()Kills the process (Unix only)close()Closes the process (cleans up resources)exitcodeExit code of the process (if finished)

The multiprocessing module also introduces APIs not available in threading, including the ability to terminate, interrupt, or kill a running process.


The Process Pool Pattern​

Creating a new process for each task is expensive. The Pool object manages a fixed number of worker processes that execute tasks from a queue.

Basic Pool Usage​


from multiprocessing import Pool

import time



def cpu_intensive(x):

    # Simulate CPU-bound work

    result = 0

    for i in range(10_000_000):

        result += i * x

    return result



if __name__ == "__main__":

    # Create a pool with the number of available CPU cores

    with Pool() as pool:

        results = pool.map(cpu_intensive, range(8))

        print(results)

Code explanation: Pool() without arguments creates a pool with the number of CPU cores available on the system. The map() method distributes the work across the pool, blocking until all results are ready. The with statement ensures proper cleanup.

Different Pool Methods​


from multiprocessing import Pool



def square(x):

    return x * x



if __name__ == "__main__":

    with Pool(4) as pool:

        # map: applies function to each item, returns list in order

        results = pool.map(square, range(10))

        print(results)



        # map_async: non-blocking version

        async_result = pool.map_async(square, range(10))

        results = async_result.get()



        # imap: lazy iterator (yields results as they complete)

        for result in pool.imap(square, range(10)):

            print(result)



        # imap_unordered: yields results as they complete, any order

        for result in pool.imap_unordered(square, range(10)):

            print(result)



        # apply: single function call

        result = pool.apply(square, (5,))



        # apply_async: non-blocking single call

        async_result = pool.apply_async(square, (5,))

        result = async_result.get(timeout=1)

Code explanation: map returns results in the original order. imap returns an iterator that yields results as they complete. imap_unordered yields results as they complete without preserving order. apply executes a single function with arguments.

Setting the Number of Workers​


import multiprocessing



# Get the number of available CPU cores

cpu_count = multiprocessing.cpu_count()



# Best practice: use cpu_count() as a starting point

with Pool(processes=cpu_count) as pool:

    results = pool.map(process_item, items)

Code explanation: cpu_count() returns the number of logical CPU cores. Adding more processes than cores can cause contention and reduce performance. For I/O-bound tasks, you might increase the count; for CPU-bound tasks, match the number of cores.


Inter-Process Communication (IPC)​

Processes have separate memory spaces, so sharing data requires special mechanisms.

Queues: Thread-Safe Process Communication​

The multiprocessing.Queue class is a near clone of queue.Queue, designed for process-safe communication:


from multiprocessing import Process, Queue



def producer(q):

    for i in range(5):

        q.put(f"Item {i}")

    q.put(None)  # Sentinel: signal completion



def consumer(q):

    while True:

        item = q.get()

        if item is None:

            break

        print(f"Consumed: {item}")



if __name__ == "__main__":

    q = Queue()

    p1 = Process(target=producer, args=(q,))

    p2 = Process(target=consumer, args=(q,))



    p1.start()

    p2.start()



    p1.join()

    p2.join()

Code explanation: Queue provides a first-in, first-out data structure that prevents race conditions. The put() method adds data, and get() retrieves it. Using a sentinel value (None) signals completion.

Pipes: Two-Way Communication​


from multiprocessing import Process, Pipe



def sender(conn):

    conn.send("Hello from sender")

    conn.send({"key": "value"})

    conn.close()



def receiver(conn):

    while True:

        try:

            msg = conn.recv()

            print(f"Received: {msg}")

        except EOFError:

            break



if __name__ == "__main__":

    parent_conn, child_conn = Pipe()

    p = Process(target=sender, args=(child_conn,))

    p.start()

    receiver(parent_conn)

    p.join()

Code explanation: Pipe() returns a pair of connection objects representing the two ends of a pipe. Data sent on one end is received on the other. recv() blocks until data is available. EOFError is raised when the other end is closed.

Sharing Data with Shared Memory​

For simple shared values, use Value and Array:


from multiprocessing import Process, Value, Array



def increment(shared_value, shared_array):

    shared_value.value += 1

    for i in range(len(shared_array)):

        shared_array[i] += 1



if __name__ == "__main__":

    counter = Value('i', 0)  # 'i' = signed integer

    numbers = Array('i', [1, 2, 3, 4, 5])



    p = Process(target=increment, args=(counter, numbers))

    p.start()

    p.join()



    print(f"Counter: {counter.value}")

    print(f"Array: {numbers[:]}")

Code explanation: Value creates a shared memory object for a single value. Array creates a shared array. The type codes ('i' for integer) follow the same format as the array module. Access to these objects is synchronized automatically.

Managers: Sharing Complex Objects​

The Manager class creates a server process that manages shared objects:


from multiprocessing import Process, Manager



def worker(shared_list, shared_dict, key, value):

    shared_list.append(value)

    shared_dict[key] = value



if __name__ == "__main__":

    with Manager() as manager:

        shared_list = manager.list()

        shared_dict = manager.dict()



        processes = []

        for i in range(5):

            p = Process(target=worker, args=(shared_list, shared_dict, f"key_{i}", i))

            processes.append(p)

            p.start()



        for p in processes:

            p.join()



        print(f"List: {shared_list}")

        print(f"Dict: {shared_dict}")

Code explanation: Manager provides a server process that manages shared data structures. It supports lists, dicts, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value, and Array. Managers are flexible but have higher overhead than shared memory.


Synchronization Primitives​

When processes access shared resources, synchronization prevents race conditions.

Locks​


from multiprocessing import Process, Lock, Value



def increment(counter, lock):

    for _ in range(1000):

        with lock:

            counter.value += 1



if __name__ == "__main__":

    counter = Value('i', 0)

    lock = Lock()



    processes = [Process(target=increment, args=(counter, lock)) for _ in range(10)]



    for p in processes:

        p.start()

    for p in processes:

        p.join()



    print(f"Counter: {counter.value}")  # Should be 10000

Code explanation: Lock ensures only one process can execute the protected block at a time. The with lock: statement automatically acquires and releases the lock. Without synchronization, the counter would have an inconsistent value due to race conditions.

Other Synchronization Primitives​

PrimitiveDescriptionRLockReentrant lock (can be acquired multiple times by same process)SemaphoreAllows a fixed number of processes to access a resourceEventSignaling between processesConditionAdvanced synchronization with wait/notifyBarrierSynchronizes a fixed number of processes at a point


Process Start Methods​

Python supports three process start methods, each with different behavior:

MethodDescriptionAvailabilityspawnStarts a fresh Python interpreter processWindows, macOS (default), UnixforkCopies the parent process (fast but can cause issues)Unix onlyforkserverServer process that forks new processesUnix only


import multiprocessing as mp



# Check the current start method

print(mp.get_start_method())



# Set the start method (must be done before any Process creation)

mp.set_start_method('spawn', force=True)

Code explanation: Windows only supports spawn. On Unix, fork is faster but can cause problems with threading and file descriptors. forkserver provides a safer alternative to fork. The force=True parameter allows changing the method after it has been implicitly set.


Common Pitfalls and Solutions​

1. Pickling Errors​

The multiprocessing module uses pickle to serialize data sent between processes. Not all objects can be pickled.

Solution: Ensure all arguments and return values are picklable. Use global functions (not lambdas). For complex objects, implement __getstate__ and __setstate__ methods.

2. Deadlocks​

Deadlocks occur when processes wait for each other to release resources.

Solution: Use timeouts on operations. Acquire locks in a consistent order. Use Queue with timeouts.

3. Resource Exhaustion​

Creating too many processes can exhaust system resources.

Solution: Use Pool to limit the number of concurrent processes. Use cpu_count() as a guideline.

4. Zombie Processes​

Processes that have terminated but have not been waited for become zombies.

Solution: Always call join() on processes. Use Pool which handles cleanup automatically.

5. Overhead​

Process creation has significant overhead.

Solution: Use Pool for many small tasks. Consider concurrent.futures.ProcessPoolExecutor for a higher-level interface.


Advanced Patterns​

Worker Pool with Queues​


from multiprocessing import Process, Queue, cpu_count

import time



def worker(task_queue, result_queue):

    while True:

        try:

            task = task_queue.get(timeout=1)

        except:

            break

        result = process_task(task)

        result_queue.put(result)



def process_task(task):

    # Simulate CPU-bound work

    time.sleep(0.1)

    return task * 2



if __name__ == "__main__":

    task_queue = Queue()

    result_queue = Queue()



    # Add tasks

    for i in range(100):

        task_queue.put(i)



    # Create worker processes

    num_workers = cpu_count()

    workers = []

    for _ in range(num_workers):

        p = Process(target=worker, args=(task_queue, result_queue))

        p.start()

        workers.append(p)



    # Collect results

    results = []

    for _ in range(100):

        results.append(result_queue.get())



    # Clean up

    for p in workers:

        p.join()



    print(results)

Code explanation: This pattern creates a fixed number of worker processes that consume tasks from a queue and push results to another queue. It scales well for large workloads and provides fine-grained control.

Using concurrent.futures.ProcessPoolExecutor​

For a simpler, more modern interface, use ProcessPoolExecutor:


from concurrent.futures import ProcessPoolExecutor



def process_item(x):

    return x * x



if __name__ == "__main__":

    with ProcessPoolExecutor(max_workers=4) as executor:

        results = list(executor.map(process_item, range(10)))

        print(results)

Code explanation: ProcessPoolExecutor provides a higher-level interface for process pools. It separates task submission from result collection, making it easier to work with asynchronous workflows.


Performance Optimization Tips​

1. Use chunksize with map​


with Pool() as pool:

    # Larger chunksize reduces overhead but can imbalance work

    results = pool.map(process_item, large_list, chunksize=100)

Code explanation: chunksize controls how many items are sent to each worker at once. Larger chunks reduce communication overhead but can cause load imbalance if tasks vary in duration.

2. Avoid Large Data Transfer​

Data serialization and transfer between processes is expensive. Use shared memory when possible.

3. Batch Tasks​

Group small tasks together to reduce the overhead of individual task submission.

4. Profile First​

Use cProfile to identify bottlenecks before applying multiprocessing. Not all code benefits from parallelization.


Real-World Example: Image Processing​


from multiprocessing import Pool, cpu_count

import os

from PIL import Image



def process_image(image_path):

    """Apply a computationally expensive transformation to an image."""

    with Image.open(image_path) as img:

        # Simulate CPU-bound processing

        img = img.convert('L')  # Grayscale

        img = img.resize((img.width // 2, img.height // 2))

        output_path = f"processed_{os.path.basename(image_path)}"

        img.save(output_path)

    return output_path



if __name__ == "__main__":

    image_files = ["image1.jpg", "image2.jpg", "image3.jpg", "image4.jpg"]



    # Use all available CPU cores

    with Pool(cpu_count()) as pool:

        results = pool.map(process_image, image_files)

        print(f"Processed: {results}")

Code explanation: This example demonstrates parallel image processing. Each image is processed independently, making it an ideal candidate for multiprocessing. The pool distributes the work across available CPU cores.


Summary​

Python's multiprocessing module is the definitive solution for CPU-bound parallel processing. It bypasses the GIL by spawning independent processes, each with its own interpreter and memory space. The module provides:

Key takeaways:

By mastering these concepts, you can unlock the full potential of multi-core systems and dramatically accelerate CPU-intensive Python programs.

Found this helpful? Share the knowledge...

... or support this project to keep it alive!

Buy Me a Coffee

Previous

Concurrency with Threads

Next

Asyncio & Coroutines