Why is my Python code slow?

The most common causes are choosing the wrong data structure (using a list where a set would give O(1) lookups), writing Python loops where vectorized NumPy operations would run in C, performing redundant computation that could be cached, and blocking on I/O without using concurrency. Use cProfile or scalene to identify the actual bottleneck before making any changes.

How do I profile Python code to find bottlenecks?

Use cProfile for function-level profiling: run python -m cProfile -s cumtime my_script.py from the command line. Use timeit for microbenchmarking small snippets. Use line_profiler (kernprof -l -v my_script.py) for line-by-line cost inside a specific function. Use scalene for simultaneous CPU and memory profiling that separates Python time from C extension time. Use snakeviz to visualize cProfile output as an interactive flame graph in the browser.

What is the fastest way to check if an item exists in a collection in Python?

Use a set. List membership testing is O(n) because Python scans each element sequentially. Set membership testing is O(1) because sets are backed by a hash table. On a collection of one million elements, set lookup is approximately 750 times faster than list lookup. Convert a list to a set with set(my_list) before performing repeated membership checks.

Are list comprehensions faster than for loops in Python?

Yes. List comprehensions execute their loop and append logic at the C level inside the CPython interpreter. An equivalent for loop with .append() incurs a Python-level method lookup and function call on every iteration. List comprehensions are consistently 20 to 40 percent faster than for loops with .append() for the same operation.

How do I optimize string concatenation in Python?

Use str.join() instead of the + operator in a loop. Strings are immutable, so every += operation creates a new string object, copies both operands into it, and discards the old string. In a loop this produces O(n²) allocations. str.join() calculates the total size once, allocates a single buffer, and fills it — producing O(n) behavior. The pattern is: result = " ".join(list_of_strings).

When should I use a generator instead of a list?

Use a generator when you do not need random access by index and do not need to iterate the sequence more than once. A list comprehension allocates memory for all values immediately. A generator expression produces one value at a time on demand, using O(1) memory regardless of input size. Generators are essential when processing large files, database result sets, or any data stream larger than available RAM.

What does __slots__ do and when should I use it?

By default, Python stores each object instance attributes in a __dict__, which is a dictionary attached to the instance and consumes significant memory. Defining __slots__ = ("x", "y") replaces that dictionary with a fixed C array of attribute slots, reducing per-instance memory by approximately 40 to 50 percent and slightly speeding up attribute access. Use __slots__ when you create large numbers of instances of a class with a fixed, known set of attributes — coordinates, events, records, or any lightweight data container.

What is the GIL and how does it affect Python performance?

The Global Interpreter Lock (GIL) is a mutex inside CPython that prevents more than one thread from executing Python bytecode at the same time. This means Python threads do not provide parallelism for CPU-bound work. For CPU-bound tasks, use multiprocessing to spawn separate processes each with their own GIL. For I/O-bound tasks, use asyncio or threading — the GIL is released while threads wait for I/O, so they can overlap their waiting time effectively.

When should I use multiprocessing versus asyncio in Python?

Use multiprocessing for CPU-bound work — numerical computation, image processing, data transformation — where you need to utilize multiple CPU cores. Use asyncio for I/O-bound work — HTTP requests, database queries, file reads — where your code spends most of its time waiting rather than computing. asyncio handles thousands of concurrent I/O operations in a single thread. multiprocessing handles parallel computation across all available cores, but carries inter-process communication overhead that only pays off for large workloads.

How does NumPy make Python faster?

NumPy stores data as raw C-type values packed contiguously in memory, rather than as Python objects. This layout allows NumPy to apply SIMD CPU instructions that process multiple values simultaneously in a single clock cycle. Vectorized NumPy operations — array arithmetic, boolean indexing, aggregation functions — run 10 to 100 times faster than equivalent Python loops. The critical rule is to avoid iterating over NumPy arrays with Python for loops, which eliminates the performance advantage entirely.

What is Numba and when should I use it?

Numba is a library that compiles Python functions to native machine code using LLVM at the first function call. Add the @jit(nopython=True) decorator to a function and subsequent calls run at near-C speed. Numba is most effective on numerical loops that cannot be expressed as vectorized NumPy operations — for example, algorithms with data-dependent branching or recursive structure. It requires no code changes beyond the decorator and produces the largest gains on math-heavy loops.

What is PyPy and how is it different from CPython?

PyPy is an alternative Python interpreter that includes a tracing JIT compiler. It monitors which code paths run frequently and compiles them to native machine code at runtime. For pure Python workloads — loops, data structure manipulation, string processing — PyPy typically produces 4 to 10 times speedups over CPython with zero code changes. Its main limitations are incomplete support for some C extensions and JIT warm-up time that makes it less effective for short-lived scripts.

What is Cython and how does it differ from Numba?

Cython is a superset of Python that compiles to C. You add static type declarations (cdef int i, cdef double x) to variables in hot code paths, and the Cython compiler generates a C extension module that runs at C speed. Numba works as a decorator on existing Python functions and compiles at runtime with no build step. Cython requires a build step (compiling to a .so or .pyd file) but provides more control and integrates more easily with existing C libraries. Use Numba for quick numeric acceleration; use Cython for C-level performance in production libraries.

How do I reduce memory usage in Python?

The most impactful changes are: use generators instead of lists for large sequences (O(1) vs O(n) memory), use __slots__ in classes you instantiate many times (reduces per-instance size by 40 to 50 percent), use the array module instead of lists for large collections of simple numeric values, avoid unnecessary object copies by using in-place methods, and use tracemalloc to identify which lines of code are responsible for the largest allocations.

Why is functools.lru_cache useful and when should I use it?

lru_cache stores the return value of a function for each unique combination of arguments. When the function is called again with the same arguments, the cached result is returned immediately without re-executing the function body. Use it on pure functions — functions whose output depends only on their inputs — that are called repeatedly with the same values. The most dramatic gains come from recursive functions: memoizing fibonacci(n) reduces its complexity from O(2ⁿ) exponential to O(n) linear.

Does placing import statements inside functions improve performance?

It depends on the use case. Placing imports at module level is correct for standard dependencies — the import runs once at startup and the name is then a fast local lookup. Placing an import inside a function that is called repeatedly is significantly slower because Python checks sys.modules and rebinds the name on every call. However, placing an import inside a function is the correct pattern for lazy loading — deferring the import of a heavy or optional dependency until the first time the function is actually called, avoiding startup cost when that function may never run.

How do I sort a list efficiently in Python?

Python list.sort() and sorted() both use Timsort, a C-implemented hybrid sort that is already highly optimized. To get the most out of it, use the key parameter instead of a comparison function — the key function is called once per element, while a comparison function is called O(n log n) times. Use operator.itemgetter() or operator.attrgetter() as key functions instead of lambda where possible, as they are implemented in C and slightly faster. Use list.sort() when you do not need the original order, since it sorts in place without allocating a new list.

What is the fastest way to count element frequencies in Python?

Use collections.Counter. It is implemented in C and optimized specifically for counting hashable objects: from collections import Counter; counts = Counter(my_list). It is faster than a manual defaultdict(int) loop and significantly faster than a plain dict with conditional checks. Counter also provides useful methods like most_common(n) for retrieving the n most frequent elements.

How do local variables make Python code faster?

CPython resolves local variable names using a direct index into the function frame local array, which corresponds to the LOAD_FAST bytecode instruction. Global variable resolution requires a dictionary lookup in both the local and global namespace. In a tight loop, wrapping performance-critical code in a function and using local variables for frequently accessed values — including built-ins like len, range, and append — reduces the per-iteration cost of name resolution measurably.

What is attribute lookup overhead and how do I eliminate it in loops?

In Python, every obj.method access triggers a dictionary lookup on the object and its class hierarchy at runtime. Inside a loop that runs millions of iterations, this per-iteration cost accumulates. The solution is to cache the method reference in a local variable before entering the loop: upper = str.upper; append = result.append, then call upper(word) and append(item) inside the loop. This eliminates the repeated dictionary traversal and reduces execution time by approximately 30 percent for large lists.

How do I use tracemalloc to find memory leaks in Python?

Call tracemalloc.start() before the code you want to inspect, then call tracemalloc.take_snapshot() afterward and print snapshot.statistics("lineno") to see a ranked list of allocations sorted by size. Each entry shows the file, line number, and total bytes allocated from that line. This makes it straightforward to identify which specific line of code is responsible for unexpected memory growth, whether from large list allocations, repeated string construction, or objects that are not being garbage collected.

One post tagged with "Python"

View All Tags

Python Performance Optimization–Faster, Leaner Code

May 31, 2026 · 23 min read

Enrico Hochkoeppler

Web Developer & TutoriaLibre Founder

Python is often criticized for being slow, but the reality is more nuanced: poorly written Python is slow. Well-optimized Python — one that delegates work to C, avoids redundant computation, and chooses the right data structure — runs fast enough for the overwhelming majority of production workloads. This guide covers every major optimization category, from algorithmic improvements that change the complexity class of your code, to micro-optimizations that eliminate overhead in hot loops, to concurrency patterns that bypass the GIL altogether.

Tags:

Python
Performance
Memory Management

Read more

Found this helpful? Share the knowledge...

... or support this project to keep it alive!

Buy Me a Coffee