- Courses
- 💻Programming Languages
- 🐍Python
- Standard Library Deep Dive
- Itertools & Functools
On this page
Itertools and functools in real Python programs
By Enrico Hochkoeppler
Web Developer & TutoriaLibre Founder
Purpose of this lesson
This lesson shows how itertools and functools help you write code that is faster, more memory‑efficient, and cleaner than naïve list‑based approaches.
Instead of thinking of them as “a bag of functions to memorize”, you should see them as tools for building pipelines:
itertoolslets you stream data through steps without ever holding everything in memory.functoolslets you shape and reuse functions so those pipelines stay readable and composable.
By the end, you should be able to look at a problem and say:
“This is a pipeline problem; I can solve it with iterators and a few function helpers.”
Itertools as a way of thinking about data
Why iterators matter more than lists
A list says: “I have all the data, right now.”
An iterator says: “I will produce data, one item at a time, when you ask.”
For small inputs, both feel the same. For large inputs—millions of rows, log streams, API responses—lists become a problem:
- They allocate memory for every element.
- They force you to wait until everything is built before you can start processing.
itertools works with iterators, so you can start processing immediately, and you never need to store everything at once. You build a chain of steps where each step consumes items and produces new ones.
A good mental model:
“
itertoolsis a toolbox for wiring together conveyor belts of data.”
Infinite iterators: generating data on demand
How infinite iterators are actually safe
At first, “infinite iterator” sounds dangerous. But they are safe because you control how many values you consume.
Take count() as an example. It represents “keep counting forever”, but you usually combine it with something that stops:
from itertools import count, islice
numbers = count(10, 2) # 10, 12, 14, 16, ...
first_five = list(islice(numbers, 5))
print(first_five) # [10, 12, 14, 16, 18]
Here, count() could go on forever, but islice() says “give me only the first 5”.
You never build an infinite list; you just pull a finite prefix from an infinite source.
cycle() works similarly: it keeps looping over a sequence. You might use it to assign alternating labels:
from itertools import cycle, islice
labels = cycle(["A", "B"])
assigned = list(islice(labels, 6))
print(assigned) # ['A', 'B', 'A', 'B', 'A', 'B']
repeat() is useful when you need the same value many times, without building a list of copies:
from itertools import repeat
for msg in repeat("ping", 3):
print(msg)
The key idea: infinite iterators are not about infinity, they are about not having to decide a fixed size up front. You decide how much to consume.
Combinatoric iterators: exploring possibilities without blowing up memory
When you generate permutations, combinations, or Cartesian products naïvely, you often create huge lists. itertools lets you explore these spaces lazily.
Imagine you want all pairs of a number and a letter:
from itertools import product
pairs = product([1, 2], ["A", "B"])
for p in pairs:
print(p)
product() does not build [(1, 'A'), (1, 'B'), (2, 'A'), (2, 'B')] in memory first.
It yields one pair at a time. If you only need the first few, you never pay for the rest.
The same applies to permutations() and combinations(). For example, generating all 2‑element permutations of [1, 2, 3]:
from itertools import permutations
for p in permutations([1, 2, 3], 2):
print(p)
This is extremely useful in search algorithms and testing, where you might want to iterate over possibilities without committing to storing them all.
The pattern to remember:
“When I need to explore combinations or permutations, I should stream them, not pre‑build them.”
Filtering iterators: controlling when the conveyor belt passes items
Filtering in itertools is about deciding which items move forward in your pipeline and when they stop.
filterfalse() is like filter(), but inverted: it keeps items where the condition is false. Suppose you want all even numbers:
from itertools import filterfalse
numbers = [1, 2, 3, 4, 5, 6]
evens = filterfalse(lambda x: x % 2, numbers)
print(list(evens)) # [2, 4, 6]
dropwhile() and takewhile() are about prefixes of data:
dropwhile()skips items at the beginning while a condition is true, then lets everything else through.takewhile()does the opposite: it lets items through while the condition is true, then stops.
from itertools import dropwhile, takewhile
data = [1, 2, 3, 4, 1, 2]
print(list(dropwhile(lambda x: x < 3, data))) # [3, 4, 1, 2]
print(list(takewhile(lambda x: x < 3, data))) # [1, 2]
These are perfect when your data is already ordered in some meaningful way (e.g., sorted by time or value), and you want to cut off a prefix based on a condition.
The mental model:
“Filtering iterators decide which items are allowed to continue down the pipeline, and where the pipeline stops.”
Transforming and combining iterators: building pipelines
Real power appears when you start chaining operations.
chain() lets you treat multiple sequences as one continuous stream:
from itertools import chain
data = chain([1, 2], [3, 4], [5])
print(list(data)) # [1, 2, 3, 4, 5]
You never build [1, 2, 3, 4, 5] explicitly; you just walk through each input in turn.
islice() lets you cut out a slice from any iterator, not just lists:
from itertools import islice
data = range(100) # already an iterator‑like object
middle = list(islice(data, 10, 20))
print(middle) # [10, 11, ..., 19]
groupby() is a bit different: it groups consecutive items that share a key. It assumes your data is already grouped or sorted by that key:
from itertools import groupby
for key, group in groupby("aaabbc"):
print(key, list(group))
# a ['a', 'a', 'a']
# b ['b', 'b']
# c ['c']
This is very useful in data processing pipelines where you first sort by a field, then process each group.
The big idea:
“With
itertools, I don’t think in terms of ‘build a list, then another list’.
I think in terms of ‘data flows through these steps, one item at a time’.”
Functools as tools for shaping behavior
Why function tools matter
If itertools is about data flow, functools is about function behavior:
- Remembering results so you don’t recompute them.
- Pre‑filling arguments so functions fit better into callbacks and pipelines.
- Adapting old styles of functions to new APIs.
Instead of writing new functions over and over, you transform existing ones.
lru_cache: trading memory for speed
lru_cache() wraps a function so that when you call it with the same arguments again, it returns the stored result instead of recomputing.
Consider a naïve recursive Fibonacci:
from functools import lru_cache
@lru_cache(maxsize=128)
def fib(n: int) -> int:
if n < 2:
return n
return fib(n - 1) + fib(n - 2)
Without caching, fib(35) calls itself thousands of times with the same arguments. With lru_cache, each distinct n is computed once, then reused.
The trade‑off is clear:
- You spend memory to store results.
- You save time by avoiding repeated work.
This is ideal for pure functions (same input → same output, no side effects) that are called often with the same arguments.
partial: pre‑configuring functions for reuse
Sometimes you have a general function, but in a specific context you always use the same argument values. Instead of writing a new wrapper function, you can “lock in” some arguments with partial().
from functools import partial
def power(base, exp):
return base ** exp
square = partial(power, exp=2)
cube = partial(power, exp=3)
print(square(5)) # 25
print(cube(2)) # 8
Here, square and cube are just specialized versions of power. This is especially useful when you pass functions into other functions, like map, GUI callbacks, or web handlers.
The pattern:
“I have a general function, but in this pipeline I always use it with these arguments.
I’ll create a partial instead of rewriting it.”
reduce: folding a sequence into a single value
reduce() takes a function and a sequence and repeatedly applies the function to accumulate a result.
from functools import reduce
from operator import add
total = reduce(add, [1, 2, 3, 4])
print(total) # 10
Conceptually, this is:
- Start with the first two elements:
add(1, 2) → 3 - Then combine with the next:
add(3, 3) → 6 - Then again:
add(6, 4) → 10
In modern Python, a lot of reduce() use cases are clearer as sum(), min(), max(), or comprehensions. But it’s still useful when your combining logic is more complex than a simple built‑in.
The important thing is to recognize the pattern: “I want to collapse many items into one value using a rule.”
cmp_to_key: adapting old comparison functions
Older Python code often used “comparison functions” that returned negative/zero/positive values. Modern sorting APIs expect a key function instead.
cmp_to_key() converts an old‑style comparator into a key object that sorted() can use:
from functools import cmp_to_key
def compare(a, b):
return a - b
sorted_list = sorted([3, 1, 2], key=cmp_to_key(compare))
print(sorted_list) # [1, 2, 3]
You won’t use this every day, but it’s valuable when you’re porting legacy code or integrating with older libraries.
Putting itertools and functools together: a real pipeline
Let’s build a small but realistic example:
You have a huge stream of numbers, and you want to:
- Double each number.
- Keep only those greater than 100.
- Take the first 10 that match.
You don’t want to build giant lists at any step.
from itertools import islice
from functools import partial
def transform(x):
return x * 2
def is_large(x):
return x > 100
double = partial(map, transform)
filter_large = partial(filter, is_large)
data = range(1_000_000) # imagine this is a huge stream
pipeline = islice(filter_large(double(data)), 10)
result = list(pipeline)
print(result)
Walk through what happens:
datais a lazy range, not a list.double(data)creates a lazymapthat doubles each value as needed.filter_large(...)creates a lazyfilterthat only lets values > 100 through.islice(..., 10)stops after 10 matching values.
At no point do you build a list of a million elements. You only ever hold a few items in memory at a time. This is exactly the kind of pattern itertools and functools are designed for.
Best practices to internalize
- Think in iterators, not lists, when dealing with large or streaming data.
- Use
itertoolsto compose steps: generate, filter, transform, group, and slice. - Use
lru_cache()for expensive pure functions that are called repeatedly. - Use
partial()to adapt functions to fit into pipelines and callbacks without rewriting them. - Use
reduce()only when it genuinely clarifies a “fold into one value” pattern. - Combine these tools to build clear, efficient pipelines instead of deeply nested loops and temporary lists.
Summary
itertools and functools are not about memorizing dozens of functions. They are about changing how you design solutions:
- With
itertools, you design data flows that are lazy and memory‑efficient. - With
functools, you design function flows that are reusable and composable.
Once you start thinking in terms of pipelines—data moving through iterator steps, shaped by small reusable functions—these modules stop being “extra stuff” and become part of how you naturally write Python.
Found this helpful? Share the knowledge...
... or support this project to keep it alive!
Previous
Random & Secrets
Next
Collections
-
Purpose of this lesson
-
Itertools as a way of thinking about data
- Why iterators matter more than lists
-
Infinite iterators: generating data on demand
- How infinite iterators are actually safe
-
Combinatoric iterators: exploring possibilities without blowing up memory
-
Filtering iterators: controlling when the conveyor belt passes items
-
Transforming and combining iterators: building pipelines
-
Functools as tools for shaping behavior
- Why function tools matter
-
lru_cache: trading memory for speed
-
partial: pre‑configuring functions for reuse
-
reduce: folding a sequence into a single value
-
cmp_to_key: adapting old comparison functions
-
Putting itertools and functools together: a real pipeline
-
Best practices to internalize
-
Summary
