On this page

Itertools and functools in real Python programs

Enrico Hochkoeppler Profile

By Enrico Hochkoeppler

Web Developer & TutoriaLibre Founder

Purpose of this lesson​

This lesson shows how itertools and functools help you write code that is faster, more memory‑efficient, and cleaner than naïve list‑based approaches.

Instead of thinking of them as “a bag of functions to memorize”, you should see them as tools for building pipelines:

By the end, you should be able to look at a problem and say:

“This is a pipeline problem; I can solve it with iterators and a few function helpers.”


Itertools as a way of thinking about data​

Why iterators matter more than lists​

A list says: “I have all the data, right now.”
An iterator says: “I will produce data, one item at a time, when you ask.”

For small inputs, both feel the same. For large inputs—millions of rows, log streams, API responses—lists become a problem:

itertools works with iterators, so you can start processing immediately, and you never need to store everything at once. You build a chain of steps where each step consumes items and produces new ones.

A good mental model:

“itertools is a toolbox for wiring together conveyor belts of data.”


Infinite iterators: generating data on demand​

How infinite iterators are actually safe​

At first, “infinite iterator” sounds dangerous. But they are safe because you control how many values you consume.

Take count() as an example. It represents “keep counting forever”, but you usually combine it with something that stops:


from itertools import count, islice



numbers = count(10, 2)          # 10, 12, 14, 16, ...

first_five = list(islice(numbers, 5))

print(first_five)               # [10, 12, 14, 16, 18]

Here, count() could go on forever, but islice() says “give me only the first 5”.
You never build an infinite list; you just pull a finite prefix from an infinite source.

cycle() works similarly: it keeps looping over a sequence. You might use it to assign alternating labels:


from itertools import cycle, islice



labels = cycle(["A", "B"])

assigned = list(islice(labels, 6))

print(assigned)                 # ['A', 'B', 'A', 'B', 'A', 'B']

repeat() is useful when you need the same value many times, without building a list of copies:


from itertools import repeat



for msg in repeat("ping", 3):

    print(msg)

The key idea: infinite iterators are not about infinity, they are about not having to decide a fixed size up front. You decide how much to consume.


Combinatoric iterators: exploring possibilities without blowing up memory​

When you generate permutations, combinations, or Cartesian products naïvely, you often create huge lists. itertools lets you explore these spaces lazily.

Imagine you want all pairs of a number and a letter:


from itertools import product



pairs = product([1, 2], ["A", "B"])

for p in pairs:

    print(p)

product() does not build [(1, 'A'), (1, 'B'), (2, 'A'), (2, 'B')] in memory first.
It yields one pair at a time. If you only need the first few, you never pay for the rest.

The same applies to permutations() and combinations(). For example, generating all 2‑element permutations of [1, 2, 3]:


from itertools import permutations



for p in permutations([1, 2, 3], 2):

    print(p)

This is extremely useful in search algorithms and testing, where you might want to iterate over possibilities without committing to storing them all.

The pattern to remember:

“When I need to explore combinations or permutations, I should stream them, not pre‑build them.”


Filtering iterators: controlling when the conveyor belt passes items​

Filtering in itertools is about deciding which items move forward in your pipeline and when they stop.

filterfalse() is like filter(), but inverted: it keeps items where the condition is false. Suppose you want all even numbers:


from itertools import filterfalse



numbers = [1, 2, 3, 4, 5, 6]

evens = filterfalse(lambda x: x % 2, numbers)

print(list(evens))              # [2, 4, 6]

dropwhile() and takewhile() are about prefixes of data:


from itertools import dropwhile, takewhile



data = [1, 2, 3, 4, 1, 2]



print(list(dropwhile(lambda x: x < 3, data)))   # [3, 4, 1, 2]

print(list(takewhile(lambda x: x < 3, data)))   # [1, 2]

These are perfect when your data is already ordered in some meaningful way (e.g., sorted by time or value), and you want to cut off a prefix based on a condition.

The mental model:

“Filtering iterators decide which items are allowed to continue down the pipeline, and where the pipeline stops.”


Transforming and combining iterators: building pipelines​

Real power appears when you start chaining operations.

chain() lets you treat multiple sequences as one continuous stream:


from itertools import chain



data = chain([1, 2], [3, 4], [5])

print(list(data))               # [1, 2, 3, 4, 5]

You never build [1, 2, 3, 4, 5] explicitly; you just walk through each input in turn.

islice() lets you cut out a slice from any iterator, not just lists:


from itertools import islice



data = range(100)               # already an iterator‑like object

middle = list(islice(data, 10, 20))

print(middle)                   # [10, 11, ..., 19]

groupby() is a bit different: it groups consecutive items that share a key. It assumes your data is already grouped or sorted by that key:


from itertools import groupby



for key, group in groupby("aaabbc"):

    print(key, list(group))

# a ['a', 'a', 'a']

# b ['b', 'b']

# c ['c']

This is very useful in data processing pipelines where you first sort by a field, then process each group.

The big idea:

“With itertools, I don’t think in terms of ‘build a list, then another list’.
I think in terms of ‘data flows through these steps, one item at a time’.”


Functools as tools for shaping behavior​

Why function tools matter​

If itertools is about data flow, functools is about function behavior:

Instead of writing new functions over and over, you transform existing ones.


lru_cache: trading memory for speed​

lru_cache() wraps a function so that when you call it with the same arguments again, it returns the stored result instead of recomputing.

Consider a naïve recursive Fibonacci:


from functools import lru_cache



@lru_cache(maxsize=128)

def fib(n: int) -> int:

    if n < 2:

        return n

    return fib(n - 1) + fib(n - 2)

Without caching, fib(35) calls itself thousands of times with the same arguments. With lru_cache, each distinct n is computed once, then reused.

The trade‑off is clear:

This is ideal for pure functions (same input → same output, no side effects) that are called often with the same arguments.


partial: pre‑configuring functions for reuse​

Sometimes you have a general function, but in a specific context you always use the same argument values. Instead of writing a new wrapper function, you can “lock in” some arguments with partial().


from functools import partial



def power(base, exp):

    return base ** exp



square = partial(power, exp=2)

cube = partial(power, exp=3)



print(square(5))    # 25

print(cube(2))      # 8

Here, square and cube are just specialized versions of power. This is especially useful when you pass functions into other functions, like map, GUI callbacks, or web handlers.

The pattern:

“I have a general function, but in this pipeline I always use it with these arguments.
I’ll create a partial instead of rewriting it.”


reduce: folding a sequence into a single value​

reduce() takes a function and a sequence and repeatedly applies the function to accumulate a result.


from functools import reduce

from operator import add



total = reduce(add, [1, 2, 3, 4])

print(total)        # 10

Conceptually, this is:

In modern Python, a lot of reduce() use cases are clearer as sum(), min(), max(), or comprehensions. But it’s still useful when your combining logic is more complex than a simple built‑in.

The important thing is to recognize the pattern: “I want to collapse many items into one value using a rule.”


cmp_to_key: adapting old comparison functions​

Older Python code often used “comparison functions” that returned negative/zero/positive values. Modern sorting APIs expect a key function instead.

cmp_to_key() converts an old‑style comparator into a key object that sorted() can use:


from functools import cmp_to_key



def compare(a, b):

    return a - b



sorted_list = sorted([3, 1, 2], key=cmp_to_key(compare))

print(sorted_list)  # [1, 2, 3]

You won’t use this every day, but it’s valuable when you’re porting legacy code or integrating with older libraries.


Putting itertools and functools together: a real pipeline​

Let’s build a small but realistic example:
You have a huge stream of numbers, and you want to:

  1. Double each number.
  2. Keep only those greater than 100.
  3. Take the first 10 that match.

You don’t want to build giant lists at any step.


from itertools import islice

from functools import partial



def transform(x):

    return x * 2



def is_large(x):

    return x > 100



double = partial(map, transform)

filter_large = partial(filter, is_large)



data = range(1_000_000)  # imagine this is a huge stream



pipeline = islice(filter_large(double(data)), 10)

result = list(pipeline)

print(result)

Walk through what happens:

At no point do you build a list of a million elements. You only ever hold a few items in memory at a time. This is exactly the kind of pattern itertools and functools are designed for.


Best practices to internalize​


Summary​

itertools and functools are not about memorizing dozens of functions. They are about changing how you design solutions:

Once you start thinking in terms of pipelines—data moving through iterator steps, shaped by small reusable functions—these modules stop being “extra stuff” and become part of how you naturally write Python.

Found this helpful? Share the knowledge...

... or support this project to keep it alive!

Buy Me a Coffee

Previous

Random & Secrets

Next

Collections