On this page

Iterators & Custom Iterables

Enrico Hochkoeppler Profile

By Enrico Hochkoeppler

Web Developer & TutoriaLibre Founder

Why iterators are fundamental in Python​

In Python, almost everything that can be looped over relies on the same mechanism: the iterator protocol.

for loops, comprehensions, generators, file objects, and many standard‑library tools all depend on it.

Once you understand this protocol, you can:

How a for loop really works​

When you write:


for item in collection:

    ...

Python does not treat collection specially. It performs two explicit steps:

  1. Calls iter(collection) to obtain an iterator object.
  2. Repeatedly calls next(iterator) until a StopIteration exception is raised.

Any object that supports these two steps participates fully in Python’s iteration ecosystem.

The iterator protocol in practice​

An iterator is an object that:

A minimal custom iterator:


class Counter:

    def __init__(self, limit: int) -> None:

        self.limit = limit

        self.current = 0



    def __iter__(self):

        return self



    def __next__(self) -> int:

        if self.current >= self.limit:

            raise StopIteration

        value = self.current

        self.current += 1

        return value

Usage:


for n in Counter(3):

    print(n)

# 0, 1, 2

This class is both an iterable and its own iterator.

Distinguishing iterables from iterators​

The distinction is important:

Most built‑in collections are iterables that return a fresh iterator each time:


data = [1, 2, 3]

it1 = iter(data)

it2 = iter(data)

it1 and it2 are independent iterators over the same underlying iterable.

Designing a reusable custom iterable​

A reusable iterable should return a new iterator each time __iter__() is called.

A common pattern is to separate the iterable and the iterator into two classes.


class Countdown:

    def __init__(self, start: int) -> None:

        self.start = start



    def __iter__(self):

        return CountdownIterator(self.start)



class CountdownIterator:

    def __init__(self, current: int) -> None:

        self.current = current



    def __iter__(self):

        return self



    def __next__(self) -> int:

        if self.current <= 0:

            raise StopIteration

        value = self.current

        self.current -= 1

        return value

Each for loop over Countdown receives a fresh CountdownIterator, so the iterable can be used multiple times without sharing state between loops.

Using iterators for memory‑efficient processing​

Iterators are especially valuable when working with large or unbounded data sources, because they produce values one at a time.

Example: streaming lines from a file without loading it entirely:


def read_lines(path: str):

    with open(path, encoding="utf-8") as f:

        for line in f:

            yield line.rstrip("\n")

This function returns an iterator (a generator).

You can loop over it, pass it to sum, any, max, or chain it with other iterators, while keeping memory usage low.

Building composable iterable pipelines​

You can design classes that accept an iterable and expose a new iterable view, forming a pipeline of transformations.


class EvenNumbers:

    def __init__(self, iterable):

        self.iterable = iterable



    def __iter__(self):

        for n in self.iterable:

            if n % 2 == 0:

                yield n



class Squares:

    def __init__(self, iterable):

        self.iterable = iterable



    def __iter__(self):

        for n in self.iterable:

            yield n * n

Usage:


pipeline = Squares(EvenNumbers(range(10)))

for value in pipeline:

    print(value)

Each stage is lazy: values are produced only when requested by the consumer.

Implementing stateful iterators​

Some iterators need to maintain internal state beyond a simple counter.

A common example is grouping items into fixed‑size batches.


class Batcher:

    def __init__(self, iterable, size: int) -> None:

        self._iterator = iter(iterable)

        self.size = size



    def __iter__(self):

        return self



    def __next__(self):

        batch = []

        for _ in range(self.size):

            try:

                item = next(self._iterator)

            except StopIteration:

                if not batch:

                    raise

                break

            batch.append(item)

        return batch

This iterator yields lists of items, each of length up to size.

It is useful when you want to process data in chunks (for example, database inserts or network calls).

When to implement custom iterables and iterators​

Implementing the iterator protocol directly is appropriate when:

By designing classes that follow the iterator protocol, you integrate seamlessly with Python’s iteration machinery and standard library, while retaining full control over how data is produced and consumed.

Found this helpful? Share the knowledge...

... or support this project to keep it alive!

Buy Me a Coffee

Previous

Advanced OOP Patterns

Next

Context Managers