- Courses
- 💻Programming Languages
- 🐍Python
- Advanced Python
- Iterators & Custom Iterables
On this page
Iterators & Custom Iterables
By Enrico Hochkoeppler
Web Developer & TutoriaLibre Founder
Why iterators are fundamental in Python
In Python, almost everything that can be looped over relies on the same mechanism: the iterator protocol.
for loops, comprehensions, generators, file objects, and many standard‑library tools all depend on it.
Once you understand this protocol, you can:
- Design objects that integrate naturally with
forloops and built‑ins - Process large or infinite data streams without loading everything into memory
- Build reusable, composable data pipelines
How a for loop really works
When you write:
for item in collection:
...
Python does not treat collection specially. It performs two explicit steps:
- Calls
iter(collection)to obtain an iterator object. - Repeatedly calls
next(iterator)until aStopIterationexception is raised.
Any object that supports these two steps participates fully in Python’s iteration ecosystem.
The iterator protocol in practice
An iterator is an object that:
- Implements
__iter__()and returns itself - Implements
__next__()and returns the next value, or raisesStopIterationwhen finished
A minimal custom iterator:
class Counter:
def __init__(self, limit: int) -> None:
self.limit = limit
self.current = 0
def __iter__(self):
return self
def __next__(self) -> int:
if self.current >= self.limit:
raise StopIteration
value = self.current
self.current += 1
return value
Usage:
for n in Counter(3):
print(n)
# 0, 1, 2
This class is both an iterable and its own iterator.
Distinguishing iterables from iterators
The distinction is important:
- An iterable is any object that can produce a new iterator when
iter()is called. Examples: lists, tuples, sets, dictionaries, strings, many custom classes. - An iterator is the object returned by
iter(), which actually yields values via__next__().
Most built‑in collections are iterables that return a fresh iterator each time:
data = [1, 2, 3]
it1 = iter(data)
it2 = iter(data)
it1 and it2 are independent iterators over the same underlying iterable.
Designing a reusable custom iterable
A reusable iterable should return a new iterator each time __iter__() is called.
A common pattern is to separate the iterable and the iterator into two classes.
class Countdown:
def __init__(self, start: int) -> None:
self.start = start
def __iter__(self):
return CountdownIterator(self.start)
class CountdownIterator:
def __init__(self, current: int) -> None:
self.current = current
def __iter__(self):
return self
def __next__(self) -> int:
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
Each for loop over Countdown receives a fresh CountdownIterator, so the iterable can be used multiple times without sharing state between loops.
Using iterators for memory‑efficient processing
Iterators are especially valuable when working with large or unbounded data sources, because they produce values one at a time.
Example: streaming lines from a file without loading it entirely:
def read_lines(path: str):
with open(path, encoding="utf-8") as f:
for line in f:
yield line.rstrip("\n")
This function returns an iterator (a generator).
You can loop over it, pass it to sum, any, max, or chain it with other iterators, while keeping memory usage low.
Building composable iterable pipelines
You can design classes that accept an iterable and expose a new iterable view, forming a pipeline of transformations.
class EvenNumbers:
def __init__(self, iterable):
self.iterable = iterable
def __iter__(self):
for n in self.iterable:
if n % 2 == 0:
yield n
class Squares:
def __init__(self, iterable):
self.iterable = iterable
def __iter__(self):
for n in self.iterable:
yield n * n
Usage:
pipeline = Squares(EvenNumbers(range(10)))
for value in pipeline:
print(value)
Each stage is lazy: values are produced only when requested by the consumer.
Implementing stateful iterators
Some iterators need to maintain internal state beyond a simple counter.
A common example is grouping items into fixed‑size batches.
class Batcher:
def __init__(self, iterable, size: int) -> None:
self._iterator = iter(iterable)
self.size = size
def __iter__(self):
return self
def __next__(self):
batch = []
for _ in range(self.size):
try:
item = next(self._iterator)
except StopIteration:
if not batch:
raise
break
batch.append(item)
return batch
This iterator yields lists of items, each of length up to size.
It is useful when you want to process data in chunks (for example, database inserts or network calls).
When to implement custom iterables and iterators
Implementing the iterator protocol directly is appropriate when:
- Your object represents a logical sequence (events, records, tasks)
- You want to hide internal storage details and expose a clean iteration interface
- You need lazy evaluation for performance or memory reasons
- You want to build reusable, composable data‑processing components
By designing classes that follow the iterator protocol, you integrate seamlessly with Python’s iteration machinery and standard library, while retaining full control over how data is produced and consumed.
Found this helpful? Share the knowledge...
... or support this project to keep it alive!
Previous
Advanced OOP Patterns
Next
Context Managers
- Why iterators are fundamental in Python
- How a for loop really works
- The iterator protocol in practice
- Distinguishing iterables from iterators
- Designing a reusable custom iterable
- Using iterators for memory‑efficient processing
- Building composable iterable pipelines
- Implementing stateful iterators
- When to implement custom iterables and iterators
