Python Iterators and Generators

A generator computes values on demand instead of building a full list first. That matters for large files, streaming data, and sequences that never end.

Environment

Examples are verified on Python 3.13; everything here is the standard library.

Iterables versus iterators

• Iterable: anything a for loop can walk, i.e. it implements __iter__. Lists, strings, dicts, sets and file objects all qualify.
• Iterator: implements both __iter__ and __next__, remembers its position, and can only move forward.

items = [1, 2, 3]
it = iter(items)      # get the iterator
next(it)              # 1
next(it)              # 2
next(it)              # 3
next(it)              # raises StopIteration

A for loop is exactly that: take the iterator, call next() repeatedly, stop on StopIteration.

for x in items:
    print(x)

# equivalent to
it = iter(items)
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    print(x)

A list can be traversed any number of times because each iter() call returns a fresh iterator. The iterator itself is single-use.

Generator functions: yield

When a function body contains yield, calling it no longer runs the body — it returns a generator object:

def count_up(n: int):
    for i in range(n):
        print(f"yielding {i}")
        yield i

gen = count_up(3)     # nothing printed yet
next(gen)             # yielding 0 → returns 0
next(gen)             # yielding 1 → returns 1
list(gen)             # yielding 2 → [2]

Each next() runs until it hits a yield, which suspends the frame and keeps the local variables. The next call resumes right after the yield. When the function returns, the generator raises StopIteration.

That suspension is the source of laziness: values are computed when the consumer asks for them.

Why generators: constant memory

Compare two ways of scanning a large log:

from pathlib import Path

# Whole file in memory: a 2 GB file costs 2 GB of RAM
lines = Path("huge.log").read_text(encoding="utf-8").splitlines()
errors = [line for line in lines if "ERROR" in line]

# Generator: only the current line is in memory
def read_lines(path: str):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

errors = (line for line in read_lines("huge.log") if "ERROR" in line)
for line in errors:
    print(line)

Generator functions and generator expressions compose into a pipeline: each stage pulls one item at a time, so memory stays flat no matter how big the input is.

yield from

yield from delegates iteration to another iterable, and it also forwards send and throw to the inner generator:

def chain(*iterables):
    for it in iterables:
        yield from it          # same as: for x in it: yield x

list(chain([1, 2], [3]))       # [1, 2, 3]

Walking a directory tree recursively becomes readable:

from pathlib import Path

def walk(path: Path):
    for child in path.iterdir():
        if child.is_dir():
            yield from walk(child)
        else:
            yield child

send, throw and close

A generator can also receive values:

def accumulator():
    total = 0
    while True:
        value = yield total      # the value of this expression comes from send()
        if value is None:
            break
        total += value

acc = accumulator()
next(acc)          # prime it: returns 0
acc.send(10)       # 10
acc.send(5)        # 15
acc.close()

• gen.send(value) injects a value that becomes the result of the suspended yield.
• gen.throw(exc) raises an exception at the suspension point.
• gen.close() ends the generator by raising GeneratorExit inside it, so finally blocks still run.

Application code rarely needs send; it shows up in coroutines and framework internals. Generators created with yield from forward all three calls to the delegate.

Writing a class-based iterator

Use a class when the iterator needs extra methods, or when you want a fresh traversal per call:

class Range2:
    def __init__(self, stop: int):
        self.stop = stop
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

list(Range2(3))    # [0, 1, 2]
list(Range2(3))    # [] — the same object is already exhausted

That last line is the classic trap: this object is its own iterator. To make it restartable, return a separate iterator object from __iter__ or use a generator function instead.

Common mistakes

1. Iterating a generator twice

gen = (x * 2 for x in [1, 2])
list(gen)     # [2, 4]
list(gen)     # [] — exhausted

Wrap the logic in a function that returns a new generator on each call.

2. Expecting return value to yield a value

return inside a generator does not produce an item; the value lands in StopIteration.value and most consumers ignore it. Use yield for output.

3. Forgetting that errors are delayed

The body only runs during iteration, so a bad path in open() raises at the first next(), far from the call that created the generator. Tracebacks point at the consumer, not the definition — keep that in mind when debugging.

4. Leaking resources by abandoning a generator

Files or connections opened inside a generator are only cleaned up when the generator is exhausted, garbage collected, or closed. Prefer with inside the generator, and call gen.close() (or use contextlib.closing) in code paths that may stop early.

5. Using generators where a list is clearer

For small, reused data a list is faster and easier to inspect (len, indexing, printing). Laziness pays off only when the input is large, expensive, or infinite.

6. A generator that silently stops early

If you return before the loop finishes — for example inside an except branch — consumers see a normal end of iteration, not an error. Make the failure explicit by raising or logging instead.

Summary

• Iterables can be traversed repeatedly; iterators move forward once.
• yield suspends the function, keeps its state, and makes computation lazy.
• Generator pipelines keep memory flat when processing large files or streams.
• yield from delegates to another iterable and forwards send/throw; send itself is for coroutine-style code.
• A class-based iterator that returns self from __iter__ is single-use too.

Next: the standard library pieces you reach for daily — datetime, collections, json, and subprocess.