Follow iterators and generators
In the decorator lesson, you separated a function from the result of calling it. Here we will separate a collection of values from an object that produces the next value when asked.
That difference matters when a framework yields chunks of a response or reads many records. With two chapter titles, you will learn to:
- Request the next item from an iterator.
- Trace where a generator pauses and resumes.
- Recognise when a sequence has already been consumed.
Ask for the next item
A list keeps its items available as a collection. An iterator keeps track of where it has reached while producing items:
titles = ["Python and Roots", "Python and Branches"]
cursor = iter(titles)
print(next(cursor))
print(next(cursor))
iter asks for an iterator over the list. next asks it for one item and advances it. The first call returns Python and Roots; the second returns Python and Branches.
A third next(cursor) raises StopIteration: there is no next item. A for loop uses this mechanism and handles normal exhaustion for you. The exception is not evidence that the two earlier items were wrong.
The list still contains both values. Creating iter(titles) again gives a new iterator starting from the beginning. Asking the exhausted cursor again does not rewind it.
Let a function yield one result at a time
A function containing yield produces a generator, a kind of iterator whose progress is described by the function:
def cleaned_titles(titles):
for title in titles:
clean = title.strip()
if clean:
yield clean
titles = [" Python and Roots ", "", "Python and Branches"]
results = cleaned_titles(titles)
Calling cleaned_titles creates a generator object. It does not immediately run the whole loop and build a result list. Work begins as the caller requests items:
print(next(results))
print(next(results))
The first request runs until the first yield and returns the cleaned first title. The generator remembers its position. The second request resumes it, skips the empty string and yields Python and Branches.
yield supplies an item and suspends the function. return ends the function. Replacing yield clean with return clean would change the function into one that stops after the first matching title, rather than producing a sequence of results.
Notice when a consumer uses up the work
You can collect the generated values with list:
results = cleaned_titles(titles)
first_pass = list(results)
second_pass = list(results)
print(first_pass)
print(second_pass)
The first list contains the two cleaned titles. The second is empty because that generator was exhausted. Create a new generator with another call if you need to repeat the work.
This can explain an apparently mysterious bug: an AI logs a stream by converting it to a list, then tries to use the same stream to form an answer. The logging step consumed it. Logging is not always an observation without an effect.
Decide whether lazy work helps
Lazy work happens as results are requested. A generator can avoid storing every result at once. It can also stop early when a caller has enough information.
That does not mean it stores nothing, uses no resources or makes every operation faster. Our example starts from a list that is already in memory. Some generators retain references to large inputs. Others hold an open file or network connection while they are paused.
If an iterator reads a file, the file must still be open when the caller requests the next item. Returning an iterator from inside a with block can be wrong if the block closes the resource before the caller uses it. Check the lifetime of both the producer and its consumer.
Similarly, an exception can happen later, during iteration rather than when the generator was created. A try that covers only results = cleaned_titles(titles) would not catch a failure raised while a later consumer requests an item.
For two public chapter titles, a list is usually clearer and its storage cost is tiny. A generator becomes useful when producing results on demand meets a real need. Choose it because of that need, not because a framework example happens to use yield.
Read a stream without assuming it is a final answer
Frameworks may expose a stream of events or text pieces. Each piece is one item in a sequence; it is not necessarily a complete answer or an approved side effect. You still need to know the event shape, the completion signal and what to do after an error.
The Python iterator skill gives you the right questions: what advances the sequence, which consumer has already read it, and when is the work finished? The framework supplies its specific event contract.
The Python iterator tutorial and generator tutorial are the source references.
Predict the next request
With the explanation closed:
- Does calling
cleaned_titles(titles)immediately clean every title? - Why is
second_passempty in the example? - What changes if
yield cleanbecomesreturn clean? - Why might converting a stream to a list for logging change later behaviour?
- Where would you put error handling if failure happens while a caller requests an item?
Compare after your attempt
The call creates a generator; requesting items runs its body. The first list conversion exhausted that generator. Replacing its only yield with return makes an ordinary function that ends at its first result. A consumer can use up a one-shot stream. Error handling must cover the iteration that can fail, while preserving enough context to diagnose the cause.
In async and await, we will separate another pair of events: creating asynchronous work and actually waiting for its result.