itertools.product() yields combinations one at a time, but it first consumes its input iterables into reusable pools. Limiting the output with islice(product(...), n) does not limit that initial input consumption. Give product finite, deliberately bounded inputs before asking for a result.
A historical CPython documentation discussion raised precisely this distinction: a nested generator expression can start producing pairs from an unbounded outer source, while product must first prepare its pools. The suggested documentation change was declined because the existing explanation already described pooling. This guide treats the behavior as a documented contract, rather than a newly discovered runtime defect. The examples below use tiny local inputs and counters; none attempts to exhaust memory.
Observe input consumption before calling next
This standalone example records each value produced by a three-item input. The first assertion runs immediately after constructing the product iterator, before any output has been requested.
from itertools import product
seen = []
def source():
for number in range(3):
seen.append(number)
yield number
pairs = product(source(), ("a", "b"))
assert seen == [0, 1, 2]
assert next(pairs) == (0, "a")
assert list(pairs) == [
(0, "b"), (1, "a"), (1, "b"), (2, "a"), (2, "b")
]
assert seen == [0, 1, 2]
print("input consumed at construction; combinations yielded afterward")The input recorder is a teaching instrument. Remove it from a real pipeline: retaining every observed item would itself use memory. This test establishes when these values are requested, without measuring the allocator or assigning a byte cost to arbitrary objects.
A generator argument is therefore insufficient to establish a streaming memory bound. Inspect the consumer as well as the producer. An iterator that yields output lazily can still retain input data, and a function call can do work before returning its iterator.
Put the input boundary inside product
The following comparison uses the same finite source twice. In the first case, only one output is taken, but all six inputs have already been consumed. In the second, only two source items are admitted to the pool.
from itertools import islice, product
def source(log):
for number in range(6):
log.append(number)
yield number
output_limited_log = []
first = list(islice(product(source(output_limited_log), "ab"), 1))
assert first == [(0, "a")]
assert output_limited_log == list(range(6))
input_limited_log = []
limited = product(islice(source(input_limited_log), 2), "ab")
assert input_limited_log == [0, 1]
assert list(limited) == [(0, "a"), (0, "b"), (1, "a"), (1, "b")]
print("output cap and input cap constrain different work")Choose the input cap from the actual job requirement. Truncating a source to two records changes the set of combinations; it is not a transparent performance fix when every source record is required. Record truncation explicitly or reject an oversized workload rather than silently claiming a complete result.
For a potentially unbounded producer, bound the producer before passing it to product. This small fixture uses count but admits exactly three integers, so construction and output both finish.
from itertools import count, islice, product
bounded = product(islice(count(10), 3), ("x", "y"))
assert list(bounded) == [
(10, "x"), (10, "y"), (11, "x"),
(11, "y"), (12, "x"), (12, "y")
]
print("finite input pool from an otherwise unbounded producer")Do not run product directly on an unbounded count to reproduce the problem. The bounded comparison demonstrates the relevant boundary without creating an uncontrolled process. A source can also block while producing one item; an item-count cap alone does not provide a time limit for that producer.
Stream one outer dimension when that matches the task
If one dimension is large or unbounded and the other is a small reusable finite collection, an explicit nested generator can stream the outer dimension. The following example keeps the inner choices in a tuple that can be traversed again for every outer value.
from itertools import count, islice
choices = ("a", "b")
pairs = ((number, choice) for number in count() for choice in choices)
assert list(islice(pairs, 5)) == [
(0, "a"), (0, "b"), (1, "a"), (1, "b"), (2, "a")
]
one_shot_choices = iter(("a", "b"))
wrong = [(number, choice) for number in range(2) for choice in one_shot_choices]
assert wrong == [(0, "a"), (0, "b")]
print("a reusable inner pool is required for repeated traversal")The second comparison explains why replacing product mechanically can lose combinations. A one-shot inner iterator is exhausted during the first outer iteration. Recreating an inner generator might instead repeat I/O or other side effects. Decide who owns that work before changing the loop.
This nested order is also not a fair enumeration of two infinite dimensions: an infinite inner source prevents the outer loop from advancing. That is a different enumeration problem. The example intentionally has one unbounded outer source and a finite reusable inner collection.
Repeated dimensions need an explicit replay policy
For a finite one-shot source that supplies the same dimension twice, distinguish passing the same iterator object twice from asking product to repeat its captured pool. The first call below consumes the iterator for its first pool and finds no items for its second pool.
from itertools import product
one_shot = iter((1, 2))
assert list(product(one_shot, one_shot)) == []
assert list(product(iter((1, 2)), repeat=2)) == [
(1, 1), (1, 2), (2, 1), (2, 2)
]
assert list(product((), (1, 2))) == []
assert list(product()) == [()]
print("repeat reuses a finite pool; reusing an iterator does not rewind it")The empty-input checks make the expected result set explicit. An empty dimension leaves no combinations, while a product with no dimensions has one empty tuple. Treat those as deliberate application cases when calculating workload size.
For ordinary finite dimensions with sizes m and n, the complete output contains m times n combinations. Pooling finite inputs avoids materializing every output tuple at once, but converting the final product to a list retains that complete output. Input size, combination count and downstream storage therefore need separate limits. This guide does not benchmark memory usage, third-party iterators or concurrent consumption.
Check the boundary before changing production code
Record the input lengths when known, whether each source is reusable, and whether reading it has side effects. Then test a small finite case with an observable input counter. Compare the required result set with any bounded alternative, including empty dimensions and a one-shot source. Finally, check the downstream consumer: saving all output in a list can dominate memory even after input pooling is controlled.
For branches that share a source and drift apart, see the tee consumer-lag guide. For an iterator already advanced by next, see the remaining-items guide. Those guides address different ownership boundaries and do not make product accept an infinite input pool.
The five embedded examples were executed locally on the runtimes recorded with this draft. They demonstrate finite input consumption and exact small result sets. They do not establish a production memory budget, a hard time limit, or safety for a producer with external side effects.