Note change in behavior for itertools.tee() with "flattening" itertables

[treyhunner]

As of Python 3.13, itertools.tee() now "flattens" other tee objects, as documented in #123885.

This behavior change seems to be from Python 3.13 because tee() objects in Python 3.12 and below do not seem to flatten.

It seems like this behavior should have a "Changed in version 3.13" call out in the documentation for `itertools.tee.

A student just ran across this change in behavior and found it confusing. I did some experimenting on different Python versions and some research and eventually discovered this new flattening behavior. A call-out in the documentation might have saved us some time.

Linked PRs

• gh-137597: Clarify flattening and bugfix for itertools.tee #137599

[pochmann3]

It was flattening before that already. What's the code you tried? There was a bug with the first iterator that got fixed. Is that the change you observed?

[pochmann3]

Here's a test of the flattening, with Python 3.7. The deeply "nested" tees are fast, thanks to the flattening. And for comparison, the deeply nested chains are slow, because they don't flatten.

from itertools import tee, chain
from time import time
import sys

n = 10**4

a, b = tee([1] * n)
for _ in range(n):
    a, b = tee(a)
t0 = time()
print(sum(a), sum(b), time() - t0)

a, b = tee([1] * n)
for _ in range(n):
    a, b = tee(b)
t0 = time()
print(sum(a), sum(b), time() - t0)

a = chain([1] * n)
for _ in range(n):
    a = chain(a)
t0 = time()
print(sum(a), time() - t0)

print(sys.version)

Output (Try it online!):

10000 10000 0.0005640983581542969
10000 10000 0.0005021095275878906
10000 1.7817797660827637
3.7.4 (default, Jul  9 2019, 16:48:28) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)]

[serhiy-storchaka]

Is not this behavior the same since Python 2.4 (ad983e7)? cc @rhettinger

[rhettinger]

A few thoughts:

• Flattening is not a new feature. It has been around since tee() was originally implemented in Python 2.4.

• tee() did have a subtle bug that went unnoticed until recently. Due to an incorrect optimization, flattening did occur but the first tee element was identical to (not a copy of) the input iterator. That has been fixed, tested, and documented in 3.12, 3.13, and the about to be final 3.14 release.

• ISTM the documentation issue that I would want fixed is to update docs in versions 3.11 and prior. However, those are in a security-fix only mode and don't accept doc updates. It would be even better if we could backport the bugfix to those older versions as well. However, our policies are at odds with those wishes.

To be clear, flattening did work in old versions, but the first returned iterator would incorrectly more forward when the other iterators pushed the input boundary forward. In those older, unbugfixed versions, the lookahead recipe can workaround the bug by skipping the first tee iterator. For example, this code works on even on very old versions of Python that haven't gotten the bugfix:

from itertools import tee

def lookahead(tee_iterator):
     "Return the next value without moving the input forward"
     [_, forked_iterator] = tee(tee_iterator, 2)
     return next(forked_iterator)

iterator = iter('abcdef')
[iterator] = tee(iterator, 1)   # Make the input peekable
assert next(iterator) == 'a'    # Move the iterator forward
assert lookahead(iterator) == 'b'
assert next(iterator) == 'b'    # Continue moving forward

[rhettinger]

Marking as closed. There was no change in behavior. All bugfixable versions of Python have been fixed and have correct docs.