Is Python `list.extend(iterator)` guaranteed to be lazy? Is Python `list.extend(iterator)` guaranteed to be lazy? python python

Is Python `list.extend(iterator)` guaranteed to be lazy?


I don't think the issue is lazy vs non lazy because, either in slice assignment or in list extend, you need all the elements of the iterator and these elements are consumed at once (in the normal case). The issue you raised is more important: are these operations atomic or not atomic? See one definition of "atomicity" in Wikipedia:

Atomicity guarantees that each transaction is treated as a single "unit", which either succeeds completely, or fails completely.

Have a look at this example (CPython 3.6.8):

>>> def new_iterator(): return (1/(i-2) for i in range(5))>>> L = []>>> L[:] = new_iterator()Traceback (most recent call last):...ZeroDivisionError: division by zero>>> L[]

The slice assignment failed because of the exception (i == 2 => 1/(i - 2) raises an exception) and the list was left unchanged. Hence, the slice assignement operation is atomic.

Now, the same example with: extend:

>>> L.extend(new_iterator())Traceback (most recent call last):...ZeroDivisionError: division by zero>>> L[-0.5, -1.0]

When the exception was raised, the two first elements were already appended to the list. The extend operation is not atomic, since a failure does not leave the list unchanged.

Should the extend operation be atomic or not? Frankly I have no idea about that, but as written in @wim's answer, the real issue is that it's not clearly stated in the documentation (and worse, the documentation asserts that extend is equivalent to the slice assignment, which is not true in the reference implementation).


Is Python list.extend(iterator) guaranteed to be lazy?

No. On the contrary, it's documented that

l.extend(iterable)

is equivalent to

l[len(l):] = iterable

In CPython, such a slice assignment will first convert a generator on the right hand side into a list anyway (see here), i.e. it's consuming the iterable all at once.

The example shown in your question is, strictly speaking, contradicting the documentation. I've filed a documentation bug, but it was promptly closed by Raymond Hettinger.

As an aside, there are less convoluted ways to demonstrate the discrepancy. Just define a failing generator:

def gen():    yield 1    yield 2    yield 3    uh-oh

Now L.extend(gen()) will modify L, but L[:] = gen() will not.