Why is a list comprehension so much faster than appending to a list?
List comprehension is basically just a "syntactic sugar" for the regular for loop. In this case the reason that it performs better is because it doesn't need to load the append attribute of the list and call it as a function at each iteration. In other words and in general, list comprehensions perform faster because suspending and resuming a function's frame, or multiple functions in other cases, is slower than creating a list on demand.
Consider the following examples :
In [1]: def f1(): ...: l = [] ...: for i in range(5): ...: l.append(i) ...: ...: ...: def f2(): ...: [i for i in range(5)] ...: In [3]: import dis In [4]: dis.dis(f1) 2 0 BUILD_LIST 0 2 STORE_FAST 0 (l) 3 4 LOAD_GLOBAL 0 (range) 6 LOAD_CONST 1 (5) 8 CALL_FUNCTION 1 10 GET_ITER >> 12 FOR_ITER 14 (to 28) 14 STORE_FAST 1 (i) 4 16 LOAD_FAST 0 (l) 18 LOAD_METHOD 1 (append) 20 LOAD_FAST 1 (i) 22 CALL_METHOD 1 24 POP_TOP 26 JUMP_ABSOLUTE 12 >> 28 LOAD_CONST 0 (None) 30 RETURN_VALUEIn [5]: In [5]: dis.dis(f2) 8 0 LOAD_CONST 1 (<code object <listcomp> at 0x7f397abc0d40, file "<ipython-input-1-45c11e415ee9>", line 8>) 2 LOAD_CONST 2 ('f2.<locals>.<listcomp>') 4 MAKE_FUNCTION 0 6 LOAD_GLOBAL 0 (range) 8 LOAD_CONST 3 (5) 10 CALL_FUNCTION 1 12 GET_ITER 14 CALL_FUNCTION 1 16 POP_TOP 18 LOAD_CONST 0 (None) 20 RETURN_VALUEDisassembly of <code object <listcomp> at 0x7f397abc0d40, file "<ipython-input-1-45c11e415ee9>", line 8>: 8 0 BUILD_LIST 0 2 LOAD_FAST 0 (.0) >> 4 FOR_ITER 8 (to 14) 6 STORE_FAST 1 (i) 8 LOAD_FAST 1 (i) 10 LIST_APPEND 2 12 JUMP_ABSOLUTE 4 >> 14 RETURN_VALUEIn [6]: You can see that on offset 18 in the first function we have an append attribute while there's no such thing in second function using list comprehension. All those extra bytecodes will make the appending approach slower and since in this case you'll have loading of the append attribute in each iteration, in the end it will make the code to take approximately twice as slower as the second function using only list comprehension.
Even factoring out the time it takes to lookup and load the append function, the list comprehension is still faster because the list is created in C, rather than built up one item at a time in Python.
# Slowtimeit.timeit(stmt=''' for i in range(10000): t.append(i)''', setup='t=[]', number=10000)# Fastertimeit.timeit(stmt=''' for i in range(10000): l(i)''', setup='t=[]; l=t.append', number=10000)# Faster stilltimeit.timeit(stmt='t = [i for i in range(10000)]', number=10000)
Citing this article, it is because the append attribute of the list isn't looked up, loaded and called as a function, which takes time and that adds up over iterations.