Barry Warsaw proprosed List Comprehensions 16 years ago in PEP 202.
The original concept was that it provided a more concise way to write loops so that:
t = [expr(x, y) for x in s1 for y in s2]
was just a short way to write:
_t = []
for x in s1:
for y in s2
_t.append(expr(x, y))
t = _t
del _t
The original implementation reflected that design. I later added the LIST_APPEND opcode to give the list comprehensions a speed advantage over the unrolled code.
The question of whether to expose the loop induction variable didn't get much discussion until I proposed Generator Expressions in PEP 279 and decided to give them the behavior of hiding the loop induction variables so that the behavior would match that of a normal unrolled generator function.
When set and dict comprehensions (displays) came along afterwards, they were given the latter behavior because there was precedent, because there was a mechanism to implement that precedent, and to provide a short-cut for then common practice of creating dicts and sets with generator comprehensions:
s = {expr(x) for x in t}
was a short-form for:
s = set(expr(x) for x in t)
The advent of Python 3 gave us an opportunity to make a four forms (listcomps, genexps, set and dict displays) consistent about hiding the loop induction variable.
The current state in Python 3 has the advantage of being consistent between all four variants and matches how mathematicians treat bound and free variables.
There are some disadvantages as well. List comprehensions can no longer be cleanly explained as being equivalent to the unrolled version. The disassembly is harder to explore be cause you need to drill into the internal code object. Tracing the execution with PDB is no fun because you go up and down the stack. It is more difficult to explain scoping -- formerly, all you had was locals/globals/builtins, but now we have locals/nonlocals/globals/builtins plus variables bound in list comps, genexps, set/dict displays plus exception instances that are only visible inside the except-block.
> Pretty sure it was a bug that they kept for backwards compat, not intentional
Yep, as demonstrated by only list comprehensions exposing this behaviour (on python 2) whereas none of generator comprehensions, dictionary comprehensions or set comprehensions do:
>>> [x for x in range(5)]
[0, 1, 2, 3, 4]
>>> x
4
>>> {y for y in range(5)}
set([0, 1, 2, 3, 4])
>>> y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined
>>> {y:y for y in range(5)}
{0: 0, 1: 1, 2: 2, 3: 3, 4: 4}
>>> y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined
>>> list(y for y in range(5))
[0, 1, 2, 3, 4]
>>> y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined
>>>
