iPython debugger raises `NameError: name ... is not defined`

You've got two core problems here. The first is that (when calling pdb.set_trace() interactively in IPython) you're debugging IPython's guts instead of the scope you wanted. The second is that list comprehension scoping rules interact badly with cases where the variables present in enclosing scopes can't be determined statically, such as in debuggers or class bodies.

The first problem pretty much only happens when typing pdb.set_trace() into an IPython interactive prompt, which isn't a very useful thing to do, so the simplest way to avoid the problem is to just not do that. If you want to do it anyway, you can enter the r command a few times until pdb says you're out of IPython's guts. (Don't overshoot, or you'll end up in a different part of IPython's guts.)

The second problem is an essentially unavoidable interaction of heavily entrenched language design decisions. Unfortunately, it's unlikely to go away. List comprehensions in a debugger only work in a global scope, not while debugging a function. If you want to build a list while debugging a function, the easiest way is probably to use the interact command and write a for loop.

Here's the full combination of effects going into this.

1. pdb.set_trace() triggers pdb on the next trace event, not at the point where pdb.set_trace() is called.

The trace function mechanism used by pdb and other Python debuggers only triggers on certain specific events, and "when a trace function is set" is unfortunately not one of those events. Normally, the next event is either a 'line' event for the next line or a 'return' event for the end of the current code object's execution, but that's not what happens here.

2. IPython sets a displayhook to customize expression statement handling.

The mechanism Python uses to display the result of expression statements is sys.displayhook. When you do 1+2 at the interactive prompt:

>>> 1+23

sys.displayhook is what prints the 3 instead of discarding it. It also sets _. When the result of an expression statement is None, such as with the expression pdb.set_trace(), sys.displayhook does nothing, but it's still called.

IPython replaces sys.displayhook with its own custom handler, responsible for printing the Out[n]: thingy, for setting entries in the Out record, for calling IPython custom pretty-printing, and for all sorts of other IPython conveniences. For our purposes, the important thing is that IPython's displayhook is written in Python, so the next trace event is a 'call' event for the displayhook.

pdb starts debugging inside IPython's displayhook.

In [1]: import pdb; pdb.set_trace()--Call--> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()-> def __call__(self, result=None):

3. List comprehensions create a new scope.

People didn't like how list comprehensions leaked the loop variable into the containing scope in Python 2, so list comprehensions get their own scope in Python 3.

4. pdb uses eval, which interacts really poorly with closure variables.

Python's closure variable mechanism relies on static scope analysis that's completely incompatible with how eval works. Thus, new scopes created inside eval have no access to closure variables; they can only access globals.

Putting it all together, in IPython, you end up debugging the IPython displayhook instead of the scope you're running interactive code in. Since you're inside IPython's displayhook, your x = 1 assignment goes into the displayhook's locals. The subsequent list comprehension would need access to the displayhook's locals to access x, but that would be a closure variable to the list comprehension, which doesn't work with eval.

Outside of IPython, sys.displayhook is written in C, so pdb can't enter it, and there's no 'call' event for it. You end up debugging the scope you intended to debug. Since you're in a global scope, x = 1 goes in globals, and the list comprehension can access it.

You would have seen the same effect if you tried to run x = 1; [x for i in range(3)] while debugging any ordinary function.

A possible solution/workaround is to run

 globals().update(locals())

before running the list comprehension in (i)pdb.