Is it possible to programmatically construct a Python stack frame and start execution at an arbitrary point in the code?

The expat python bindings included in the normal Python distribution is constructing stack frames programtically. Be warned though, it relies on undocumented and private APIs.

http://svn.python.org/view/python/trunk/Modules/pyexpat.c?rev=64048&view=auto

What you generally want are continuations, which I see is already a tag on this question.

If you have the ability to work with all of the code in the system, you may want to trydoing it this way rather than dealing with the interpreter stack internals. I'm not sure how easily this will be persisted.

http://www.ps.uni-sb.de/~duchier/python/continuations.html

In practice, I would structure your workflow engine so that your script submits action objects to a manager. The manager could pickle the set of actions at any point and allowthem to be loaded and begin execution again (by resuming the submission of actions).

In other words: make your own, application-level, stack.

Stackless python is probably the best… if you don't mind totally going over to a different python distribution. stackless can serialize everything in python, plus their tasklets. If you want to stay in the standard python distribution, then I'd use dill, which can serialize almost anything in python.

>>> import dill>>> >>> def foo(a):...   def bar(x):...     return a*x...   return bar... >>> class baz(object):...   def __call__(self, a,x):...     return foo(a)(x)... >>> b = baz()>>> b(3,2)6>>> c = baz.__call__>>> c(b,3,2)6>>> g = dill.loads(dill.dumps(globals()))>>> g{'dill': <module 'dill' from '/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/dill-0.2a.dev-py2.7.egg/dill/__init__.pyc'>, 'c': <unbound method baz.__call__>, 'b': <__main__.baz object at 0x4d61970>, 'g': {...}, '__builtins__': <module '__builtin__' (built-in)>, 'baz': <class '__main__.baz'>, '_version': '2', '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x4d39d30>, '__doc__': None}

Dill registers it's types into the pickle registry, so if you have some black box code that uses pickle and you can't really edit it, then just importing dill can magically make it work without monkeypatching the 3rd party code.

Here's dill pickling the whole interpreter session...

>>> # continuing from above>>> dill.dump_session('foobar.pkl')>>>>>> ^Ddude@sakurai>$ pythonPython 2.7.5 (default, Sep 30 2013, 20:15:49) [GCC 4.2.1 (Apple Inc. build 5566)] on darwinType "help", "copyright", "credits" or "license" for more information.>>> import dill>>> dill.load_session('foobar.pkl')>>> c(b,3,2)6

dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

You also asked for where it's used to save interpreter state?

IPython can use dill to save the interpreter session to a file. https://nbtest.herokuapp.com/github/ipython/ipython/blob/master/examples/parallel/Using%20Dill.ipynb

klepto uses dill to support in-memory, to-disk, or to-database caching that avoids recomputation. https://github.com/uqfoundation/klepto/blob/master/tests/test_cache_info.py

mystic uses dill to save the checkpoints for large optimization jobs by saving the state of the optimizer as it's in progress. https://github.com/uqfoundation/mystic/blob/master/tests/test_solver_state.py

There are a couple other packages that use dill to save state of objects or sessions.