The "right" way to add python scripting to a non-python application

One effective way to accomplish this is to use a message-passing/communicating processes architecture, allowing you to accomplish your goal with Python, but not limiting yourself to Python.

------------------------------------| App  <--> Ext. API <--> Protocol | <--> (Socket) <--> API.py <--> Script------------------------------------

This diagram attempts to show the following: Your application communicates with external processes (for example Python) using message passing. This is efficient on a local machine, and can be portable because you define your own protocol. The only thing that you have to give your users is a Python library that implements your custom API, and communicates using a Send-Receive communication loop between your user's script and your application.

Define Your Application's External API

Your application's external API describes all of the functions that an external process must be able to interact with. For example, if you wish for your Python script to be able to draw a red circle in your application, your external API may include Draw(Object, Color, Position).

Define A Communication Protocol

This is the protocol that external processes use to communicate with your application through it's external API. Popular choices for this might be XML-RPC, SunRPC, JSON, or your own custom protocol and data format. The choice here needs to be sufficient for your API. For example, if you are going to be transferring binary data then JSON might require base64 encoding, while SunRPC assumes binary communication.

Build Your Application's Messaging System

This is as simple as an infinite loop receiving messages in your communication protocol, servicing the request within your application, and replying over the same socket/channel. For example, if you chose JSON then you would receive a message containing instructions to execute Draw(Object, Color, Position). After executing the request, you would reply to the request.

Build A Messaging Library For Python (or whatever else)

This is even simpler. Again, this is a loop sending and receiving messages on behalf the library user (i.e. your users writing Python scripts). The only thing this library must do is provide a programmatic interface to your Application's External API and translate requests into your communication protocol, all hidden from your users.

Using Unix Sockets, for example, will be extremely fast.

Plugin/Application Rendezvous

A common practice for discovering application plugins is to specify a "well known" directory where plugins should be placed. This might be, for example:

~/.myapp/plugins

The next step is for your application to look into this directory for plugins that exist. Your application should have a some smarts to be able to distinguish between Python scripts that are, and are not, real scripts for your application.

Let's assume that your communication protocol specifies that each script will communicate using JSON over StdInput/StdOuput. A simple, effective approach is to specify in your protocol that the first time a script runs it sends a MAGIC_ID to standard out. That is, your application reads the first, say, 8 bytes, and looks for a specific 64-bit value that identifies it as a script.

Additionally, you should include in your External API methods that allow your scripts to identify themselves. For example, a script should be able to inform the application through the External API things such as Name, Description, Capabilities, Expectations, essentially informing the application what it is, and what it will be doing.

I don't see the problem with embedding Python with, for example, Boost.Python. You'll get everything you ask for:

• It will be embedded, and it will be an interpreter (with enough access to implement auto-completion and such)
• You can create a new interpreter per-script and have completely separated python environments
• ... and as transparent as it's possible

I mean, you'll still have to expose and implement an API, but 1) this is a good thing, 2) Blender does it too and 3) I really can't think of another way that leverages you from this work...

PS: I have little experience with Python / Boost.Python but have worked extensively with Lua / LuaBind, which is kinda the same

If you really want to be as painless as possible for you and your users, consider extending python rather than embedding.

• embedding doesn't allow easy integration with other software -- only one program that embeds python can be used at once by a python script. OTOH extending means the user can use your software anywhere python runs;
• To make stuff available for the script writer, you don't have to initialize an interpreter. the interpreter will be already initialized for you, saving you work.
• You don't have to create special built-in variables and fake modules to inject into your embedded interpreter. Just give them a real extension module instead, and you can initialize everything when your module is first imported.
• You can use distutils to distribute your software
• Tools like virtualenv can be used as-is -- you or the user don't have to come up with new tools. Your user can also use her IDE/debug tool/testing framework of choice

Embedding really buys nothing to you and your users.