how to make subprocess called with call/Popen inherit environment variables

If you look at the docs for Popen, it takes an env parameter:

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process’ environment, which is the default behavior.

You've written a function that extracts the environment you want from your sourced scripts and puts it into a dict. Just pass the result as the env to the scripts you want to use it. For example:

env = {}env.update(os.environ)env.update(source('~/scripts/mySetUpFreeSurfer.sh'))env.update(source('/usr/local/freesurfer/FreeSurferEnv.sh'))# …check_output(cmd, shell=True, env=env)

Regarding

If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh.

I think you would be better off using Python to automate the process of writinga shell script newscript.sh, and then calling this script with one callsubprocess.check_output (instead of many calls to Popen, check_output,call, etc.):

newscript.sh:

#!/bin/bashsource ~/scripts/mySetUpFreeSurfer.shsource /usr/local/freesurfer/FreeSurferEnv.shrecon-all -i /media/foo/bar -subjid s1001...

and then calling

subprocess.check_output(['newscript.sh'])

import subprocessimport tempfileimport osimport statwith tempfile.NamedTemporaryFile(mode='w', delete=False) as f:    f.write('''\#!/bin/bashsource ~/scripts/mySetUpFreeSurfer.shsource /usr/local/freesurfer/FreeSurferEnv.sh''')    root = "/media/foo/"    for sub_dir in os.listdir(root):        sub = "s" + sub_dir[0:4]        anat_dir = os.path.join(root, sub_dir, "anatomical")        for directory in os.listdir(anat_dir):            time_dir = os.path.join(anat_dir, directory)            for d in os.listdir(time_dir):                dicoms_dir = os.path.join(time_dir, d, 'dicoms')                dicom_list = os.listdir(dicoms_dir)                dicom = dicom_list[0]                path = os.path.join(dicoms_dir, dicom)                cmd1 = "recon-all -i {}  -subjid {}\n".format(path, sub)                f.write(cmd1)                cmd2 = "recon-all -all -subjid {}\n".format(sub)                f.write(cmd2)filename = f.nameos.chmod(filename, stat.S_IRUSR | stat.S_IXUSR)subprocess.call([filename])os.unlink(filename)

By the way,

def source(script, update=1):    pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)    data = pipe.communicate()[0]    env = dict((line.split("=", 1) for line in data.splitlines()))    if update:        os.environ.update(env)    return env

is broken. For example, if script contains something like

VAR=`ls -1`export VAR

then

. script; env

may return output like

VAR=file1file2file3

which will result in source(script) raising a ValueError:

env = dict((line.split("=", 1) for line in data.splitlines()))ValueError: dictionary update sequence element #21 has length 1; 2 is required

There is a way to fix source: have env separate environment variables with a zero byte instead of the ambiguous newline:

def source(script, update=True):    """    http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)    http://stackoverflow.com/questions/3503719/#comment28061110_3505826 (ahal)    """    import subprocess    import os    proc = subprocess.Popen(        ['bash', '-c', 'set -a && source {} && env -0'.format(script)],         stdout=subprocess.PIPE, shell=False)    output, err = proc.communicate()    output = output.decode('utf8')    env = dict((line.split("=", 1) for line in output.split('\x00') if line))    if update:        os.environ.update(env)    return env

Fixable or not, however, you are still probably better off constructing aconglomerate shell script (as shown above) than you would be parsing env andpassing env dicts to subprocess calls.