Parse config files, environment, and command-line arguments, to get a single collection of options Parse config files, environment, and command-line arguments, to get a single collection of options python python

Parse config files, environment, and command-line arguments, to get a single collection of options


The argparse module makes this not nuts, as long as you're happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @, makes it so that,

my_prog --foo=bar

is equivalent to

my_prog @baz.conf

if @baz.conf is,

--foobar

You can even have your code look for foo.conf automatically by modifying argv

if os.path.exists('foo.conf'):    argv = ['@foo.conf'] + argvargs = argparser.parse_args(argv)

The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.


UPDATE: I finally got around to putting this on pypi. Install latest version via:

   pip install configargparser

Full help and instructions are here.

Original post

Here's a little something that I hacked together. Feel free suggest improvements/bug-reports in the comments:

import argparseimport ConfigParserimport osdef _identity(x):    return x_SENTINEL = object()class AddConfigFile(argparse.Action):    def __call__(self,parser,namespace,values,option_string=None):        # I can never remember if `values` is a list all the time or if it        # can be a scalar string; this takes care of both.        if isinstance(values,basestring):            parser.config_files.append(values)        else:            parser.config_files.extend(values)class ArgumentConfigEnvParser(argparse.ArgumentParser):    def __init__(self,*args,**kwargs):        """        Added 2 new keyword arguments to the ArgumentParser constructor:           config --> List of filenames to parse for config goodness           default_section --> name of the default section in the config file        """        self.config_files = kwargs.pop('config',[])  #Must be a list        self.default_section = kwargs.pop('default_section','MAIN')        self._action_defaults = {}        argparse.ArgumentParser.__init__(self,*args,**kwargs)    def add_argument(self,*args,**kwargs):        """        Works like `ArgumentParser.add_argument`, except that we've added an action:           config: add a config file to the parser        This also adds the ability to specify which section of the config file to pull the         data from, via the `section` keyword.  This relies on the (undocumented) fact that        `ArgumentParser.add_argument` actually returns the `Action` object that it creates.        We need this to reliably get `dest` (although we could probably write a simple        function to do this for us).        """        if 'action' in kwargs and kwargs['action'] == 'config':            kwargs['action'] = AddConfigFile            kwargs['default'] = argparse.SUPPRESS        # argparse won't know what to do with the section, so         # we'll pop it out and add it back in later.        #        # We also have to prevent argparse from doing any type conversion,        # which is done explicitly in parse_known_args.          #        # This way, we can reliably check whether argparse has replaced the default.        #        section = kwargs.pop('section', self.default_section)        type = kwargs.pop('type', _identity)        default = kwargs.pop('default', _SENTINEL)        if default is not argparse.SUPPRESS:            kwargs.update(default=_SENTINEL)        else:              kwargs.update(default=argparse.SUPPRESS)        action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)        kwargs.update(section=section, type=type, default=default)        self._action_defaults[action.dest] = (args,kwargs)        return action    def parse_known_args(self,args=None, namespace=None):        # `parse_args` calls `parse_known_args`, so we should be okay with this...        ns, argv = argparse.ArgumentParser.parse_known_args(self, args=args, namespace=namespace)        config_parser = ConfigParser.SafeConfigParser()        config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]        config_parser.read(config_files)        for dest,(args,init_dict) in self._action_defaults.items():            type_converter = init_dict['type']            default = init_dict['default']            obj = default            if getattr(ns,dest,_SENTINEL) is not _SENTINEL: # found on command line                obj = getattr(ns,dest)            else: # not found on commandline                try:  # get from config file                    obj = config_parser.get(init_dict['section'],dest)                except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): # Nope, not in config file                    try: # get from environment                        obj = os.environ[dest.upper()]                    except KeyError:                        pass            if obj is _SENTINEL:                setattr(ns,dest,None)            elif obj is argparse.SUPPRESS:                pass            else:                setattr(ns,dest,type_converter(obj))        return ns, argvif __name__ == '__main__':    fake_config = """[MAIN]foo:barbar:1"""    with open('_config.file','w') as fout:        fout.write(fake_config)    parser = ArgumentConfigEnvParser()    parser.add_argument('--config-file', action='config', help="location of config file")    parser.add_argument('--foo', type=str, action='store', default="grape", help="don't know what foo does ...")    parser.add_argument('--bar', type=int, default=7, action='store', help="This is an integer (I hope)")    parser.add_argument('--baz', type=float, action='store', help="This is an float(I hope)")    parser.add_argument('--qux', type=int, default='6', action='store', help="this is another int")    ns = parser.parse_args([])    parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}    config_defaults = {'foo':'bar','bar':1}    env_defaults = {"baz":3.14159}    # This should be the defaults we gave the parser    print ns    assert ns.__dict__ == parser_defaults    # This should be the defaults we gave the parser + config defaults    d = parser_defaults.copy()    d.update(config_defaults)    ns = parser.parse_args(['--config-file','_config.file'])    print ns    assert ns.__dict__ == d    os.environ['BAZ'] = "3.14159"    # This should be the parser defaults + config defaults + env_defaults    d = parser_defaults.copy()    d.update(config_defaults)    d.update(env_defaults)    ns = parser.parse_args(['--config-file','_config.file'])    print ns    assert ns.__dict__ == d    # This should be the parser defaults + config defaults + env_defaults + commandline    commandline = {'foo':'3','qux':4}     d = parser_defaults.copy()    d.update(config_defaults)    d.update(env_defaults)    d.update(commandline)    ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])    print ns    assert ns.__dict__ == d    os.remove('_config.file')

TODO

This implementation is still incomplete. Here's a partial TODO list:

Conform to documented behavior

  • (easy) Write a function that figures out dest from args in add_argument, instead of relying on the Action object
  • (trivial) Write a parse_args function which uses parse_known_args. (e.g. copy parse_args from the cpython implementation to guarantee it calls parse_known_args.)

Less Easy Stuff…

I haven't tried any of this yet. It's unlikely—but still possible!—that it could just work…


There's library that does exactly this called configglue.

configglue is a library that glues together python's optparse.OptionParser and ConfigParser.ConfigParser, so that you don't have to repeat yourself when you want to export the same options to a configuration file and a commandline interface.

It also supports environment variables.

There's also another library called ConfigArgParse which is

A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.

You might be interested in PyCon talk about configuration by Łukasz Langa - Let Them Configure!