Parse config files, environment, and command-line arguments, to get a single collection of options
The argparse module makes this not nuts, as long as you're happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @
, makes it so that,
my_prog --foo=bar
is equivalent to
my_prog @baz.conf
if @baz.conf
is,
--foobar
You can even have your code look for foo.conf
automatically by modifying argv
if os.path.exists('foo.conf'): argv = ['@foo.conf'] + argvargs = argparser.parse_args(argv)
The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.
UPDATE: I finally got around to putting this on pypi. Install latest version via:
pip install configargparser
Full help and instructions are here.
Original post
Here's a little something that I hacked together. Feel free suggest improvements/bug-reports in the comments:
import argparseimport ConfigParserimport osdef _identity(x): return x_SENTINEL = object()class AddConfigFile(argparse.Action): def __call__(self,parser,namespace,values,option_string=None): # I can never remember if `values` is a list all the time or if it # can be a scalar string; this takes care of both. if isinstance(values,basestring): parser.config_files.append(values) else: parser.config_files.extend(values)class ArgumentConfigEnvParser(argparse.ArgumentParser): def __init__(self,*args,**kwargs): """ Added 2 new keyword arguments to the ArgumentParser constructor: config --> List of filenames to parse for config goodness default_section --> name of the default section in the config file """ self.config_files = kwargs.pop('config',[]) #Must be a list self.default_section = kwargs.pop('default_section','MAIN') self._action_defaults = {} argparse.ArgumentParser.__init__(self,*args,**kwargs) def add_argument(self,*args,**kwargs): """ Works like `ArgumentParser.add_argument`, except that we've added an action: config: add a config file to the parser This also adds the ability to specify which section of the config file to pull the data from, via the `section` keyword. This relies on the (undocumented) fact that `ArgumentParser.add_argument` actually returns the `Action` object that it creates. We need this to reliably get `dest` (although we could probably write a simple function to do this for us). """ if 'action' in kwargs and kwargs['action'] == 'config': kwargs['action'] = AddConfigFile kwargs['default'] = argparse.SUPPRESS # argparse won't know what to do with the section, so # we'll pop it out and add it back in later. # # We also have to prevent argparse from doing any type conversion, # which is done explicitly in parse_known_args. # # This way, we can reliably check whether argparse has replaced the default. # section = kwargs.pop('section', self.default_section) type = kwargs.pop('type', _identity) default = kwargs.pop('default', _SENTINEL) if default is not argparse.SUPPRESS: kwargs.update(default=_SENTINEL) else: kwargs.update(default=argparse.SUPPRESS) action = argparse.ArgumentParser.add_argument(self,*args,**kwargs) kwargs.update(section=section, type=type, default=default) self._action_defaults[action.dest] = (args,kwargs) return action def parse_known_args(self,args=None, namespace=None): # `parse_args` calls `parse_known_args`, so we should be okay with this... ns, argv = argparse.ArgumentParser.parse_known_args(self, args=args, namespace=namespace) config_parser = ConfigParser.SafeConfigParser() config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files] config_parser.read(config_files) for dest,(args,init_dict) in self._action_defaults.items(): type_converter = init_dict['type'] default = init_dict['default'] obj = default if getattr(ns,dest,_SENTINEL) is not _SENTINEL: # found on command line obj = getattr(ns,dest) else: # not found on commandline try: # get from config file obj = config_parser.get(init_dict['section'],dest) except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): # Nope, not in config file try: # get from environment obj = os.environ[dest.upper()] except KeyError: pass if obj is _SENTINEL: setattr(ns,dest,None) elif obj is argparse.SUPPRESS: pass else: setattr(ns,dest,type_converter(obj)) return ns, argvif __name__ == '__main__': fake_config = """[MAIN]foo:barbar:1""" with open('_config.file','w') as fout: fout.write(fake_config) parser = ArgumentConfigEnvParser() parser.add_argument('--config-file', action='config', help="location of config file") parser.add_argument('--foo', type=str, action='store', default="grape", help="don't know what foo does ...") parser.add_argument('--bar', type=int, default=7, action='store', help="This is an integer (I hope)") parser.add_argument('--baz', type=float, action='store', help="This is an float(I hope)") parser.add_argument('--qux', type=int, default='6', action='store', help="this is another int") ns = parser.parse_args([]) parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6} config_defaults = {'foo':'bar','bar':1} env_defaults = {"baz":3.14159} # This should be the defaults we gave the parser print ns assert ns.__dict__ == parser_defaults # This should be the defaults we gave the parser + config defaults d = parser_defaults.copy() d.update(config_defaults) ns = parser.parse_args(['--config-file','_config.file']) print ns assert ns.__dict__ == d os.environ['BAZ'] = "3.14159" # This should be the parser defaults + config defaults + env_defaults d = parser_defaults.copy() d.update(config_defaults) d.update(env_defaults) ns = parser.parse_args(['--config-file','_config.file']) print ns assert ns.__dict__ == d # This should be the parser defaults + config defaults + env_defaults + commandline commandline = {'foo':'3','qux':4} d = parser_defaults.copy() d.update(config_defaults) d.update(env_defaults) d.update(commandline) ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4']) print ns assert ns.__dict__ == d os.remove('_config.file')
TODO
This implementation is still incomplete. Here's a partial TODO list:
- (easy) Interaction with parser defaults
- (easy) If type conversion doesn't work, check against how
argparse
handles error messages
Conform to documented behavior
- (easy) Write a function that figures out
dest
fromargs
inadd_argument
, instead of relying on theAction
object - (trivial) Write a
parse_args
function which usesparse_known_args
. (e.g. copyparse_args
from thecpython
implementation to guarantee it callsparse_known_args
.)
Less Easy Stuff…
I haven't tried any of this yet. It's unlikely—but still possible!—that it could just work…
- (hard?) Mutual Exclusion
- (hard?) Argument Groups (If implemented, these groups should get a
section
in the config file.) - (hard?) Sub Commands (Sub-commands should also get a
section
in the config file.)
There's library that does exactly this called configglue.
configglue is a library that glues together python's optparse.OptionParser and ConfigParser.ConfigParser, so that you don't have to repeat yourself when you want to export the same options to a configuration file and a commandline interface.
It also supports environment variables.
There's also another library called ConfigArgParse which is
A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.
You might be interested in PyCon talk about configuration by Łukasz Langa - Let Them Configure!