Which is the best way to allow configuration options be overridden at the command line in Python?

I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:

Default with no user input:

$ ./argparse-partial.pyOption is "default"

Default from configuration file:

$ cat argparse-partial.config [Defaults]option=Hello world!$ ./argparse-partial.py -c argparse-partial.config Option is "Hello world!"

Default from configuration file, overridden by commandline:

$ ./argparse-partial.py -c argparse-partial.config --option overrideOption is "override"

argprase-partial.py follows. It is slightly complicated to handle -h for help properly.

import argparseimport ConfigParserimport sysdef main(argv=None):    # Do argv default this way, as doing it in the functional    # declaration sets it at compile time.    if argv is None:        argv = sys.argv    # Parse any conf_file specification    # We make this parser with add_help=False so that    # it doesn't parse -h and print help.    conf_parser = argparse.ArgumentParser(        description=__doc__, # printed with -h/--help        # Don't mess with format of description        formatter_class=argparse.RawDescriptionHelpFormatter,        # Turn off help, so we print all options in response to -h        add_help=False        )    conf_parser.add_argument("-c", "--conf_file",                        help="Specify config file", metavar="FILE")    args, remaining_argv = conf_parser.parse_known_args()    defaults = { "option":"default" }    if args.conf_file:        config = ConfigParser.SafeConfigParser()        config.read([args.conf_file])        defaults.update(dict(config.items("Defaults")))    # Parse rest of arguments    # Don't suppress add_help here so it will handle -h    parser = argparse.ArgumentParser(        # Inherit options from config_parser        parents=[conf_parser]        )    parser.set_defaults(**defaults)    parser.add_argument("--option")    args = parser.parse_args(remaining_argv)    print "Option is \"{}\"".format(args.option)    return(0)if __name__ == "__main__":    sys.exit(main())

Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.

I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:

subp.set_defaults(**dict(conffile.items(subn)))

This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.

A more complete example is below:

####### content of example.cfg:# [sub1]# verbosity=10# gggg=3.5# [sub2]# host=localhostimport ConfigParserimport argparseparser = argparse.ArgumentParser()subparsers = parser.add_subparsers()parser_sub1 = subparsers.add_parser('sub1')parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')parser_sub1.add_argument('-G', type=float, dest='gggg')parser_sub2 = subparsers.add_parser('sub2')parser_sub2.add_argument('-H','--host', dest='host')conffile = ConfigParser.SafeConfigParser()conffile.read('example.cfg')for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):    subp.set_defaults(**dict(conffile.items(subn)))print parser.parse_args(['sub1',])# Namespace(gggg=3.5, verbosity=10)print parser.parse_args(['sub1', '-V', '20'])# Namespace(gggg=3.5, verbosity=20)print parser.parse_args(['sub1', '-V', '20', '-G','42'])# Namespace(gggg=42.0, verbosity=20)print parser.parse_args(['sub2', '-H', 'www.example.com'])# Namespace(host='www.example.com')print parser.parse_args(['sub2',])# Namespace(host='localhost')