Robust endless loop for server written in Python

I would do this in a similar way you're thinking of, using the 'you shall not pass' Gandalf exception handler except Exception to catch all non-system-exiting exceptions while creating a black-listed set of exceptions that should pass and end be re-raised.

Using the Gandalf handler will make sure GeneratorExit, SystemExit and KeyboardInterrupt (all system-exiting exceptions) pass and terminate the program if no other handlers are present higher in the call stack. Here is where you can check with type(e) that a __class__ of a caught exception e actually belongs in the set of black-listed exceptions and re-raise it.

As a small demonstration:

import exceptions  # Py2.x only# dictionary holding {exception_name: exception_class}excptDict = vars(exceptions)exceptionNames = ['MemoryError', 'OSError', 'SystemError'] # and others# set containing black-listed exceptionsblackSet = {excptDict[exception] for exception in exceptionNames}

Now blackSet = {OSError, SystemError, MemoryError} holding the classes of the non-system-exiting exceptions we want to not handle.

A try-except block can now look like this:

try:    # calls that raise exceptions:except Exception as e:    if type(e) in blackSet: raise e # re-raise    # else just handle it

An example which catches all exceptions using BaseException can help illustrate what I mean. (this is done for demonstration purposes only, in order to see how this raising will eventually terminate your program). Do note: I'm not suggesting you use BaseException; I'm using it in order to demonstrate what exception will actually 'pass through' and cause termination (i.e everything that BaseException catches):

for i, j in excptDict.iteritems():    if i.startswith('__'): continue  # __doc__ and other dunders    try:        try:            raise j        except Exception as ex:            # print "Handler 'Exception' caught " + str(i)            if type(ex) in blackSet:                raise ex               except BaseException:        print "Handler 'BaseException' caught " + str(i)# prints exceptions that would cause the system to exit     Handler 'BaseException' caught GeneratorExitHandler 'BaseException' caught OSErrorHandler 'BaseException' caught SystemExitHandler 'BaseException' caught SystemErrorHandler 'BaseException' caught KeyboardInterruptHandler 'BaseException' caught MemoryErrorHandler 'BaseException' caught BaseException

Finally, in order to make this Python 2/3 agnostic, you can try and import exceptions and if that fails (which it does in Python 3), fall-back to importing builtins which contains all Exceptions; we search the dictionary by name so it makes no difference:

try:    import exceptions    excDict = vars(exceptions)except ImportError:    import builtins     excDict = vars(builtins)

I don't know if there's a smarter way to actually do this, another solution might be instead of having a try-except with a signle except, having 2 handlers, one for the black-listed exceptions and the other for the general case:

try:    # calls that raise exceptions:except tuple(blackSet) as be:  # Must go first, of course.    raise beexcept Exception as e:    # handle the rest

The top-most exception is BaseException. There are two groups under that:

• Exception derived
• everything else

Things like Stopiteration, ValueError, TypeError, etc., are all examples of Exception.

Things like GeneratorExit, SystemExit and KeyboardInterrupt are not descended from Execption.

So the first step is to catch Exception and not BaseException which will allow you to easily terminate the program. I recommend also catching GeneratorExit as 1) it should never actually be seen unless it is raised manually; 2) you can log it and restart the loop; and 3) it is intended to signal a generator has exited and can be cleaned up, not that the program should exit.

The next step is to log each exception with enough detail that you have the possibility of figuring out what went wrong (when you later get around to debugging).

Finally, you have to decide for yourself which, if any, of the Exception derived exceptions you want to terminate on: I would suggest RuntimeError and MemoryError, although you may be able to get around those by simply stopping and restarting your server loop.

So, really, it's up to you.

If there is some other error (such as IOError when trying to load a config file) that is serious enough to quit on, then the code responsible for loading the config file should be smart enough to catch that IOError and raise SystemExit instead.

As far as whitelist/blacklist -- use a black list, as there should only be a handful, if any, Exception-based exceptions that you need to actually terminate the server on.