Overflow

The logistic sigmoid function is proned to overflow in NumPy as the signal strength increase. Try to append the following code line:

np.clip( signal, -500, 500 )

This will limit the values in NumPy matrises to be within the given interval. This will in turn prevent the precision overflow in the sigmoid function.

>>> arrarray([[-900, -600, -300],       [   0,  300,  600]])>>> np.clip( arr, -500, 500)array([[-500, -500, -300],       [   0,  300,  500]])

Implementation

This is the snippet I'm using in my projects:

def sigmoid_function( signal ):    # Prevent overflow.    signal = np.clip( signal, -500, 500 )    # Calculate activation signal    signal = 1.0/( 1 + np.exp( -signal ))    return signal#end

Why does the Sigmoid function overflow?

As the training progress, the network improves its precision. As this precision approaches perfection, the sigmoid signal will either approach 1 from below or 0 for above. Eg: either 0.99999999999... or 0.00000000000000001...

Since NumPy is focused on performing highly accurate numerical operations, it will try to maintain the highest possible precision and thus cause an overflow error. Note: This error message could be ignored by setting:

np.seterr( over='ignore' )