flask-cache memcache connection auto-reconnect
Being disconnected from ELB is very common and also very difficult to debug. Here are a few things that might help:
Debugging Ideas
- Attempt to debug the problem in a staging environment with only oneinstance connected to ELB.
Make sure you have application logging with time stamps and that if you catch all exceptions in Python (which is generally not a great idea), that you log the exception. It is possible you have a subtle and hidden bug that appears to be something else if you are catching all exceptions.
Simulate the failure (i.e. manually remove "one" instance from ELB), now look at your logs and make sure you see this manifested in your logs. If you can reproduce the same behavior than you can figure out how to fix it.
Look into a web service automated testing tool like https://loader.io/. This can be very helpful to simulate the conditions when the disconnects appear to happen.
Try the same application with a different load balancer, i.e. HAProxy (I would potentially try this last).