How to properly configure Apache Httpd as Load Balancer where some hosts may be unavailable

Clients will not keep the connection open endlessly. Check your Apache server-tuning.conf and look for the KeepAliveTimeout setting. Lower it to something sensible.

Your changes to connectiontimeout and retry are indeed what you have to do. I'd lower connectiontimeout though. 10 seconds is still ages. If the back end is in the same location why not set it in miliseconds? connectiontimeout=200ms should leave plenty of time to set up the connection.

I think I found at least sort of a workaround or simple solution. mod_proxy seems to have a very long connectiontimeout by default (300 seconds). if you do not set it differently, it will take a long time until offline nodes are detected as being in "err" state.

By setting a short connectiontimeout and increasing the retry I could make it work better for me:

BalancerMember ajp://host1:8280 route=jvmRoute-8280 connectiontimeout=10 retry=600

This will ensure that failing connections are detected fairly quickly and Apache does not retry too often to reach failing servers. Unfortunately it seems Apache uses actual requests for checking the balance members and thus from time to time single requests may be slow when it tries to reach a server previously put into err-state. It seems there is no heartbeat or watchdog feature. For something like this other load balancing solutions bring such features, notably HAProxy

Read up on mod_proxy and mod_proxy_balancer for more details.

Additionally server-status via mod_status and balance manager via a page provided by mod_balancer have been a great help in diagnosing this!

It seems you have forgotten the ping tag (Actually it's called CPING - 100-Continue)

Like so:

<Proxy "balancer://www">    BalancerMember "http://192.168.0.100:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    BalancerMember "http://192.168.0.101:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    BalancerMember "http://192.168.0.102:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    BalancerMember "http://192.168.0.103:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    BalancerMember "http://192.168.0.104:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    BalancerMember "http://192.168.0.105:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    BalancerMember "http://192.168.0.106:80" max=128 ttl=300 retry=60 connectiontimeout=5 timeout=300 ping=2    SetEnv proxy-nokeepalive 1</Proxy>ProxyPass "/www/" "balancer://www/"ProxyPassReverse "/www/" "balancer://www/"