GRPc-netty upstream connect error or disconnect/reset before headers GRPc-netty upstream connect error or disconnect/reset before headers kubernetes kubernetes

GRPc-netty upstream connect error or disconnect/reset before headers


From github

We faced the same error. The reason for us was because tcp_keepalive was set too high for our upstream service. We changed keepalive_time to 300 seconds and the problem went away.

It makes sense in our case because we have Envoy pointing to a network load balancer (aws) which has a 350s idle timeout.

We added this to our envoy config.

clusters:    - name: grpc-service      connect_timeout: 0.25s      http2_protocol_options: {}      upstream_connection_options:        tcp_keepalive:          keepalive_time: 300