Kubernetes WatchConnectionManager: Exec Failure: HTTP 403

This is happened in the Kubernetes v1.15.3, Kubernetes v1.14.6, Kubernetes v1.13.10 version, the project spark operator have a workaround adding adding the kubernetes-client last version (kubernetes-client-4.4.2.jar) and you need to delete the actually version in your image, you can add the next lines in your Dockerfile

RUN rm $SPARK_HOME/jars/kubernetes-client-3.0.0.jarADD https://repo1.maven.org/maven2/io/fabric8/kubernetes-client/4.4.2/kubernetes-client-4.4.2.jar $SPARK_HOME/jars

If you ever get an Invocation error after applying this fix, you might want to upgrade the kubernetes-model-*.jar to 4.4.2 as well.

But if you can't/don't want to upgrade your k8s-client from 3.0.0 to 4.4.2 since it's quite a huge hop and could result to legacy issues, here's a more in-depth (and more technical) solution and explanation as to what happened (ref: #SPARK-28921)

When the Kubernetes URL used doesn't specify a port (e.g., https://example.com/api/v1/...), the origin header for watch requests ends up with a port of -1 (e.g. https://example.com:-1). This happens because calling getPort() on a java.net.URL object that does not have a port explicitly specified will always return -1. The return value was always just inserted into the origin header.

https://github.com/fabric8io/kubernetes-client/pull/1669

As you can see here, the fix wasn't applied until kubernetes-client-4.4.x. What I did was patch the current .jar and build a customized .jar:

1. Clone / download Kubernetes-client source code from repo
2. Apply the changes in this commit.
3. Build the .jar file using maven plugin.
4. Replace the /opt/spark/jars/kubernetes-client-3.0.0.jar with the customized .jar.

if the error still exits downgrade Kubernetes version(v.1.14.3) and use spark 2.4.3as there has been recent update in Kubernetes with the fix in golang.

reference https://andygrove.io/2019/08/apache-spark-regressions-eks/