Flink on K8S: how do I provide Flink configuration to the cluster?

According to the docs, all configuration has to be passed via a yaml configuration file.

It seems that jobmanager.heap.size is a common option that can be configured.

That being said, the approach on kubernetes is a little different when it comes to providing this configuration file.

The next piece of the puzzle is figuring out what the current start command is for the container you are trying to launch. I assumed you were using the official flink docker image which is good because the Dockerfile is opensource (link to repo at the bottom). They are using a complicated script to launch the flink container, but if you dig through that script you will see that it's reading the configuration yaml from /opt/flink/conf/flink-conf.yaml. Instead of trying to change this, it'll probably be easier to just mount a yaml file at that exact path in the pod with your configuration values.

Here's the github repo that has these Dockerfiles for reference.

Next question is what should the yaml file look like?From their docs:

All configuration is done in conf/flink-conf.yaml, which is expected to be a flat collection of YAML key value pairs with format key: value.

So, I'd imagine you'd create flink-conf.yaml with the following contents:

jobmanager.heap.size: 2048m

And then mount it in your kubernetes pod at /opt/flink/conf/flink-conf.yaml and it should work.

From a kubernetes perspective, it might make the most sense to make a configmap of that yaml file, and mount the config map in your pod as a file. See reference docs

Specifically, you are most interested in creating a configmap from a file and Adding a config map as a volume

Lastly, I'll call this out but I won't recommend it because of the fact that the owners of flink have marked it as an incubating feature currently, they have started providing a helm chart for Flink, and I can see that they are passing flink-conf.yaml as a config map in the helm chart templates (ignore values surrounded with {{ }} - that is helm template syntax). Here is where they mount their config map into a pod.