Failed to mount Splunk config On Kubernetes - ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf

Based on the direction pointed out by @Amit-Kumar-Gupta I'll try also to give a full solution.

So this PR change makes it so that containers cannot write to secret, configMap, downwardAPI and projected volumes since the runtime will now mount them as read-only.
This change is since v1.9.4 and can lead to issues for various applications which chown or otherwise manipulate their configs.

When Splunk boots, it registers all the config files in various locations on the filesystem under ${SPLUNK_HOME} which is in our case /opt/splunk.
The error specified in the my question reflect that splunk failed to manipulate all the relevant files in the /opt/splunk/etc directory because of the change in the mounting mechanism.

Now for the solution.

Instead of mounting the configuration file directly inside the /opt/splunk/etc directory we'll use the following setup:

We'll start the docker container with a default.yml file which will be mounted in /tmp/defaults/default.yml.

For that, we'll create the default.yml file with:
docker run splunk/splunk:latest create-defaults > ./default.yml

Then, We'll go to the splunk: block and add a config: sub block under it:

splunk:  conf:    inputs:      directory: /opt/splunk/etc/system/local      content:          monitor:///opt/splunk/var/log/syslog-logs:            disabled : 0            index : syslog-index    outputs:      directory: /opt/splunk/etc/system/local      content:          tcpout:splunk-indexer:            server: splunk-indexer:9997

This setup will generate two files with a .conf postfix (Remember that the sub block start with conf:) which be owned by the correct Splunk user and group.

The inputs: section will produce the a inputs.conf with the following content:

[monitor:///opt/splunk/var/log/syslog-logs]disabled = 0index=syslog-index

In a similar way, the outputs: block will resemble the following:

[tcpout:splunk-receiver]server=splunk-receiver:9997

This is instead of the passing an environment variable directly like I did in the origin code:

SPLUNK_FORWARD_SERVER: splunk-receiver:9997

Now everything is up and running (:

Full setup of the forwarder.yaml:

apiVersion: apps/v1kind: Deploymentmetadata:  name: splunk-forwarder  labels:    app: splunk-forwarder-app    tier: splunkspec:  selector:    matchLabels:      app: splunk-forwarder-app      track: stable  replicas: 1  template:    metadata:      labels:        app: splunk-forwarder-app        tier: splunk        track: stable    spec:      volumes:      - name: configmap-forwarder        configMap:          name: splunk-forwarder-config      containers:      - name: splunk-forwarder        image: splunk/splunk:latest        imagePullPolicy : Always        env:        - name: SPLUNK_START_ARGS          value: --accept-license --answer-yes        - name: SPLUNK_PASSWORD          valueFrom:            secretKeyRef:              name: splunk-secret              key: password        volumeMounts:        - name: configmap-forwarder          mountPath: /tmp/defaults/default.yml          subPath: "default.yml"

For further reading:

https://splunk.github.io/docker-splunk/ADVANCED.html

https://github.com/splunk/docker-splunk/blob/develop/docs/ADVANCED.md

https://www.splunk.com/blog/2018/12/17/deploy-splunk-enterprise-on-kubernetes-splunk-connect-for-kubernetes-and-splunk-insights-for-containers-beta-part-1.html

https://splunk.github.io/splunk-ansible/ADVANCED.html#inventory-script

https://static.rainfocus.com/splunk/splunkconf18/sess/1521146368312001VwQc/finalPDF/FN1089_DockerizingSplunkatScale_Final_1538666172485001Loc0.pdf

There are two questions here: (1) why are you seeing that error message, and (2) how to achieve the desired behaviour you're hoping to achieve that you're trying to express through your Deployment and ConfigMap. Unfortunately, I don't believe there's a "cloud-native" way to achieve what you want, but I can explain (1), why it's hard to do (2), and point you to something that might give you a workaround.

The error message:

ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe $SPLUNK_HOME or $SPLUNK_ETC is set wrong?

does not imply that you've set those environment variables incorrectly (necessarily), it implies that Splunk is looking for a file in that location and can't read a file there, and it's providing a hint that maybe you've put the file in another place but forgot to give Splunk the hint (via the $SPLUNK_HOME or $SPLUNK_ETC environment variables) to look elsewhere.

The reason why it can't read /opt/splunk/etc/splunk-launch.conf is because, by default, the /opt/splunk directory would be populated with tons of subdirectories and files with various configurations, but because you're mounting a volume at /opt/splunk/etc/system/local/inputs.conf, nothing can be written to /opt/splunk.

If you simply don't mount that volume, or mount it somewhere else (e.g. /foo/inputs.conf) the Deployment will start fine. Of course the problem is that it won't know anything about your inputs.conf, and it'll use the default /opt/splunk/etc/system/local/inputs.conf it writes there.

I assume what you want to do is allow Splunk to generate all the directories and files it likes, you only want to set the contents of that one file. While there is a lot of nuance about how Kubernetes deals with volume mounts, in particular those coming from ConfigMaps, and in particular when using subPath, at the end of the day I don't think there's a clean way to do what you want.

I did an Internet search for "splunk kubernetes inputs.conf" and this was my first result: https://www.splunk.com/blog/2019/02/11/deploy-splunk-enterprise-on-kubernetes-splunk-connect-for-kubernetes-and-splunk-insights-for-containers-beta-part-2.html. This is from official splunk.com, and it's advising running things like kubectl cp and kubectl exec to:

"Exec" into the master pod, and run ... commands, to copy (configuration) into the (target) directory and chown to splunk user.

🤷🏾‍♂️

One solution that worked for me in K8s deployment was:

1. Ammend below to the image Dockerfile

     #RUN chmod -R 755 /opt/ansible  #RUN echo "  ignore_errors: yes" >> /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml

2. Then use that same image in your deployment from your private repo with belo env variables:#has to run as root otherwise won't let you write to $SPLUNK_HOME/S

   env:- name: SPLUNK_START_ARGSvalue: --accept-license --answer-yes --no-prompt- name: SPLUNK_USERvalue: root