Where do docker logs go when splunk server is not reachable
I am the author of the Splunk Logging Driver.
In case if Splunk is unavailable, driver holds small buffer in memory and keeps retrying. Configuration for the size of the buffer is documented no official docs for the driver https://docs.docker.com/engine/admin/logging/splunk/
SPLUNK_LOGGING_DRIVER_BUFFER_MAXIf driver cannot connect to remote server, what is the maximum amount of messages it can hold in buffer for retries.
Unfortunately this is not ideal, considering that this buffer can be filled pretty quickly and that increasing the buffer to higher number can affect your containers. But this is how most of the drivers written.
I have built another solution delivering logs and metrics to Splunk. This solution includes tiny image with collector and Splunk Certified Application. It is built on top of json-file driver, which means that when Splunk is unavailable it will just keep retrying from the position of the log files. The logs files can have their own settings for rotation, this can be configured with dockerd daemon configuration. You can read another benefits of our solution Comparing with Splunk Logging Driver. And how to get started with Monitoring Docker.