What is the architecture for recover logs from a storage account to a Elasticsearch deployed on KUBERNETES? What is the architecture for recover logs from a storage account to a Elasticsearch deployed on KUBERNETES? kubernetes kubernetes

What is the architecture for recover logs from a storage account to a Elasticsearch deployed on KUBERNETES?


There is an approach you could consider for accomplishing your task. Kubernetes in Azure allows you to deploy Azure File Shares with your containers. If you move your logs to a file share, you should be able to accomplish your project.

I'd recommend checking Diego's post here, it shows how to access logs from a storage account, specifically FileShare.

here's a blurb from the tutorial:

1- *Create an Azure Storage account with your own parameters (deployment model: resource manager; type: general purpose). You will need the Azure Storage account name in the next step.

2- Modify the storageAccount parameter in this .yaml file with your Azure Storage account name and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-storageclass.yaml. It will create a Kubernetes volume using your Azure File Storage account.

3- Deploy this .yaml file to your Kubernetes cluster: kubectl apply -f sidecar-pvc.yaml. It will create a volume claim for your volume in order to use it in your pod.

4- Modify your application deployment .yaml file by adding (modify the logFileDirectory parameter) this content and deploy it to your Kubernetes cluster. It will add the volume to your pod and store on it the logFilesDirectory.

5- Modify the logReaderName (you will filter the logs using this parameter), logFileDirectory (x2) and the logFileName with your data in this .yaml file and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-logreaderpod.yaml. It will create the Log Reader pod and write the logFile content to the STDOUT.

The Log Reader pod uses tail command to write in the STDOUT. You can modify the tail command, for example, to write different files (extension .log) in the same STDOUT: tail -n+1 -f //*.logOnce you deploy the Log Reader, you can start to check the logs filtered by the pod name (you selected it when you deployed the last .yaml file):

kubectl get podskubectl logs <podname>