Trying to use Logstash to index FROM Cloudwatch Logs Trying to use Logstash to index FROM Cloudwatch Logs elasticsearch elasticsearch

Trying to use Logstash to index FROM Cloudwatch Logs


Other posters have mentioned that CloudFormation templates are available that will stream your logs to Amazon Elasticsearch, but if you want to go through Logstash first, this logstash plugin may be of use to you:

https://github.com/lukewaite/logstash-input-cloudwatch-logs/

This plugin allows you to ingest specific CloudWatch Log Groups, or a series of groups that match a prefix into your Logstash pipeline, and work with the data as you will. It is published on RubyGems, and can be installed like a normal Logstash plugin: bin/logstash-plugin install logstash-input-cloudwatch_logs.


As already pointed out by BMW, AWS has just introduced a dedicated CloudWatch Logs Subscription Consumer, which provides one click access to a complete CloudWatch Logs + Elasticsearch + Kibana stack by means of a resp. AWS CloudFormation template, as further illustrated in the introductory blog post.

Given you seem to have an ELK stack readily available, it shouldn't be too complex to adjust the AWS sample template to target your own endpoints instead.


In order to use the CloudFormation template (as per BMW's answer) it needs to be customized, part of this would be providing your account ID and region as a CF Resource.AWS: : AccountId and AWS: : Region are pseudo parameters that return the AWS account ID of the account in which the stack is being created, such as 123456789012, and a string representing the AWS Region in which the encompassing resource is being created, such as us-west-2. (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html)