Cloudwatch to Elasticsearch parse/tokenize log event before push to ES

From what I can see your best bet is what you're suggesting, a CloudWatch log triggered lambda that reformats the logged data into your ES prefered format and then posts it into ES.

You'll need to subscribe this lambda to your CloudWatch logs. You can do this on the lambda console, or the cloudwatch console (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html).

The lambda's event payload will be: { "awslogs": { "data": "encoded-logs" } }. Where encoded-logs is a Base64 encoding of a gzipped JSON.

For example, the sample event (https://docs.aws.amazon.com/lambda/latest/dg/eventsources.html#eventsources-cloudwatch-logs) can be decoded in node, for example, using:

const zlib = require('zlib');const data = event.awslogs.data;const gzipped = Buffer.from(data, 'base64');const json = zlib.gunzipSync(gzipped);const logs = JSON.parse(json);console.log(logs);/*  { messageType: 'DATA_MESSAGE',    owner: '123456789123',    logGroup: 'testLogGroup',    logStream: 'testLogStream',    subscriptionFilters: [ 'testFilter' ],    logEvents:     [ { id: 'eventId1',         timestamp: 1440442987000,         message: '[ERROR] First test message' },       { id: 'eventId2',         timestamp: 1440442987001,         message: '[ERROR] Second test message' } ] }*/

From what you've outlined, you'll want to extract the logEvents array, and parse this into an array of strings. I'm happy to give some help on this too if you need it (but I'll need to know what language you're writing your lambda in- there are libraries for tokenizing ODL- so hopefully it's not too hard).

At this point you can then POST these new records directly into your AWS ES Domain. Somewhat crypitcally the S3-to-ES guide gives a good outline of how to do this in python: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-s3-lambda-es

You can find a full example for a lambda that does all this (by someone else) here: https://github.com/blueimp/aws-lambda/tree/master/cloudwatch-logs-to-elastic-cloud