In this configuration you are directing Cloudwatch Logs to send log records to Kinesis Firehose, which is in turn configured to write the data it receives to both S3 and ElasticSearch. Thus the AWS services you are using are talking to each other as follows:

In order for one AWS service to talk to another the first service must assume a role that grants it access to do so. In IAM terminology, "assuming a role" means to temporarily act with the privileges granted to that role. An AWS IAM role has two key parts:

• The assume role policy, that controls which services and/or users may assume the role.
• The policies controlling what the role grants access to. This decides what a service or user can do once it has assumed the role.

Two separate roles are needed here. One role will grant Cloudwatch Logs access to talk to Kinesis Firehose, while the second will grant Kinesis Firehose access to talk to both S3 and ElasticSearch.

For the rest of this answer, I will assume that Terraform is running as a user with full administrative access to an AWS account. If this is not true, it will first be necessary to ensure that Terraform is running as an IAM principal that has access to create and pass roles.

Access for Cloudwatch Logs to Kinesis Firehose

In the example given in the question, the aws_cloudwatch_log_subscription_filter has a role_arn whose assume_role_policy is for AWS Lambda, so Cloudwatch Logs does not have access to assume this role.

To fix this, the assume role policy can be changed to use the service name for Cloudwatch Logs:

resource "aws_iam_role" "cloudwatch_logs" {  name = "cloudwatch_logs_to_firehose"  assume_role_policy = jsonencode({    "Version": "2012-10-17",    "Statement": [      {        "Action": "sts:AssumeRole",        "Principal": {          "Service": "logs.us-east-1.amazonaws.com"        },        "Effect": "Allow",        "Sid": "",      },    ],  })}

The above permits the Cloudwatch Logs service to assume the role. Now the role needs an access policy that permits writing to the Firehose Delivery Stream:

resource "aws_iam_role_policy" "cloudwatch_logs" {  role = aws_iam_role.cloudwatch_logs.name  policy = jsonencode({    "Statement": [      {        "Effect": "Allow",        "Action": ["firehose:*"],        "Resource": [aws_kinesis_firehose_delivery_stream.test_stream.arn],      },    ],  })}

The above grants the Cloudwatch Logs service access to call into any Kinesis Firehose action as long as it targets the specific delivery stream created by this Terraform configuration. This is more access than is strictly necessary; for more information, see Actions and Condition Context Keys for Amazon Kinesis Firehose.

To complete this, the aws_cloudwatch_log_subscription_filter resource must be updated to refer to this new role:

resource "aws_cloudwatch_log_subscription_filter" "test_kinesis_logfilter" {  name            = "test_kinesis_logfilter"  role_arn        = aws_iam_role.cloudwatch_logs.arn  log_group_name  = "loggorup.log"  filter_pattern  = ""  destination_arn = aws_kinesis_firehose_delivery_stream.test_stream.arn  # Wait until the role has required access before creating  depends_on = aws_iam_role_policy.cloudwatch_logs}

Unfortunately due to the internal design of AWS IAM, it can often take several minutes for a policy change to come into effect after Terraform submits it, so sometimes a policy-related error will occur when trying to create a new resource using a policy very soon after the policy itself was created. In this case, it's often sufficient to simply wait 10 minutes and then run Terraform again, at which point it should resume where it left off and retry creating the resource.

Access for Kinesis Firehose to S3 and Amazon ElasticSearch

The example given in the question already has an IAM role with a suitable assume role policy for Kinesis Firehose:

resource "aws_iam_role" "firehose_role" {  name = "firehose_test_role"  assume_role_policy = jsonencode({    "Version": "2012-10-17",    "Statement": [      {        "Action": "sts:AssumeRole",        "Principal": {          "Service": "firehose.amazonaws.com"        },        "Effect": "Allow",        "Sid": ""      }    ]  })}

The above grants Kinesis Firehose access to assume this role. As before, this role also needs an access policy to grant users of the role access to the target S3 bucket:

resource "aws_iam_role_policy" "firehose_role" {  role = aws_iam_role.firehose_role.name  policy = jsonencode({    "Statement": [      {        "Effect": "Allow",        "Action": ["s3:*"],        "Resource": [aws_s3_bucket.bucket.arn]      },      {        "Effect": "Allow",        "Action": ["es:ESHttpGet"],        "Resource": ["${aws_elasticsearch_domain.es.arn}/*"]      },      {        "Effect": "Allow",        "Action": [            "logs:PutLogEvents"        ],        "Resource": [            "arn:aws:logs:*:*:log-group:*:log-stream:*"        ]      },    ],  })}

The above policy allows Kinesis Firehose to perform any action on the created S3 bucket, any action on the created ElasticSearch domain, and to write log events into any log stream in Cloudwatch Logs. The final part of this is not strictly necessary, but is important if logging is enabled for the Firehose Delivery Stream, or else Kinesis Firehose is unable to write logs back to Cloudwatch Logs.

Again, this is more access than strictly necessary. For more information on the specific actions supported, see the following references:

• Action and Context Keys for Amazon S3
• Grant Firehose Access to an Amazon Elasticsearch Service Destination

Since this single role has access to write to both S3 and to ElasticSearch, it can be specified for both of these delivery configurations in the Kinesis Firehose delivery stream:

resource "aws_kinesis_firehose_delivery_stream" "test_stream" {  name        = "terraform-kinesis-firehose-test-stream"  destination = "elasticsearch"  s3_configuration {    role_arn           = aws_iam_role.firehose_role.arn    bucket_arn         = aws_s3_bucket.bucket.arn    buffer_size        = 10    buffer_interval    = 400    compression_format = "GZIP"  }  elasticsearch_configuration {    domain_arn = aws_elasticsearch_domain.es.arn    role_arn   = aws_iam_role.firehose_role.arn    index_name = "test"    type_name  = "test"  }  # Wait until access has been granted before creating the firehose  # delivery stream.  depends_on = [aws_iam_role_policy.firehose_role]}

With all of the above wiring complete, the services should have the access they need to connect the parts of this delivery pipeline.

This same general pattern applies to any connection between two AWS services. The important information needed for each case is:

• The service name for the service that will initiate the requests, such as logs.us-east-1.amazonaws.com or firehose.amazonaws.com. These are unfortunately generally poorly documented and hard to find, but can usually be found in policy examples within each service's user guide.
• The names of the actions that need to be granted. The full set of actions for each service can be found in AWS Service Actions and Condition Context Keys for Use in IAM Policies. Unfortunately again the documentation for specifically which actions are required for a given service-to-service integration is generally rather lacking, but in simple environments (notwithstanding any hard regulatory requirements or organizational policies around access) it usually suffices to grant access to all actions for a given service, using the wildcard syntax used in the above examples.