Allowing Lambda in a VPC to access an Elasticsearch domain in the same VPC Allowing Lambda in a VPC to access an Elasticsearch domain in the same VPC elasticsearch elasticsearch

Allowing Lambda in a VPC to access an Elasticsearch domain in the same VPC


Noticed two problems in your setup

  1. You are creating only one subnet in the stack and assigning only one subnet to Lambda. You need to assign multiple subnets to the Lambda
  2. You need to fix access policy for ES. I manually updated to "Don't require signing requst with IAM credential" for my testing in console.

I updated Cloudformation template to create python based lambda handler to query elastic search from the same vpc. It is not complete but if you make sure of the above mentioned issues then it should work.

AWSTemplateFormatVersion: 2010-09-09Description: The AWS CloudFormation tutorialResources:  SomeDeploymentBucket:    Type: 'AWS::S3::Bucket'  AppLogGroup:    Type: 'AWS::Logs::LogGroup'    Properties:      LogGroupName: /aws/lambda/some-lambda    # ========= The Lambda Execution Role =========  IamRoleLambdaExecution:    Type: 'AWS::IAM::Role'    Properties:      AssumeRolePolicyDocument:        Version: 2012-10-17        Statement:          - Effect: Allow            Principal:              Service:                - lambda.amazonaws.com            Action:              - 'sts:AssumeRole'      Policies:        - PolicyName: !Join            - '-'            - - dev              - some-app              - lambda          PolicyDocument:            Version: 2012-10-17            Statement:              - Effect: Allow                Action:                  - 's3:*'                  - 'rds-db:connect'                  - 'rds:*'                  - 'es:*'                Resource: '*'      Path: /      RoleName: !Join        - '-'        - - some-app          - dev          - eu-west-1          - lambdaRole      ManagedPolicyArns:        - !Join          - ''          - - 'arn:'            - !Ref 'AWS::Partition'            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'    # ========= The Lambda =========  AppLambdaFunction:    Type: AWS::Lambda::Function    DependsOn:      - AppLogGroup      - IamRoleLambdaExecution    Properties:      FunctionName: some-lambda      Handler: index.lambda_handler      Runtime: python2.7      Timeout: 60      MemorySize: 1024      Role: !GetAtt        - IamRoleLambdaExecution        - Arn      VpcConfig:        SecurityGroupIds:          - !Ref xxxVPCSecurityGroup        SubnetIds:          - !Ref xxxLambdaSubnet1          - !Ref xxxLambdaSubnet2      Code:        ZipFile: !Sub |          from __future__ import print_function          import boto3          iam = boto3.client('iam')          def lambda_handler(event, context):            print('called lambda_handler')    # ========= VPC =========  xxxVPC:    Type: 'AWS::EC2::VPC'    Properties:      CidrBlock: 172.31.0.0/16      InstanceTenancy: default      EnableDnsSupport: 'true'      EnableDnsHostnames: 'true'  xxxVPCSecurityGroup:    Type: 'AWS::EC2::SecurityGroup'    Properties:      GroupName: VPC SG      GroupDescription: VPC Security Group      VpcId: !Ref xxxVPC  xxxLambdaSubnet1:    Type: 'AWS::EC2::Subnet'    Properties:      VpcId: !Ref xxxVPC      CidrBlock: 172.31.32.0/20  xxxLambdaSubnet2:    Type: 'AWS::EC2::Subnet'    Properties:      VpcId: !Ref xxxVPC      CidrBlock: 172.31.16.0/20    # ========= Elasticsearch =========  xxxESSecurityGroup:    Type: 'AWS::EC2::SecurityGroup'    Properties:      GroupName: ES SG      GroupDescription: ES Security group      VpcId: !Ref xxxVPC      SecurityGroupIngress:        - IpProtocol: -1          FromPort: 0          ToPort: 65535          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup  xxxElasticSearch:    Type: 'AWS::Elasticsearch::Domain'    Properties:      AccessPolicies:        Version: 2012-10-17        Statement:          - Action:              - 'es:*'              - 'ec2:*'              - 's3:*'            Principal:              AWS:                - '*'            Resource: '*'            Effect: Allow      DomainName: es-xxx-domain      EBSOptions:        EBSEnabled: true        Iops: 0        VolumeSize: 20        VolumeType: "gp2"      AdvancedOptions:        rest.action.multi.allow_explicit_index: 'true'      ElasticsearchClusterConfig:        InstanceCount: 1        InstanceType: m4.large.elasticsearch        DedicatedMasterEnabled: 'false'      VPCOptions:        SecurityGroupIds:          - !Ref xxxESSecurityGroup        SubnetIds:          - !Ref xxxLambdaSubnet1

Updated Lambda function handler code

  import urllib2def lambda_handler(event, context):    print('called lambda_handler')    data = ''    url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'    req = urllib2.Request(url, data, {'Content-Type': 'application/json'})    f = urllib2.urlopen(req)    for x in f:        print(x)    f.close()