Amazon Kinesis + Integration Tests

It might already be too late to give the solution but I will add what my team has done to replicate AWS resources locally as we use a lot of Kinesis, DynamoDb, S3 and cloudWatch.

We have created wrappers around Localstack -> https://github.com/localstack/localstack that allow us to spin up local instances of the necessary services as docker containers using docker-compose.

A typical docker-compose.yml file for us looks like:

version: '2'services:  localstack:    image: "localstack/localstack"    environment:      - SERVICES=kinesis,dynamodb,cloudwatch    ports:      - "4568"      - "4569"      - "4582"

Then during the setup phase for the integration-tests, our wrapper fires up docker-compose up and runs the tests against the local infrastructure. Later during tear-down, the wrapper kills the containers.

I ran into the same issue and the only mock implementation I found so far was a nodejs one: https://github.com/mhart/kinesaliteIt does the trick - I managed to run my Java Kinesis client against it, just had to set the endpoint on the kinesis.properties:

kinesisEndpoint=http://localhost:4567

The downside is that it is not trivial to use it during build time tests - need to figure a way to start the mock kinesis before the test (using a maven plugin or something), didn't get to it yet..

Just a small addition to the existing answers. BTW, they are great, you should really use tools like localstack to start fake AWS services before the test during the test phase.

If you're using JUnit 5 in your tests, your life could be even simpler with JUnit 5 extensions for AWS, a few JUnit 5 extensions that could be useful for testing AWS-related code. These extensions can be used to inject clients for AWS service mocks provided by tools like localstack. Both AWS Java SDK v 2.x and v 1.x are supported:

@ExtendWith(DynamoDB.class)class AmazonDynamoDBInjectionTest {    @AWSClient(        endpoint = Endpoint.class    )    private AmazonDynamoDB client;    @Test    void test() throws Exception {        Assertions.assertNotNull(client);        Assertions.assertEquals(            Collections.singletonList("table"),            client.listTables().getTableNames().stream().sorted().collect(Collectors.toList())        );    }}

Here, client will be just injected in your test class and configured according to the Endpoint configuration class.