DynamoDB: How to distribute workload over the month?

Having been in a similar place myself before, I used, and now recommend to you, to process the raw data:

• as often as you reasonably can (start with daily)
• to a format as close as possible to the desired report output
• with as much calculation/CPU intensive work done as possible

leaving as little to do at report time as possible.

This approach is entirely scaleable - the incremental frequency can be:

• reduced to as small a window as needed
• parallelised if required

It also, makes possible re-running past months reports on demand, as the report generation time should be quite small.

In my example, I shipped denormalized, pre-processed (financial calculations) data every hour to a data warehouse, then reporting just involved a very basic (and fast) SQL query.

This had the additional benefit of spreading the load on the production database server to lots of small bites, instead of bringing it to its knees once a week at invoice time (30000 invoiced produced every week).

I would use the service kinesis to produce a daily and almost real time billing.for this purpose I would create a special DynamoDB table just for the calculated data.(other option is to run it on flat files)then I would add a process which will send events to kinesis service just after you update the regular DynamoDB table.

thus when you reach the end of the month you can just execute whatever post billing calculations you have and create your CSV files from the already calculated table.

I hope that helps.

Take a look at Dynamic DynamoDB. It will increase/decrease the throughput when you need it without any manual intervention. The good news is you will not need to change the way the export job is done.