How to efficiently import many large JSON files directly from S3 into MongoDB

mongodb amazon-s3

You don't need to store intermediate files, you can pipe the output of s3 file to stdout and you can get input to mongoimport from stdin.

Your full command would look something like:

s3cmd get s3://<yourFilename> - | mongoimport -d <dbName> -c <collectionName>

note the - which says send the file to stdout rather than to a filename.

mongodb amazon-s3

The alternative of using s3cmd is to use aws s3 which has fewer features but comes installed with AWS CLI.

The command would look like:

aws S3 cp <your s3 URI> - | \mongoimport \--db <dbName> \--collection <collectionName> \

The - sends the file to stdout

CodeHunter