Using gzip to compress files to transfer with aws command Using gzip to compress files to transfer with aws command linux linux

Using gzip to compress files to transfer with aws command


$ gzip -c file.txt | aws s3 cp - s3://my_bucket/file.txt.gz

Unless you desire to have a .gz locally of file.txt, this allows you to accomplish the gzip and transfer in one step, leaving file.txt in tact.

Newer versions of the AWS CLI now allow you to steam, UNIX style, via '-' character.


Replace the | with &&. The | means pipe, which runs the aws command immediately without waiting for gzip to finish or even start. Also the | will do nothing here, since its purpose is to send the stdout output of gzip to the stdin input of aws. There is no stdout output from gzip in that form.

If you really want gzip to send its output to stdout and not write the file file.txt.gz, then you need to use gzip -c file.txt. Then you need a way for aws to take in that data. The typical way this is specified in Unix utilities is to replace the file name with -. However I don't know if gzip -c file.txt | aws s3 cp - s3://my_bucket/ will work.