Using gzip to compress files to transfer with aws command
$ gzip -c file.txt | aws s3 cp - s3://my_bucket/file.txt.gz
Unless you desire to have a .gz locally of file.txt, this allows you to accomplish the gzip and transfer in one step, leaving file.txt in tact.
Newer versions of the AWS CLI now allow you to steam, UNIX style, via '-' character.
Replace the |
with &&
. The |
means pipe, which runs the aws
command immediately without waiting for gzip
to finish or even start. Also the |
will do nothing here, since its purpose is to send the stdout output of gzip
to the stdin input of aws
. There is no stdout output from gzip
in that form.
If you really want gzip
to send its output to stdout and not write the file file.txt.gz
, then you need to use gzip -c file.txt
. Then you need a way for aws
to take in that data. The typical way this is specified in Unix utilities is to replace the file name with -
. However I don't know if gzip -c file.txt | aws s3 cp - s3://my_bucket/
will work.