How to do I run encrypted distcp from hdfs to s3? How to do I run encrypted distcp from hdfs to s3? hadoop hadoop

How to do I run encrypted distcp from hdfs to s3?


Recent Hadoop versions should do the transfer using client side encryption just fine.

To be safe and not fall victim to specifics of your setup, I would simply ensure secure transfer by putting a bucket policy like the below one in place, forbidding non-secure transfer.

{    "Statement":[        {            "Action": "s3:*",            "Effect":"Deny",            "Principal": "*",            "Resource":"arn:aws:s3:::bucketname/*",            "Condition":{                "Bool":                { "aws:SecureTransport": false }            }        }    ]}