Hadoop distcp No AWS Credentials provided Hadoop distcp No AWS Credentials provided hadoop hadoop

Hadoop distcp No AWS Credentials provided


It appears the format of credentials flags has changed since the previous version. The following command works:

hadoop distcp \  -Dfs.s3a.access.key=[accesskey] \  -Dfs.s3a.secret.key=[secretkey] \  -Dfs.s3a.fast.upload=true \  -update \  s3a://[bucket]/[folder]/[filename] hdfs:///some/path


In case if some one came for with same error using -D hadoop.security.credential.provider.path, please ensure your credentials store(jceks file ) is located in distributed file system(hdfs) as distcp starts form one of the node manager node so it can access the same.


Koen's answer helped me, here is my version.

hadoop distcp \  -Dfs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider \  -Dfs.s3a.access.key=[accesskey] \  -Dfs.s3a.secret.key=[secretkey] \  -Dfs.s3a.session.token=[sessiontoken] \  -Dfs.s3a.fast.upload=true \  hdfs:///some/path s3a://[bucket]/[folder]/[filename]