hadoop copy a local file system folder to HDFS
In Short
hdfs dfs -put <localsrc> <dest>
In detail with an example:
Checking source and target before placing files into HDFS
[cloudera@quickstart ~]$ ll files/total 132-rwxrwxr-x 1 cloudera cloudera 5387 Nov 14 06:33 cloudera-manager-rwxrwxr-x 1 cloudera cloudera 9964 Nov 14 06:33 cm_api.py-rw-rw-r-- 1 cloudera cloudera 664 Nov 14 06:33 derby.log-rw-rw-r-- 1 cloudera cloudera 53655 Nov 14 06:33 enterprise-deployment.json-rw-rw-r-- 1 cloudera cloudera 50515 Nov 14 06:33 express-deployment.json[cloudera@quickstart ~]$ hdfs dfs -lsFound 1 itemsdrwxr-xr-x - cloudera cloudera 0 2017-11-14 00:45 .sparkStaging
Copy files HDFS using -put
or -copyFromLocal
command
[cloudera@quickstart ~]$ hdfs dfs -put files/ files
Verify the result in HDFS
[cloudera@quickstart ~]$ hdfs dfs -lsFound 2 itemsdrwxr-xr-x - cloudera cloudera 0 2017-11-14 00:45 .sparkStagingdrwxr-xr-x - cloudera cloudera 0 2017-11-14 06:34 files[cloudera@quickstart ~]$ hdfs dfs -ls filesFound 5 items-rw-r--r-- 1 cloudera cloudera 5387 2017-11-14 06:34 files/cloudera-manager-rw-r--r-- 1 cloudera cloudera 9964 2017-11-14 06:34 files/cm_api.py-rw-r--r-- 1 cloudera cloudera 664 2017-11-14 06:34 files/derby.log-rw-r--r-- 1 cloudera cloudera 53655 2017-11-14 06:34 files/enterprise-deployment.json-rw-r--r-- 1 cloudera cloudera 50515 2017-11-14 06:34 files/express-deployment.json
If you copy a folder from local then it will copy folder with all its sub folders to HDFS.
For copying a folder from local to hdfs, you can use
hadoop fs -put localpath
or
hadoop fs -copyFromLocal localpath
or
hadoop fs -put localpath hdfspath
or
hadoop fs -copyFromLocal localpath hdfspath
Note:
If you are not specified hdfs path then folder copy will be copy to hdfs with the same name of that folder.
To copy from hdfs to local
hadoop fs -get hdfspath localpath