How to programmatically create/touch a file in hdfs?
The java hadoop FileSystem api provides these types of helpers.
Here is a way to replicate a classic touch
for hdfs:
import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.fs.FSDataOutputStream;import java.io.IOException;public static void touch(String filePath) throws IOException { FileSystem hdfs = FileSystem.get(new Configuration()); Path fileToTouch = new Path(filePath); FSDataOutputStream fos = null; // If the file already exists, we append an empty String just to modify // the timestamp: if (hdfs.exists(fileToTouch)) { fos = hdfs.append(new Path(filePath)); fos.writeBytes(""); } // Otherwise, we create an empty file: else { fos = hdfs.create(new Path(filePath)); } fos.close();}
This creates an empty file if the file doesn't already exist:
hdfs.create(new Path(filePath)).close();
And appends an empty String to the file if it already exist, in order to modify the timestamp:
hdfs.append(new Path(filePath)).writeBytes("");
Configuration configuration = new Configuration();FileSystem hdfs = FileSystem.get( new URI( "hdfs://localhost:54310" ), configuration );Path file = new Path("hdfs://localhost:54310/s2013/batch/table.html");if ( hdfs.exists( file )) { hdfs.delete( file, true ); } OutputStream os = hdfs.create( file, new Progressable() { public void progress() { out.println("...bytes written: [ "+bytesWritten+" ]"); } });BufferedWriter br = new BufferedWriter( new OutputStreamWriter( os, "UTF-8" ) );br.write("Hello World");br.close();hdfs.close();