.net File.Copy very slow when copying many small files (not over network)

One thing to consider is whether your copy has a user interface that updates during the copy. If so, make sure your copy is running on a separate thread, or both your UI will freeze up during the copy, and the copy will be slowed down by making blocking calls to update the UI.

I have written a similar program and in my experience, my code ran faster than a windows explorer copy (not sure about xcopy from the command prompt).

Also if you have a UI, don't update on every file; instead update every X megabytes or every Y files (whichever comes first), this keeps down the amount of updating to something the UI can actually handle. I used every .5MB or 10 files; those may not be optimal but it noticeably increased my copy speed and UI responsiveness.

Another way to speed things up is to use the Enumerate functions instead of Get functions (e.g. EnumerateFiles instead of GetFiles). These functions start returning results as soon as possible instead of waiting to return everything when the list is finished being built. They return an Enumerable, so you can just call foreach on the result: foreach(string file in System.IO.Directory.EnumerateDirectories(path)). For my program this also made a noticeable difference in speed, and would be even more helpful in cases like yours where you are dealing with directories containing many files.

One of the things that slows down IO operations the most on rotational disks is moving the disk head.

It's reasonable to assume and probably quite accurate that your many small files (that all are related to each other) are closer together on the disk than they are close to the destination of the copy (assuming you're copying from one part of a disk to another part of the same disk). If you copy for a bit then write for a bit, you open a window of opportunity for other processes to move the disk head on the source or target disk.

One thing that XCopy does much better than Copy (meaning in both cases the commands) is that XCopy reads in a bunch of files before starting to write out those files to the destination.

If you are copying files on the same disk, try allocating a large buffer to read in many files at once, then write out those files once the buffer is full).

If you are reading from one disk and writing to another disk, try starting up one thread to read from the source disk and a separate thread to write to the other disk.

There are two algorithms for faster file copy:

If source and destination are different disks Then:

• One thread reading files continuously and storing in a buffer.
• Another thread writing files continuously from that buffer.

If source and destination is same disk then:

• Read a fixed chunk of bytes, say 8K at a time, no matter how many files that is.
• Write that fixed chunk to destination, either in one file or in multiple files.

This way you will get significant performance.

Alternative is you just invoke xcopy from your .net code. Why bother doing it using File.Copy. You can capture xcopy output using Process.StandardOutput and show on the screen in order to show user what's going on.