Graceful File Reading without Locking

After the discussions in the comments my proposal would be like so:

• Create a partition on your data server, about 5GB for safety.
• Create a Windows Service Project in C# that would monitor your data driver / location.
• When a file has been modified then create a local copy of the file, containing the same directory structure and place on the new partition.
• Create another service that would do the following:
  • Monitor Bandwidth Usages
  • Monitor file creations on the temporary partition.
  • Transfer several files at a time (Use Threading) to your FTP Server, abiding by the bandwidth usages at the current time, decreasing / increasing the worker threads depending on network traffic.
  • Remove the files from the partition that have successfully transferred.

So basically you have your drives:

• C: Windows Installation
• D: Share Storage
• X: Temporary Partition

Then you would have following services:

• LocalMirrorService - Watches D: and copies to X: with the dir structure
• TransferClientService - Moves files from X: to ftp server, removes from X:
  • Also use multi threads to move multiples and monitors bandwidth.

I would bet that this is the idea that you had in mind but this seems like a reasonable approach as long as your really good with your application development and your able create a solid system that would handle most issues.

When a user edits a document in Microsoft Word for instance, the file will change on the share and it may be copied to X: even though the user is still working on it, within windows there would be an API see if the file handle is still opened by the user, if this is the case then you can just create a hook to watch when the user actually closes the document so that all there edits are complete, then you can migrate to drive X:.

this being said that if the user is working on the document and there PC crashes for some reason, the document / files handle may not get released until the document is opened at a later date, thus causing issues.

For anyone in a similar situation (I'm assuming the person who asked the question implemented a solution long ago), I would suggest an implementation of rsync.

rsync.net's Windows Backup Agent does what is described in method 1, and can be run as a service as well (see "Advanced Usage"). Though I'm not entirely sure if it has built-in bandwidth limiting...

Another (probably better) solution that does have bandwidth limiting is Duplicati. It also properly backs up currently-open or locked files. Uses SharpRSync, a managed rsync implementation, for its backend. Open source too, which is always a plus!