Write to a file from multiple threads asynchronously c# Write to a file from multiple threads asynchronously c# multithreading multithreading

Write to a file from multiple threads asynchronously c#


For those who prefer code, I am using following to do remote logging from web apps...

public static class LoggingExtensions{    static ReaderWriterLock locker = new ReaderWriterLock();    public static void WriteDebug(this string text)    {        try        {            locker.AcquireWriterLock(int.MaxValue); //You might wanna change timeout value             System.IO.File.AppendAllLines(Path.Combine(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase).Replace("file:\\", ""), "debug.txt"), new[] { text });        }        finally        {            locker.ReleaseWriterLock();        }    }}

Hope this saves you some time


Have a look at Asynchronous I/O. This will free up the cpu to continue with other tasks.
Combine with ReaderWriterLock as @Jack B Nimble mentioned

If by

writing to the file system as efficient as possible

you mean making the actual file I/O as fast as possible you are going to have a hard time speeding it up much, disk is just physically slower. Maybe SSD's?


What I would do is have separate worker thread(s) dedicated to the task of writing files out. When one of your other threads needs to write some data out, it should call a function to add the data to an ArrayList (or some other container/class). Inside this function, there should be a lock statement at the top to prevent more than one thread from executing simultaneously. After adding the reference to the ArrayList it returns and continues on with its chores. There are a couple of ways to handle the writing thread(s). Probably the simplest is to simply put it into an infinite loop with a sleep statement at the end so that it does not chew up your cpu(s). Another way is to use thread primitives and go into a wait state when there is no more data to be written out. This method implies that you would have to activate the thread with something like the ManualResetEvent.Set method.

There are many different ways to read in and write out files in .NET. I have written a benchmark program and give the results in my blog:

http://designingefficientsoftware.wordpress.com/2011/03/03/efficient-file-io-from-csharp

I would recommend using the Windows ReadFile and WriteFile methods if you need performance. Avoid any of the asynchronous methods since my benchmark results show that you get better performance with synchronous I/O methods.