File Access Strategy in a Multi-Threaded Environment (Web App)

Here is the code that I use to make sure a file is not locked by another process. It's not 100% foolproof, but it gets the job done most of the time:

    /// <summary>    /// Blocks until the file is not locked any more.    /// </summary>    /// <param name="fullPath"></param>    bool WaitForFile(string fullPath)    {        int numTries = 0;        while (true)        {            ++numTries;            try            {                // Attempt to open the file exclusively.                using (FileStream fs = new FileStream(fullPath,                    FileMode.Open, FileAccess.ReadWrite,                     FileShare.None, 100))                {                    fs.ReadByte();                    // If we got this far the file is ready                    break;                }            }            catch (Exception ex)            {                Log.LogWarning(                   "WaitForFile {0} failed to get an exclusive lock: {1}",                     fullPath, ex.ToString());                if (numTries > 10)                {                    Log.LogWarning(                        "WaitForFile {0} giving up after 10 tries",                         fullPath);                    return false;                }                // Wait for the lock to be released                System.Threading.Thread.Sleep(500);            }        }        Log.LogTrace("WaitForFile {0} returning true after {1} tries",            fullPath, numTries);        return true;    }

Obviously you can tweak the timeouts and retries to suit your application. I use this to process huge FTP files that take a while to be written.

If you're locking on a object stored as a static then the lock should work for all threads in the same Application Domain, but perhaps you need to upload a code sample so we can have a look at the offending lines.

That said, one thought would be to check if IIS is configured to run in Web Garden mode (i.e. more than 1 process executing your application) which would break your locking logic. While you could fix such a situation with a mutex it'd be easier to reconfigure your application to execute in a single process, although you'd be wise to check the performance before and after messing with the web garden settings as it can potentially affect performance.

You could maybe create the file with a temporary name ("data.xml_TMP"), and when it's ready change the name to what it is supposed to be. That way, no other process will be accessing it before it is ready.