How to write safe/correct multi-threaded code in .NET?

This could be a massive list - read Joe Duffy's excellent "Concurrent Programming On Windows" for much more detail. This is pretty much a brain dump...

• Try to avoid calling into significant chunks of code while you own a lock
• Avoid locking on references which code outside the class might also lock on
• If you ever need to acquire more than one lock at a time, always acquire those locks in the same order
• Where reasonable, use immutable types - they can be shared freely between threads
• Other than immutable types, try to avoid the need to share data between threads
• Avoid trying to make your types threadsafe; most types don't need to be, and usually the code which needs to share data will need to control the locking itself
• In a WinForms app:
  • Don't perform any long-running or blocking operations on the UI thread
  • Don't touch the UI from any thread other than the UI thread. (Use BackgroundWorker, Control.Invoke/BeginInvoke)
• Avoid thread-local variables (aka thread-statics) where possible - they can lead to unexpected behaviour, particularly on ASP.NET where a request may be served by different threads (search for "thread agility" and ASP.NET)
• Don't try to be clever. Lock-free concurrent code is hugely difficult to get right.
• Document the threading model (and thread safety) of your types
• Monitor.Wait should almost always be used in conjunction with some sort of check, in a while loop (i.e. while (I can't proceed) Monitor.Wait(monitor))
• Consider the difference between Monitor.Pulse and Monitor.PulseAll carefully every time you use one of them.
• Inserting Thread.Sleep to make a problem go away is never a real fix.
• Have a look at "Parallel Extensions" and the "Coordination and Concurrency Runtime" as ways of making concurrency simpler. Parallel Extensions is going to be part of .NET 4.0.

In terms of debugging, I don't have very much advice. Using Thread.Sleep to boost the chances of seeing race conditions and deadlocks can work, but you've got to have quite a reasonable understanding of what's wrong before you know where to put it. Logging is very handy, but don't forget that the code goes into a sort of quantum state - observing it via logging is almost bound to change its behaviour!

I'm not sure how well this will help for the particular application you're working with, but here are two approaches borrowed from functional programming for writing multithreaded code:

Immutable objects

If you need to share state between threads, the state should be immutable. If one thread needs to make a change to the object, it creates a brand new version of the object with the change instead of mutating the object's state.

Immutability does not inherently limit the kind of code you can write, nor is it inefficient. There are lots of implementations of immutable stacks, a variety of immutable trees that form the basis of maps and sets, and other kinds of immutable data structures, and many (if not all) immutable data structures are just as efficient as their mutable counterparts.

Since objects are immutable, its not possible for one thread to mutate shared state under your nose. This means you don't need to acquire locks to write multithreaded code. This approach eliminates a whole class of errors related to deadlocking, livelocking, and raceconditions.

Erlang-style message passing

You don't need to learn the language, but have a look at Erlang to see how it approaches concurrency. Erlang apps can scale pretty much indefinitely because each process is completely seperate from all the others (note: these are not exactly processes, but not exactly threads either).

Processes fire up and simply spin a loop waiting for messages: messages are recieved in the form of tuples, which the process can then pattern match against to see if the message is meaningful. Processes can send other messages, but they are indifferent to whoever recieves the message.

Advantanges to this style is an elimination of locks, when one process fails it doesn't bring down your entire app. Here's a nice summary of Erlang-style concurrency: http://www.defmacro.org/ramblings/concurrency.html

Use FIFOs. Lots of them. It's the ancient secret of the hardware programmer, and it's saved my bacon more than once.