ASP.NET Multithreading Web Requests

Multi-threading is the right choice, but I would call it doing stuff asynchronously.

Anyway, you should know that multi-threading works different in IIS.

IIS worker process will finish a request once all child threads end, and this is a big problem, because you don't want to hold a worker process for a long time but re-use it for other requests. Actually it's a thread pool.

This is why ASP.NET offers its own approach to implement asynchronity and if you use the right approach, IIS will be able to process more requests at once because asynchronous work will be executed outside IIS process model.

I would suggest you to read more about ASP.NET async:

• http://www.hanselman.com/blog/TheMagicOfUsingAsynchronousMethodsInASPNET45PlusAnImportantGotcha.aspx

• http://blogs.msdn.com/b/tmarq/archive/2010/04/14/performing-asynchronous-work-or-tasks-in-asp-net-applications.aspx

Conclusion: use asynchronous work and this will make a more efficient use of server resources, but first learn more about how to do it in the right way!

Multithreading is not recommended on ASP.NET; because ASP.NET/IIS does its own multithreading, and any multithreading you do will interfere with those heuristics.

What you really want is concurrency, more specifically asynchronous concurrency since your operations are I/O-bound.

The best approach is to use HttpClient with the Task-based Asynchronous Pattern:

public async Task<OccupationSearch> GetOccupationAsync(string requestUrl){  // You can also reuse HttpClient instances instead of creating a new one each time  using (var client = new HttpClient())  {    var response = client.GetStringAsync(requestUrl);    return new JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));  }}

This is asynchronous, and you can easily make it concurent by using Task.WhenAll:

List<string> urls = ...;OccupationSearch[] results = await Task.WhenAll(urls.Select(GetOccupationAsync));

I would argue that Multithreading (within reason) would provide benefits as the way your code is written know the calls to GetResponse().GetResponseStream() are blocking.

One of the easiest ways to improve performance is to use a Parallel.ForEach:

        var urls = new List<string>();        var results = new ConcurrentBag<OccupationSearch>();        Parallel.ForEach(urls, url =>        {            WebRequest request = WebRequest.Create(requestUrl);            string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();            var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));            results.Add(result);        });

If you are using .NET 4.5 the new approach is to use async/await. Msdn has a pretty extensive article on this very topic here: http://msdn.microsoft.com/en-us/library/hh300224.aspx and http://msdn.microsoft.com/en-us/library/hh696703.aspx

Scott Hanselman also has a good looking Blog Post on this topic: http://www.hanselman.com/blog/TheMagicOfUsingAsynchronousMethodsInASPNET45PlusAnImportantGotcha.aspx