HttpWebRequest - can I do multiple calls at the sametime from multiple threads

To be honest, I don't believe the multi-threading you're attempting to do is going to get you any performance gains. And seeing that there is no threshold for the number of threads you could be creating here the potential for worse performance than a single threaded (sequential) operation exists.

The ideal situation is that you have an asynchronous work flow. where your loop is something like this:

GetAsyncRequestMakeAsyncRequestReceiveResponseAsyncProcessResponseWaitForAllRequestProcessingToComplete(Optionally)

Such that the result of each step feeds into the next (if there is a result) and the next. And you are processing the responses as soon as you receive them rather than accumulating (joining/blocking) all responses before proceeding to process them. This kind of thing can be easily done with Tasks and ContinueWith in .NET 4.0 and seeing that you've using .NET 4.0 I'd strongly suggest you do it as described above.

But, if you're unable to convert your processing into an async workflow, then...

The method shown below is the method that makes the call out to the Url and returns a response. The method uses Async calls but is blocking since your design seems to be as such.

static string GetWebResponse(string url, NameValueCollection parameters){  var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);  httpWebRequest.ContentType = "application/x-www-form-urlencoded";  httpWebRequest.Method = "POST";  var sb = new StringBuilder();  foreach (var key in parameters.AllKeys)    sb.Append(key + "=" + parameters[key] + "&");  sb.Length = sb.Length - 1;  byte[] requestBytes = Encoding.UTF8.GetBytes(sb.ToString());  httpWebRequest.ContentLength = requestBytes.Length;  using (var requestStream = httpWebRequest.GetRequestStream())  {    requestStream.Write(requestBytes, 0, requestBytes.Length);    requestStream.Close();  }  Task<WebResponse> responseTask = Task.Factory.FromAsync<WebResponse>(httpWebRequest.BeginGetResponse, httpWebRequest.EndGetResponse, null);  using (var responseStream = responseTask.Result.GetResponseStream())  {    var reader = new StreamReader(responseStream);    return reader.ReadToEnd();  }}

You'd call it like this:

  ServicePointManager.DefaultConnectionLimit = 20;//Please test different numbers here  var tasks = new List<Task<string>>();  for (int i = 1990; i < 2090; i++)  {    var postParameters = new NameValueCollection();    postParameters.Add("data", i.ToString());    tasks.Add(Task.Factory.StartNew(() => { return GetWebResponse("http://www.abc123.com", postParameters); }));  }  Task.WaitAll(tasks.ToArray());  //At this point tasks[0].Result will be the result (The Response) of the first task  //tasks[1].Result will be the result of the second task and so on.

See if this works for you.

If you truly need multi-threaded capability, of course seeing that you're hitting only one site, one would have to measure the performance benefits since the site needs to be able to handle the onslaught of requests and on the client the cost of creating threads, only to do some I/O bound task could end up being too costly and end with no performance gains.

Besides, without adjusting the DefaultConnectionLimit in the ServicePointManager, you'll never get more than 2 threads either ways since you're going against one domain and the default limit is 2 threads per domain.

I'd stick with the code I've presented and if there is a performance problem only then would I look at doing it some other way.

Edit:when using Async I/O you're not using worker threads but I/O thread. So basically you don't want to use QueueUserWorkItem (to create threads) or you're not creating threads yourself.

The code I've presented is using Async I/O and if going to execute multiple requests simutaneously and as fast as possible.

The for loop (in the second code listing) will finish almost immediately even though in the example it loops for 100 iterations, and will then wait for all I/O requests to finish. The ThreadPool and OS will handle running the I/O jobs as soon as and as fast as possible. In fact because these jobs are I/O bound, you won't see your CPU utilization go up either (unless you're doing cpu bound work later on).

Just play with the ServiceManager.DefaultConnectionLimit to get more speedup if needed. Note that this affects the service (the server) as well, since if you make a lot of simultaneous requests the server you're calling gets heavily loaded and that may not be something you want. So it's a balance you need to strike.

After the call to Task.WaitAll, you can iterate over your tasks collection and get at the results of each task using the syntax shown in the commented line in the code listing.

It depends on how you're doing it. And if you're getting responses triggered from one thread in another, you're doing it wrong.

Perhaps a good way to do this would be to devise a unit of work which accepts your URL and post information as parameters. It will then fire up a new HttpWebRequest instance, handle the response and pass back this response (with any necessary modification/cleanup).

These units of work could then be started in discrete threads, with the responses collected. Once all threads are completed, you can then process the results.