Disabling ASP.NET HttpHandler response caching
A late response for others that suffers the same problem:
This is a known issue:KB 2504047
This issue occurs because the unique requests that try to access the same resources are cached as MapPathCacheInfo objects for 10 minutes.
While the objects are cached for 10 minutes, the memory consumption of the W3wp.exe process increases significantly.
You can download the hotfix here
I think this is less of an issue of caching, and more of an issue of "high memory utilization".
Two things,
If you use an IDisposable friendly object (try one that can use the "using" keyword). This will allow you to dispose of the object sooner than later, creating less pressure for the garbage collector in the long run.
for (int i = 0; i < 10000; i++) { using (System.Net.WebClient c = new System.Net.WebClient()) { System.IO.Stream stream = c.OpenRead(String.Format("http://url.com/{0}", i)); }}From your pseudo-code, I can only assume that you're using System.Net.WebHttpRequest which isn't disposable and will probably hang around longer than it should if you are making successive calls.
Secondly, if you are making successive calls to an external server, I'd put delays between each calls. This will give some breathing room as your processor will process the for-loop much faster than the network will have time to respond (and will just keep queue up the requests and slowing down the speed of ones actually being processed).
System.Threading.Thread.Sleep(250);Obviously, the best solution would be to make a single call with a list of the users you want to retreive and just deal with one webrequest/response.
Ensure the IsReusable property is set to false so IIS doesn't reuse the same request process to handle the request.