No garbage collection while PowerShell pipeline is executing

A couple of things to point out here. First, GC calls do work in the pipeline. Here's a pipeline script that only invokes the GC:

1..10 | Foreach {[System.GC]::Collect()}

Here's the perfmon graph of GCs during the time the script ran:

However, just because you invoke the GC it doesn't mean the private memory usage will return to the value you had before your script started. A GC collect will only collect memory that is no longer used. If there is a rooted reference to an object, it is not eligible to be collected (freed). So while GC systems typically don't leak in the C/C++ sense, they can have memory hoards that hold onto objects longer than perhaps they should.

In looking at this with a memory profiler it seems the bulk of the excess memory is taken up by a copy of the string with parameter binding info:

The root for these strings look like this:

I wonder if there is some logging feature that is causing PowerShell to hang onto a string-ized form pipeline bound objects?

BTW in this specific case, it is much more memory efficient to assign to $null to ignore the output:

$null = GetDummyData

Also, if you need to simply edit a file, check out the Edit-File command in the PowerShell Community Extensions 3.2.0. It should be memory efficient as long as you don't use the SingleString switch parameter.

It's not at all uncommon to find that the native cmdlets don't satisfy perfectly when you're doing something unusual like processing a massive text file. Personally, I've found working with large files in Powershell is much better when you script it with System.IO.StreamReader:

$SR = New-Object -TypeName System.IO.StreamReader -ArgumentList 'C:\super-huge-file.txt';while ($line = $SR.ReadLine()) {    Do-Stuff $line;}$SR.Close() | Out-Null;

Note that you should use the absolute path in the ArgumentList. For me it always seems to assume you're in your home directory with relative paths.

Get-Content is simply meant to read the entire object into memory as an array and then outputs it. I think it just calls System.IO.File.ReadAllLines().

I don't know of any way to tell Powershell to discard items from the pipeline immediately upon completion, or that a function may return items asynchronously, so instead it preserves order. It may not allow it because it has no natural way to tell that the object isn't going to be used later on, or that later objects won't need to refer to earlier objects.

The other nice thing about Powershell is that you can often adopt the C# answers, too. I've never tried File.ReadLines, but that looks like it might be pretty easy to use, too.