Increasing stack size with WinRM for ScriptMethods

You have two problems that conspire to make this difficult. Neither is most effectively solved by increasing your stack size, if such a thing is possible (I don't know if it is).

First, as you've experienced, remoting adds overhead to calls that reduces the available stack. I don't know why, but it's easily demonstrated that it does. This could be due to the way runspaces are configured, or how the interpreter is invoked, or due to increased bookkeeping -- I don't know the ultimate cause(s).

Second and far more damningly, your method produces a bunch of nested exceptions, rather than just one. This happens because the script method is, in effect, a script block wrapped in another exception handler that rethrows the exception as a MethodInvocationException. As a result, when you call foo(N), a block of nested exception handlers is set up (paraphrased, it's not actually PowerShell code that does this):

try {    try {         ...         try {             throw "error"         } catch {             throw [System.Management.Automation.MethodInvocationException]::new(                 "Exception calling ""foo"" with ""1"" argument(s): ""$($_.Exception.Message)""",                  $_.Exception             )         }         ...     } catch {         throw [System.Management.Automation.MethodInvocationException]::new(             "Exception calling ""foo"" with ""1"" argument(s): ""$($_.Exception.Message)""",              $_.Exception         )     } } catch {     throw [System.Management.Automation.MethodInvocationException]::new(         "Exception calling ""foo"" with ""1"" argument(s): ""$($_.Exception.Message)""",          $_.Exception     ) }

This produces a massive stack trace that eventually overflows all reasonable boundaries. When you use remoting, the problem is exacerbated by the fact that even if the script executes and produces this huge exception, it (and any results the function does produce) can't be successfully remoted -- on my machine, using PowerShell 5, I don't get a stack overflow error but a remoting error when I call foo(10).

The solution here is to avoid this particular deadly combination of recursive script methods and exceptions. Assuming you don't want to get rid of either recursion or exceptions, this is most easily done by wrapping a regular function:

$object = New-Object PSObject$object | Add-Member ScriptMethod foo {    param($depth)    function foo($depth) {        if ($depth -eq 0) {            throw "error"        }        else {            foo ($depth - 1)        }    }    foo $depth}

While this produces much more agreeable exceptions, even this can quite quickly run out of stack when you're remoting. On my machine, this works up to foo(200); beyond that I get a call depth overflow. Locally, the limit is far higher, though PowerShell gets unreasonably slow with large arguments.

As a scripting language, PowerShell wasn't exactly designed to handle recursion efficiently. Should you need more than foo(200), my recommendation is to bite the bullet and rewrite the function so it's not recursive. Classes like Stack<T> can help here:

$object = New-Object PSObject$object | Add-Member ScriptMethod foo {    param($depth)    $stack = New-Object System.Collections.Generic.Stack[int]    $stack.Push($depth)    while ($stack.Count -gt 0) {        $item = $stack.Pop()        if ($item -eq 0) {            throw "error"        } else {            $stack.Push($item - 1)        }    }}

Obviously foo is trivially tail recursive and this is overkill, but it illustrates the idea. Iterations could push more than one item on the stack.

This not only eliminates any problems with limited stack depth but is a lot faster as well.

Might be worth checking this out if you are overrunning the available memory in your remote session: Running Java remotely using PowerShell

I know it's for running a Java app but the solution updates the max memory available to a remote WinRM session.