Remove-Item errors when run at end of TFS build
You should be able to delete everything if you go from the bottom up. Sort the results of Get-ChildItem by their full name in descending order before deleting the items:
Get-ChildItem -Path $directory -Force -Recurse | Sort-Object -Property FullName -Descending | Remove-Item -Recurse -Force
Not sure this is your fix but I do it just a bit differently
Get-ChildItem $FolderPath | ForEach-Object ($_){ Remove-Item $_.FullName -Recurse -Force}