Illegal characters in Powershell filepath error
In a later comment you say that it's a malicious file, so presumably its name was crafted to evade normal attempts to delete it.
Indeed, " is not a legal filename character on NTFS on Windows, and attempts to target such a file directly will fail - no escaping technique will help.
You may be able to gain access to the file via enumeration / wildcard matching, obtain its short 8.3 name, and use that to delete it.
Update: The OP has confirmed that deleting via the 8.3 file name worked.
$filelist = @("c$\Users\$($_.Name)\101__lyrics.vbs","c$\Users\$($_.Name)\101" "" "lyrics.vbs","c$\Users\$($_.Name)\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\101" "" "lyrics.vbs>")
PowerShell fails when multiple strings are placed together. In more detail:
"c$\Users\$($_.Name)\101" "" "lyrics.vbs"
is a syntax error. To include a double quote in a double-quoted string use the back-tick (`) as an escape:
"c$\Users\$($_.Name)\101`" `"`" `"lyrics.vbs"