What does ShFileOperation do when the recycle bin is full?
The best way to find out is to actually do it. Made my recycle bin be minimum 1 percent of drive. Created a bunch of large files and used your function to move them to recycle bin.
What I am finding out (on XP anyways) is that the function always moves it to the recycle bin; but deletes permanently the oldest deleted file. So it appears when the recycle bin fills up it employs a "first in - first out" type approach to decide which file to boot out.
I was not able to get the function to return false. Perhaps creating a file too large for the allocated recycle bin do this.