git tries to delete a directory on checkout
This error message comes from this patch (in compat/mingw.c), and this one (also in compat/mingw.c).
It usually means your directory is "busy", or isn't empty (but contains only private files, ie non-versioned files).
So make sure that, when checking out a branch, you don't have a running process making your specific directory "busy".
Ted M Young helpfully comments:
Another definition of "busy" is that it's your current directory!
It seems safest to do acdto the root directory and then change directories to avoid this error.