Can windows distinguish 32-bit and 64-bit DLLs?

Windows can tell whether a given DLL is compiled for 32 bit or 64 bit platforms, but that won't help in the situation you describe. The DLL loader for your app process will stop looking as soon as it finds a DLL in the system path that fits the file name requirements for the DLL import. There are no other qualifications for matching code DLLs. (as noted in the comments, non-code resource DLLs are a different story. I suspect resource DLLs are not loaded by the core program loader but by a resource manager with different rules and objectives.)

If the first DLL in the path is 32 bit and your app is 32 bit, then the DLL load will work. If the app is 64 bit, it will fail to load the DLL and the process will abort.

If you want two DLLs to coexist on the system path, you need to give them unique file names.

As an alternative to putting the 64-bit dll in \windows\system32 and the 32-bit one in \windows\syswow64, I have found that it also works if you put the 32-bit one in a subdirectory of \Program Files (x86) and the 64-bit one in the corresponding subdirectory of \Program Files, with both of those subdirectories included in PATH.