How to alter the recursive locking behaviour of Windows Mutex?
You cannot alter the fact that Windows Mutexes are recursive. And while Posix threads are not recursive by default, you can use pthread_mutexattr_settype() with the PTHREAD_MUTEX_RECURSIVE flag to make one so.
Locking a mutex in Windows is actually quite an expensive operation, and best suited to inter-process synchronisation. For a mutex used only within a single process, a critical section is normally used, however these are re-entrant also. As nobugz states, you would need to use a semaphore, initialised with a maximum count of 1, to get non-recursive synchronisation.
A semaphore object is like a special counter, that can be atomically incremented and decremented across threads (or processes, if created shared). By creating one with a maximum count of 1, you get the non-recursive behaviour you require.
As long as you program in Windows, avoid reimplementing the behavior of its Mutex. It being re-entrant by the same thread is absolutely essential for its defined behavior.
A sync object without thread affinity is a semaphore that counts to 1. Use CreateSemaphore().
Fwiw, it is very curious that you need this kind of behavior. It sounds like you are trying to use the same sync object in multiple places inappropriately. You can use the semaphore but you'll lose concurrency potential. Consider using more than one mutex instead.
I suggest to use Read/Write locks (aka SRW). Like Windows Mutex, they are not recursive. Like Critical Sections, they are light and do not call kernel if they are free (Benchmarks).