Linux of equivalent CryptProtectMemory

The closest equivalent in Linux to CryptProtectMemory() is gcry_malloc_secure() in libgcrypt. The secure memory allocated will be locked in memory; gcry_free() will zeroize and deallocate it. Other crypto libraries have similar calls, for instance the template secure_vector in Botan.

Another approach is indeed to use the lower-level POSIX call mlock() on the whole buffer. The burden of zeroizing the buffer is with you though. You must manually call memset()) when the buffer is not used anymore or when your program terminates.

CryptProtectMemory() seems to do something slightly different than any of the two approaches above: it creates a small, random session key and uses it to encrypt the buffer. The benefit is that you only need to lock and finally zeroize only the very small page where the key resides, and not the whole buffer. That may make a difference if the buffer is very big. However, we will not be able to operate or process data in the buffer. There is also a small time window when secret data is swappable.

I'm wondering if encrypting (part of the) memory of your process will end up in a chicken-egg problem. I mean, the password to encrypt/decrypt the block(s) of memory of interest should be somewhere in memory, inside the address space of your process. If some malicious code can access/inspect your process address space at runtime you are not going to solve the problem (but you are making the observer life harder ;))

If you are working in user space you can build some wrapper to encrypt/decrypt your variables/memory using any available crypto library (ie. OpenSSL libcrypto), I guess. You could create some sort of "secure variable" object, but be aware that some crypto algo requires padding (size of base types might need to be modified accordingly)In kernel space you can use the LKCF (Linux Kernel Crypto API)