The Key Transport Algorithm is a bit tricky to handle, and it may not serve its purpose (I see you noted that you'd like CAPI to support RSAencryption; trust me, I would too). It looks like you've alaready detected the bulk of your problem - The generated message appears is valid, but your method makes it necessary to use CryptEncryptMessage, which won't work well/at all in the long run.

Step 1 - Examine the Code

CRYPT_ENCRYPT_MESSAGE_PARA EncryptMessageParams;EncryptMessageParams.cbSize = sizeof(CMSG_ENVELOPED_ENCODE_INFO);EncryptMessageParams.dwMsgEncodingType = PKCS_7_ASN_ENCODING;EncryptMessageParams.ContentEncryptionAlgorithm.pszObjId = szOID_NIST_AES256_CBC;EncryptMessageParams.ContentEncryptionAlgorithm.Parameters.cbData = 0;EncryptMessageParams.ContentEncryptionAlgorithm.Parameters.pbData = 0;EncryptMessageParams.hCryptProv = NULL;EncryptMessageParams.pvEncryptionAuxInfo = NULL;EncryptMessageParams.dwFlags = 0;EncryptMessageParams.dwInnerContentType = 0;BYTE pbEncryptedBlob[640000];DWORD pcbEncryptedBlob = 640000;BOOL retval =  CryptEncryptMessage(&EncryptMessageParams, cRecipientCert, pRecipCertContextArray, pbMsgText, dwMsgTextSize, pbEncryptedBlob, &pcbEncryptedBlob);

Pretty basic, isn't it? Although efficient, it's not really getting the problem done. If you look at this:

EncryptMessageParams.dwFlags = 0;EncryptMessageParams.dwInnerContentType = 0;

you will see that it is pre-defined, but used only in the definition of retval. However, I could definitely see this as a micro-optimization, and not really useful if we're going to re-write the code. However, I've outlined the basic steps blow to integrate this without a total re-do of the code (so you can keep on using the same parameters):

Step 2 - Editing the Parameters

As @owlstead mentioned in his comments, the Crypto API is not very user-friendly. However, you've done a great job with limited resources. What you'll wanna add is a Cryptographic Enumeration Provider to help narrow down the keys. Make sure you have either Microsoft Base Cryptographic Provider version 1.0 or Microsoft Enhanced Cryptographic Provider version 1.0 to use these efficiently. Otherwise, you'll need to add in the function like so:

DWORD cbName;DWORD dwType;DWORD dwIndex;CHAR *pszName = NULL;(regular crypt calls here)

This is mainly used to prevent the NTE_BAD_FLAGS error, although technically you could avoid this with a more low-level declaration. If you wanted, you could also create a whole new hash (although this is only necessary if the above implementation won't scale to the necessary factor of time/speed):

DWORD dwBufferLen = strlen((char *)pbBuffer)+1*(0+5);HCRYPTHASH hHash;HCRYPTKEY hKey;HCRYPTKEY hPubKey;BYTE *pbKeyBlob;BYTE *pbSignature;DWORD dwSigLen;DWORD dwBlobLen;(use hash as normal w/ crypt calls and the pbKeyBlobs/Signatures)

Make sure to vaildate this snippet before moving on. You can do so easily like so:

if(CryptAcquireContext(&hProv, NULL, NULL, PROV_RSA_FULL, 0)) {     printf("CSP context acquired.\n");}

If you're documenting or releasing, might want to add a void MyHandleError(char *s) to catch the error so someone who edits but fails can catch it quickly.

By the way, the first time you run it you'll have to create a new set because there's no default. A nice one-liner that can be popped into an if is below:

CryptAcquireContext(&hCryptProv, NULL, NULL, PROV_RSA_FULL, CRYPT_NEWKEYSET)

Remember that syncing server resources will not be as efficient as doing the re-work I suggested in the first step. This is what I will be explaining below:

Step 3 - Recode and Relaunch

As a programmer, re-coding might seem like a waste of time, but it can definitely help you out in the long run. Remember that you'll still have to code in the custom params when encoding/syncing; I'm not going to hand-feed you all the code like a baby. It should be well sufficient to show you the basic outlines.

I'm definitely assuming that you're trying to handle to the current user's key container within a particular CSP; otherwise, I don't really see the use of this. If not, you can do some basic edits to suit your needs.

Remember, we're going to bypass CryptEncryptMessage by using CryptReleaseContext, which directly releases the handle acquired by the CryptAcquireContext function. Microsoft's standard on the CAC is below:

BOOL WINAPI CryptAcquireContext(  _Out_  HCRYPTPROV *phProv,  _In_   LPCTSTR pszContainer,  _In_   LPCTSTR pszProvider,  _In_   DWORD dwProvType,  _In_   DWORD dwFlags);

Note that Microsoft's scolding you if you're using a user interface:

If the CSP must display the UI to operate, the call fails and the NTE_SILENT_CONTEXT error code is set as the last error. In addition, if calls are made to CryptGenKey with the CRYPT_USER_PROTECTED flag with a context that has been acquired with the CRYPT_SILENT flag, the calls fail and the CSP sets NTE_SILENT_CONTEXT.

This is mainly server code, and the ERROR_BUSY will definitely be displayed to new users when there are multiple connections, especially those with a high latency. Above 300ms will just cause a NTE_BAD_KEYSET_PARAM or similar to be called, due to the timeout without even a proper error being received. (Transmission problems, anyone with me?)

Unless you're concerned about multiple DLL's (which this doesn't support due to NTE_PROVIDER_DLL_FAIL errors), the basic set up to grab crypt services clientside would be as below (copied directly from Microsoft's examples):

if (GetLastError() == NTE_BAD_KEYSET) {   if(CryptAcquireContext(      &hCryptProv,       UserName,       NULL,       PROV_RSA_FULL,       CRYPT_NEWKEYSET))     {      printf("A new key container has been created.\n");    }    else    {      printf("Could not create a new key container.\n");      exit(1);    }  }  else  {      printf("A cryptographic service handle could not be "          "acquired.\n");      exit(1);   }

However simple this may seem, you definitely don't want to get stuck passing this on to the key exchange algorithm (or whatever else you have handling this). Unless you're using symmetric session keys (Diffie-Hellman/KEA), the exchange keypair can be used to encrypt session keys so that they can be safely stored and exchanged with other users.

Someone named John Howard has written a nice Hyper-V Remote Management Configuration Utility (HVRemote) which is a large compilation of the techniques discussed here. In addition to using the basic crypts and keypairs, they can be used to permit ANONYMOUS LOGON remote DCOM access (cscript hvremote.wsf, to be specific). You can see many of the functions and techniques in his latest crypts (you'll have to narrow the query) on his blog:

http://blogs.technet.com/b/jhoward/

If you need any more help with the basics, just leave a comment or request a private chat.

Conclusion

Although it's pretty simple once you realize the basic server-side methods for hashing and how the client grabs the "crypts", you'll be questioning why you even tried the encryption during transmits. However, without the crypting clientside, encrypts would definitely be the only secure way to transmit what was already hashed.

Although you might argue that the packets could be decrypted and hashed off the salts, consider that both in-outgoing would have to be processed and stored in the correct timing and order necessary to re-hash clientside.