Automated Code Signing - Protecting the private key

First of all I will try to answer your questions separately:

1. Protects the private key from being copied:
   Only the cryptographic hardware (smartcard or hsm) can really protect the key from being copied. Windows certificate store (even with non-exportable option as you correctly noted) or PKCS#12 (PFX) file provide only the false sense of protection.
2. Prevents the private key from being used by unauthorized users to sign code:
   IMO this requires user interaction such as entering password or PIN. If you pass the password as a parameter there is always a possibility that other processes would be able to gain it i.e. from process info, logs etc.
3. Given the private key password is provided by an authorized user, makes the private key for signing by build service:
   Cryptographic hardware (smartcard or hsm) accessible via CSP (windows certificate store) with interactively entered PIN should work with signtool without any problems.

I agree that the requirement of user interaction may not be exactly convenient for an automatic build service but you will probably have to choose between secure solution with user interaction and less secure solution without user interaction.

Convenient but less secure solution no. 1: It seems to me that you have already found an acceptable solution because you did not provide any of its disadvantages.

If I abandon my original idea of storing the certificate in the machine store, there is the option of placing the certificate pfx file in an ACL secured folder on the build machine that only the build process and signing users have permissions on. Doing that would allow me to create a build job to use the contained private key while not exposing the file to others that have access to the machine. To use the private key, the build parameters would need to collect the private key password.

However please note that PFX file can be copied undetectably not only from the live system but also from the backups.

Convenient but less secure solution no. 2: Store private key on a smartcard that does not require PIN to be entered and allow system access only to the trusted users. This would ensure that your private key cannot be copied while it would remain easily accessible for signtool. However it would probably require you to have two separate build servers - one without the smartcard accessible to all users and one with the smartcard accessible only to the trusted users.

Inconvenient secure solution: Store private key on a smartcard that requires PIN to be entered and require user interaction (entering of the PIN) during the build process.

You could also consider signing development builds with self-signed codesigning certificate in an automatic mode and signing public release builds with trusted codesigning certificate in manual mode.

The only option left that I see is to use HSM.

You could generate private key protected by operator card/cardset (for example Thales has this sort of HSMs). Operator cardset can be set with a quorum that specifies how many operators have to insert the card before the private key can be usable by application that requested use of this private key. Thales also has a CSP that supports this feature.

The problem might be that the CSP will invoke window to show user that a card has to be inserted (and optionaly password to that card entered). This problem might be solved when you run you build server under some user account with desktop that can be logged into. When you log in as this user and start the build server it has to request the use of private key (for example by signing some dummy file or something). Windows will pop up and operators (as many as required by the quorum) will one by one insert their cards and optionally enter their passwords. After all required cards have been inserted the key will be usable by your build server. Any other application (possibly started by other user) will go through same procedure to use that key. If your build server crashes (do build servers crash? ) you will go through the same procedure.

HSM have also tamper protection so the private key is IMHO safe in there. Sorry for talking about Thales HSMs only but I personally do not have experience with any other HSM.

It seems that relying on a password-protected PFX file, rather than the certificate store, is secure. Even without ACL protection on the hard drive folder where the PFX resides, no-one who takes the PFX file could use it to sign anything, unless they also had the password.

Then, set up a parameterized Jenkins "signing" job on that computer that fully encapsulates the PFX-password so that no-one can see it. (I am contemplating a script of some sort, Powershell maybe, converted to a black-box EXE.) Authorized Jenkins jobs could chain to the signing job to accomplish the signature. The EXE would log every time it was used, as an auditing feature. Just need to figure out how to prevent unauthorized use of the signing job...there must be a way to do that?