Hook into linux authentication to run script or program when certain conditions are met

In general, the method to use to hook into Linux authentication is via PAM. Either writing your own PAM module, or by finding one that can be coerced into doing what you want.

The easiest option I can see is pam_script.

Install, then put auth optional pam_script.so in the appropriate file(s) in /etc/pam.d and write a pam_script_auth script that looks at $PAM_USER and $PAM_AUTHTOK.

Note that the script could be run as root or as the user, so storage of the password failure data needs to be done with careful attention to permissions.

The simple version without the multiple failures version is somewhat like:

if [ $PAM_USER = "jv" ] && [ $PAM_AUTHTOK = "ThePoliceHaveMe" ]; then  shredcommandfi

Good answers have already been posted explaining how to do what you want to do using the Pluggable Authentication Modules so I won't repeat them.

Three things to keep in mind:

First, when you automatically shred your encryption keys after a certain number of failed logins then you have a nasty denial of service vulnerability, where anyone can destroy all of your data by just repeatedly logging in incorrectly.

Second, you probably think that it would work when "they" get your machine but it wouldn't, because while trying to break your encryption or guess your password no one would use your system to do it. The first thing anyone would do is to copy your raw partitions and play with your data in a safe environment where they can be sure that the data they are trying to read won't get destroyed in the process.

Third, as for shredding the whole data to US DOD standards in a few milliseconds, remember to also shred the entire swap partition, or to not use one in the first place. Also, while it may seem not needed, remember to shred the contents of RAM as well, because the contents of RAM can sometimes be recovered even after power loss.

You'll need to develop a PAM module and configure your system to use this for password validation.

If this is a bit daunting, you could try PAM-script which claims the ability to run scripts as part of the authentication process. I've not tried this myself.