Jenkins user-based job security
Turns out there is a feature already in jenkins for this, no plugins necessary! In the Configure Global Security section in Manage Jenkins, click "Project-based matrix authorization strategy". Then you can configure permissions in the job configure screen for that particular job by clicking "enable project-based security".
Now you can configure your Jenkins so that "Joe can access project A, B, and C but he can't see D".