I haven't tried this, but one way to do this would be to enable Project-based Matrix Authorization Strategy in the Authorization Section of Manage Jenkins and uncheck Build option for all users for the specific job that you do not want to be built manually.