Git push failed, looking for partial file history deletion
Git allow you to remove part of your history by filtering it.
In your case, the first example should do the trick: git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD.
Nevertheless, you should read very carefully about this command and what needs to be done on your repository and your remote for this to be effective (notably, you have to use git gc in your local repository).
You can read more on this in the github help section.
As a side note, git is not really good with managing big files (look into git annex for a potential solution).