Can you ignore all, but source code files?
For example:
*!*.c!Makefile
I would explicitly put them in the project .gitignore. It's not elegant, but I imagine your project doesn't have that many of them.
Most developers usually have a build directory in their project where the actual build process in run. So, all executables, .o, .so, .a, etc. are there and this build directory is added into the .gitignore.
build
.o
.so
.a
.gitignore