How do I check if an object file is in COFF or ELF format from within C?
Check the magic number. The ELF magic number is 0x7f454C46 (0x7f + "ELF") and COFF's is 0x14c. Take care about this anyway because there are different magic numbers for COFF.
Watch out about endianness when reading these values.