Why do you need to recompile C/C++ for each OS? [duplicate]

Don't we target the CPU architecture/instruction set when compiling a C/C++ program?

No, you don't.

I mean yes, you are compiling for a CPU instruction set. But that's not all compilation is.

Consider the simplest "Hello, world!" program. All it does is call printf, right? But there's no "printf" instruction set opcode. So... what exactly happens?

Well, that's part of the C standard library. Its printf function does some processing on the string and parameters, then... displays it. How does that happen? Well, it sends the string to standard out. OK... who controls that?

The operating system. And there's no "standard out" opcode either, so sending a string to standard out involves some form of OS call.

And OS calls are not standardized across operating systems. Pretty much every standard library function that does something you couldn't build on your own in C or C++ is going to talk to the OS to do at least some of its work.

malloc? Memory doesn't belong to you; it belongs to the OS, and you maybe are allowed to have some. scanf? Standard input doesn't belong to you; it belongs to the OS, and you can maybe read from it. And so on.

Your standard library is built from calls to OS routines. And those OS routines are non-portable, so your standard library implementation is non-portable. So your executable has these non-portable calls in it.

And on top of all of that, different OSs have different ideas of what an "executable" even looks like. An executable isn't just a bunch of opcodes, after all; where do you think all of those constant and pre-initialized static variables get stored? Different OSs have different ways of starting up an executable, and the structure of the executable is a part of that.

How do you allocate memory? There's no CPU instruction for allocating dynamic memory, you have to ask the OS for the memory. But what are the parameters? How do you invoke the OS?

How do you print output? How do you open a file? How do you set a timer? How do you display a UI? All of these things require requesting services from the OS, and different OSes provide different services with different calls necessary to request them.

If I compile my C/C++ program targeting the x86 architecture, it would seem that the same program should run on any computer with the same architecture.

It is very true, but there're a few nuances.

Let's consider several cases of programs that are, from C-language point of view, OS-independent.

1. Suppose all that your program does, from the very beginning, is stress-testing the CPU by doing lots of computations without any I/O.

The machine code could be exactly the same for all the OSes (provided they all run in the same CPU mode, e.g. x86 32-bit Protected Mode). You could even write it in assembly language directly, it wouldn't need to be adapted for each OS.

But each OS wants different headers for the binaries containing this code. E.g. Windows wants PE format, Linux needs ELF, macOS uses Mach-O format. For your simple program you could prepare the machine code as a separate file, and a bunch of headers for each OS's executable format. Then all you need to "recompile" would actually be to concatenate the header and the machine code and, possibly, add alignment "footer".

So, suppose you compiled your C code into machine code, which looks as follows:

offset:  instruction  disassembly    00:  f7 e0        mul eax    02:  eb fc        jmp short 00

This is the simple stress-testing code, repeatedly doing multiplications of eax register by itself.

Now you want to make it run on 32-bit Linux and 32-bit Windows. You'll need two headers, here're examples (hex dump):

• For Linux:

000000 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00  >.ELF............<000010 02 00 03 00 01 00 00 00 54 80 04 08 34 00 00 00  >........T...4...<000020 00 00 00 00 00 00 00 00 34 00 20 00 01 00 28 00  >........4. ...(.<000030 00 00 00 00 01 00 00 00 54 00 00 00 54 80 04 08  >........T...T...<000040 54 80 04 08 04 00 00 00 04 00 00 00 05 00 00 00  >T...............<000050 00 10 00 00                                      >....<

• For Windows (* simply repeats previous line until the address below * is reached):

000000 4d 5a 80 00 01 00 00 00 04 00 10 00 ff ff 00 00  >MZ..............<000010 40 01 00 00 00 00 00 00 40 00 00 00 00 00 00 00  >@.......@.......<000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<000030 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 00  >................<000040 0e 1f ba 0e 00 b4 09 cd 21 b8 01 4c cd 21 54 68  >........!..L.!Th<000050 69 73 20 70 72 6f 67 72 61 6d 20 63 61 6e 6e 6f  >is program canno<000060 74 20 62 65 20 72 75 6e 20 69 6e 20 44 4f 53 20  >t be run in DOS <000070 6d 6f 64 65 2e 0d 0a 24 00 00 00 00 00 00 00 00  >mode...$........<000080 50 45 00 00 4c 01 01 00 ee 71 b4 5e 00 00 00 00  >PE..L....q.^....<000090 00 00 00 00 e0 00 0f 01 0b 01 01 47 00 02 00 00  >...........G....<0000a0 00 02 00 00 00 00 00 00 00 10 00 00 00 10 00 00  >................<0000b0 00 10 00 00 00 00 40 00 00 10 00 00 00 02 00 00  >......@.........<0000c0 01 00 00 00 00 00 00 00 03 00 0a 00 00 00 00 00  >................<0000d0 00 20 00 00 00 02 00 00 40 fb 00 00 03 00 00 00  >. ......@.......<0000e0 00 10 00 00 00 10 00 00 00 00 01 00 00 00 00 00  >................<0000f0 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00  >................<000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<*000170 00 00 00 00 00 00 00 00 2e 66 6c 61 74 00 00 00  >.........flat...<000180 04 00 00 00 00 10 00 00 00 02 00 00 00 02 00 00  >................<000190 00 00 00 00 00 00 00 00 00 00 00 00 60 00 00 e0  >............`...<0001a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<*000200

Now if you append your machine code to these headers and, for Windows, also append a bunch of null bytes to make file size 1024 bytes, you'll get valid executables that will run on the corresponding OS.

2. Suppose now that your program wants to terminate after doing some amount of calculations.

   Now it has two options:

   1. Crash—e.g. by execution of an invalid instruction (on x86 it could be UD2). This is easy, OS-independent, but not elegant.

   2. Ask the OS to correctly terminate the process. At this point we need an OS-dependent mechanism to do this.

On x86 Linux it would be

xor ebx, ebx ; zero exit codemov eax, 1   ; __NR_exitint 0x80     ; do the system call (the easiest way)

On x86 Windows 7 it would be

    ; First call terminates all threads except caller thread, see for details:    ; http://www.rohitab.com/discuss/topic/41523-windows-process-termination/    mov eax, 0x172  ; NtTerminateProcess_Wind7    mov edx, terminateParams    int 0x2e        ; do the system call    ; Second call terminates current process    mov eax, 0x172    mov edx, terminateParams    int 0x2eterminateParams:    dd 0, 0 ; processHandle, exitStatus

Note that on other Windows version you'd need another system call number. The proper way to call NtTerminateProcess is via yet another nuance of OS-dependence: shared libraries.

3. Now your program wants to load some shared library to avoid reinventing some wheels.

OK, we've seen that our executable file formats are different. Suppose that we've taken this into account and prepared the import sections for the file targeting each of the target OS. There's still a problem: the way to call a function—the so called calling convention—for each OS is different.

E.g. suppose the C-language function your program needs to call returns a structure containing two int values. On Linux the caller would have to allocate some space (e.g. on the stack) and pass the pointer to it as the first parameter to the function being called, like so:

sub esp, 12 ; 4*2+alignment: stack must be 16-byte alignedpush esp    ;                right before the call instructioncall myFunc

On Windows you'd get the first int value of the structure in EAX, and the second in EDX, without passing any additional parameters to the function.

There are other nuances like different name mangling schemes (though these can differ between compilers even on the same OS), different data types (e.g. long double on MSVC vs long double on GCC) etc., but the above mentioned ones are the most important differences between the OSes from the point of view of the compiler and linker.