If I do not pass enough parameters when calling a function in a DLL, what will happen?

32-bit

Default calling convention is __cdecl, which means the caller pushes parameters onto the stack right-to-left then cleans up the stack after the call returns.

So in your case, the caller:

1. Pushes b
2. Pushes a
3. Pushes the return address
4. Calls the function.

At this point the stack looks like this (assume 4 byte pointers for example, and remember the stack pointer travels backwards when you push things):

+-----+ <--- this is where esp is after pushing stuff| ret | [esp]+-----+|  a  | [esp+4]+-----+|  b  | [esp+8]+-----+ <--- this is where esp was before we started| ??? | [esp+12 and beyond]+-----+

Ok, great. Now the problem happens on the callee side. The callee is expecting parameters to be at certain locations on the stack, so:

• a is assumed to be at [esp+4]
• b is assumed to be at [esp+8]
• c is assumed to be at [esp+12]

And this is where the issue is: We have no idea what's at [esp+12]. So the callee will see the correct values of a and b, but will interpret whatever unknown garbage happens to be at [esp+12] as c.

At that point it's pretty much undefined, and depends on what your function actually does with c.

After all this is over and the callee returns, assuming your program didn't crash, the caller will restore esp and the stack pointer will be back where it should be. So from the caller's POV everything is probably fine and the stack pointer ends up back where it's supposed to be, but the callee sees junk for c.

64-bit

The mechanics on 64-bit machines is different but the end result is roughly the same effect. Microsoft uses the following calling convention on 64-bit machines regardless of __cdecl or whatever (any convention you specify is ignored and all are treated identically):

• First four integer or pointer arguments placed in registers rcx, rdx, r8, and r9, in that order, left-to-right.
• First four floating-point arguments placed in registers xmm0, xmm1, xmm2, and xmm3, in that order, left-to-right.
• Anything remaining is pushed to the stack, right-to-left.
• The caller is responsible for restoring esp as well as restoring the values of all volatile registers after the call.

So in your case, the caller:

1. Puts a in rcx.
2. Puts b in rdx.
3. Allocates an extra 32 bytes of "shadow space" on the stack (see that MS article).
4. Pushes the return address.
5. Calls the function.

But the callee is expecting:

• a assumed to be in rcx (check!)
• b assumed to be in rdx (check!)
• c assumed to be in r8 (problem)

And so, as with the 32-bit case, the callee interprets whatever happened to be in r8 as c, and potential hijinks ensue, with the end effect depending on what the callee does with c. When it returns, assuming the program did not crash, the caller restores all volatile registers (rcx and rdx, and also generally includes r8 and friends) and restores esp.