Chapter 8: Calling Conventions

📞 What is a Calling Convention?

📖 Definition: Calling Convention

An agreement that defines:
1. Where parameters are passed (Stack? Registers?)
2. In what order parameters are pushed
3. Who cleans the Stack after the call (caller or callee)
4. Which registers are preserved

In RE, you must know the Calling Convention to understand what values are passed to a function and what it returns.

📚 cdecl - C Declaration

📖 Definition: cdecl

The standard Calling Convention for C.
Parameters: Stack (right to left)
Cleanup: Caller cleans the Stack
Return value: EAX

// C code
int result = add(1, 2, 3);

// Assembly (cdecl)
push 3              ; Parameter 3 (rightmost first)
push 2              ; Parameter 2
push 1              ; Parameter 1
call add            ; Call function
add esp, 12         ; Caller cleans (3 params × 4 bytes)
; EAX = result

📚 stdcall - Standard Call

📖 Definition: stdcall

The Calling Convention for Windows API.
Parameters: Stack (right to left)
Cleanup: Callee cleans the Stack
Return value: EAX

// Windows API call
HANDLE h = CreateFileA("test.txt", ...);

// Assembly (stdcall)
push ...            ; Parameters
push "test.txt"
call CreateFileA    ; Function cleans the Stack
; No ADD ESP after! The function does RET 28

💡 How to identify stdcall?

The function ends with RET n (e.g., RET 0x1C)
instead of a regular RET. The n specifies how many bytes to clean.

📚 fastcall

📖 Definition: fastcall

Passes the first 2 parameters in registers (ECX, EDX).
Remaining parameters: Stack
Cleanup: Callee
Faster because less memory access!

// fastcall function
int __fastcall func(int a, int b, int c);

// Assembly
mov edx, 2          ; Parameter 2 in EDX
mov ecx, 1          ; Parameter 1 in ECX
push 3              ; Parameter 3 on Stack
call func

📚 x64 Calling Convention (Windows)

In 64-bit there's a unified calling convention:

First 4 parameters: RCX, RDX, R8, R9
Remaining parameters: Stack
Shadow Space: Caller must allocate 32 bytes on Stack even if fewer than 4 parameters
Return value: RAX

// x64 Windows
func(1, 2, 3, 4, 5);

// Assembly
sub rsp, 40         ; Shadow space (32) + param 5 (8)
mov [rsp+32], 5     ; Parameter 5 on Stack
mov r9d, 4          ; Parameter 4
mov r8d, 3          ; Parameter 3
mov edx, 2          ; Parameter 2
mov ecx, 1          ; Parameter 1
call func
add rsp, 40

📊 Conventions Comparison

Feature	cdecl	stdcall	fastcall	x64 (Win)
Parameters	Stack	Stack	ECX,EDX + Stack	RCX,RDX,R8,R9 + Stack
Order	RTL	RTL	RTL	LTR for registers
Stack Cleanup	Caller	Callee	Callee	Caller
Return Value	EAX	EAX	EAX	RAX

RTL = Right To Left, LTR = Left To Right

🔍 Identification in RE

💡 How to identify Calling Convention

cdecl:

call func
add esp, XX      ; Caller cleans

stdcall:

call func
; No ADD ESP! The function does:
ret XX           ; Callee cleans

fastcall:

mov ecx, param1
mov edx, param2
call func

📝 Full Example

; Function that adds 2 numbers (cdecl)
; int add(int a, int b)

add:
    push ebp              ; prologue
    mov ebp, esp
    
    mov eax, [ebp+8]      ; EAX = a (first parameter)
    add eax, [ebp+12]     ; EAX = a + b (second parameter)
    
    pop ebp               ; epilogue
    ret                   ; return, EAX = result

; Calling the function
    push 20               ; b = 20
    push 10               ; a = 10
    call add
    add esp, 8            ; clean 2 parameters (cdecl)
    ; EAX = 30

📋 Chapter Summary

Calling Convention - rules for passing parameters and Stack cleanup
cdecl - standard C, Caller cleans
stdcall - Windows API, Callee cleans
fastcall - parameters in registers (ECX, EDX)
x64 - RCX, RDX, R8, R9 + Shadow Space
Return value always in EAX/RAX

← Chapter 7: PE Format Chapter 9: Analysis Tools →