Chapter 4

Memory Management

Stack, Heap, and how programs store data

πŸ—ΊοΈ Process Memory Layout

When a program runs, the operating system gives it its own virtual memory space. This space is divided into regions with different purposes:

Stack ↓ High addresses ↓ Grows downward Free space ↑ Grows upward Heap ↑ .bss (uninitialized) .data (data) .text (code) Low addresses

Main Regions

Region Content Permissions
.text Program code (Assembly instructions) Read + Execute
.data Initialized global variables Read + Write
.bss Uninitialized global variables Read + Write
Heap Dynamic allocations (malloc, new) Read + Write
Stack Local variables, return addresses Read + Write

πŸ“š The Stack

πŸ“– Definition: Stack

A LIFO (Last In, First Out) data structure - the last item that enters is the first to leave.
Used to store local variables, function parameters, and return addresses.

❓ What "enters" the Stack?

What actually "enters" the Stack? Data - values, numbers, addresses:

What enters Example Explanation
Register value PUSH EAX The value inside EAX enters the Stack
Constant number PUSH 5 The number 5 enters the Stack
Return address CALL func Address of next instruction enters

In short: "enters" = the value PUSH stores on Stack. "exits" = the value POP retrieves.

🎯 Two Important Concepts - Don't Confuse Them!

Concept Meaning
LIFO Order of insertion and removal: last in = first out
Grows Downward Direction of addresses: when adding data, the address number decreases
πŸ’‘ Analogy: Stack of Plates

Imagine a stack of plates:

    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚ Plate 3 β”‚  ← Last in, First out!
    β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
    β”‚ Plate 2 β”‚
    β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
    β”‚ Plate 1 β”‚  ← First in, Last out
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                

The last plate you placed is the first one you'll remove. That's LIFO.

πŸ“ What "Grows Downward" Means

In memory, high addresses are "at the top" and low addresses are "at the bottom". When you PUSH, ESP's address decreases (smaller number):

πŸ’‘ Example: PUSH Operation
High addresses (top)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” 0xFFFF0010
β”‚       empty         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ 0xFFFF000C
β”‚       empty         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ 0xFFFF0008  ← ESP was here BEFORE push
β”‚     OLD DATA        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ 0xFFFF0004  ← ESP is here AFTER push
β”‚     NEW VALUE       β”‚  (the pushed value)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Low addresses (bottom)

PUSH = Store value + ESP goes DOWN (address decreases)
POP  = Retrieve value + ESP goes UP (address increases)
                
⚠️ Why is it still LIFO?

ESP always points to the LAST item that was pushed.
When you POP, you get that last item first.
The direction of addresses doesn't change the order of operations!

πŸ“– Definition: ESP / RSP

Stack Pointer - a register that always points to the top of the Stack (the last item that entered).

πŸ“– Definition: EBP / RBP

Base Pointer - a register that points to the base of the current Stack Frame. Used to access parameters and local variables.

Stack Operations

Assembly:
PUSH EAX    ; Store EAX on Stack, ESP decreases by 4
POP EBX     ; Load value from Stack into EBX, ESP increases by 4
Before PUSH 0xFFFF0010 0xFFFF000C 0xFFFF0008 ← ESP β†’ PUSH EAX After PUSH 0xFFFF0010 0xFFFF000C 0xFFFF0008 0xFFFF0004 ← ESP (EAX value here)

πŸ”² Stack Frame

πŸ“– Definition: Stack Frame

A region on the Stack that belongs to a specific function. Contains the local variables, parameters, and return address of that function.

When a function is called, a new Stack Frame is created:

Prologue (Function Start):
push ebp        ; Save old EBP on Stack
mov ebp, esp    ; EBP now points to new Frame base  
sub esp, 0x10   ; Allocate 16 bytes for local variables
Epilogue (Function End):
mov esp, ebp    ; Restore ESP to original state
pop ebp         ; Restore old EBP
ret             ; Return to caller function
Stack Frame High addresses Parameter 2 [EBP+0xC] Parameter 1 [EBP+0x8] Return Address [EBP+0x4] Saved EBP ← EBP Local Variable 1 [EBP-0x4] Local Variable 2 ← ESP Low addresses

πŸ”‘ Important: Variable Names Don't Exist in Assembly!

In C code, there are variable names like a and b. But in Assembly, there are only memory addresses - the names disappear after compilation!

πŸ’‘ Example: C to Assembly
C code:                        Assembly:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ int a = 5;      β”‚     β†’      β”‚ mov dword [ebp-4], 5    β”‚
β”‚ int b = 10;     β”‚     β†’      β”‚ mov dword [ebp-8], 10   β”‚
β”‚ a = a + b;      β”‚     β†’      β”‚ mov eax, [ebp-4]        β”‚
β”‚                 β”‚            β”‚ add eax, [ebp-8]        β”‚
β”‚                 β”‚            β”‚ mov [ebp-4], eax        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜            β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  Named variables                Just memory addresses!
                
C Variable Assembly Location Explanation
a [EBP-4] First local variable (4 bytes from EBP)
b [EBP-8] Second local variable (8 bytes from EBP)
param1 [EBP+8] First parameter (above EBP)
⚠️ This is key for RE!

When you see mov eax, [ebp-4], think:
"Loading some local variable into EAX"

Your job in RE is to figure out what that variable represents based on how it's used!

πŸ”οΈ The Heap

πŸ“– Definition: Heap

A memory region for dynamic allocations - when you don't know ahead of time how much memory you need.
In C: malloc(), free()
In C++: new, delete

Stack Heap
Automatic allocation Manual allocation
Automatically freed at function end Must be manually freed!
Fast Slower
Limited size Larger size available
Grows downward (addresses decrease) Grows upward (addresses increase)

🌐 Virtual Memory

πŸ“– Definition: Virtual Memory

An abstract numbering system that the OS creates for each process. Each program "thinks" it has a huge memory space to itself, but the OS maps virtual addresses to real locations.

πŸ”‘ The Difference Between Virtual and Physical

Term What it is
RAM Physical Memory - the actual hardware (memory chips in the computer)
Virtual Memory Abstraction - a numbering system the OS manages (doesn't "sit" anywhere)
Virtual Address The address the program sees (e.g., 0x00401000)
Physical Address The actual address in RAM

πŸ“ Where Does Data Actually Sit?

Process A 0x00401000 Process B 0x00401000 Same virtual address! Page Table (OS manages) Virtual→Physical Physical RAM (actual hardware) 0x12340000 0x56780000 Different physical addresses! Virtual Addresses Translation Physical Storage
πŸ’‘ Where does data actually sit?
Program uses Virtual Address (e.g., 0x00401000)
                    ↓
         CPU + Page Table translate
                    ↓
        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
        β”‚  Option 1: RAM            β”‚  ← Fast! (physical memory chips)
        β”‚  Option 2: Disk           β”‚  ← Slow! (pagefile.sys / swap)
        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                

Virtual Memory doesn't "sit" anywhere - it's just a numbering system.
The actual data sits in RAM or on disk, and the OS decides where.

Advantages:

πŸ’‘ For Reverse Engineering

In RE, you always see Virtual Addresses.
You don't need to know the physical addresses - the OS handles the translation automatically.

πŸ“‹ Chapter Summary