notes/docs/lectures/malware/03_intro_x86.md

Crash Course in x86 Assembler

• Malware authors creates programs at the high-level language and use a compiler to generate machine code to by run by the CPU
• Malware analysts operate at the low-level language. Using disassembler to generate assembly code from the machine code to try and understand how the malware works

x86 Architecture

x86 architecture follows the Von Neuman architecture and has three hardware components

• CPU executes code
• Main memory (RAM) stores all data and code instructions
• An input/output system (I/O) interfaces with devices such as hard drives, keyboards and monitors

Main Memory

The main memory for a single program can be divided into the following four major sections.

Data - Contains values that are put in place when a program is initially loaded

Code - Includes the instructions fetched by the CPU to execute the programs tasks. The code controls what the program does

Heap - The heap is used for dynamic memory during program execution, to create (or allocate) new values and eliminate (free) values that the program no longer needs. The heap’s size changes frequently while the program runs

Stack - The stack is used for local variables and parameters for functions, and to help control program flow

Instructions

Each instruction is comprised of an opcode and zero or more operands.

opcode - instruction

operand - argument or data

endianess

• Whether the most significant bit is at the start or the end of a binary stream.
  • Big-endian is where the most significant bit is first
  • Little-endian is where the least significant bit is first

Disassemblers translate opcodes into human-readable instructions e.g.

B9 42 00 00 00
mov ecx, 0x42

Operands

Three types of operands are used in x86

1. Immediate operands are fixed values
2. Register operands refer to registers
3. Memory address operands refer to a memory address that contains the value of interest, typically denoted by [reg]

Registers

A register is a small amount of data storage available to the CPU, that’s really quick. There are four categories:

1. General registers are used by the CPU during execution
2. Segment registers are used to track sections of memory
3. Status flags are used to make decisions
4. Instruction pointers are used to keep track of the next instruction to execute

All general registers are 32-bits but can be referenced as either 32 or 16 bits in assembly code (for backwards compatibility reasons)

EDX - full 32-bits

DX - lower 16 bits

Registers EAX, EBX, ECX, EDX can be referenced as 8 bit registers

Some x86 instructions use specific registers by definition.

• Multiplication and division instructions always use EAX and EDX
• EAX generally contains the return value for function calls
• However these are just conventions and can change

Flags

The EFLAGS register is a status register 32-bits big, this means it can store 32 flags. During execution, each flag is either set to 1 if true

• ZF - The zero flag is set if the result of the operation was equal to zero
• CF - The carry flag is set when the result of an operation is too large or too small for the destination operand.

EIP - Instruction Pointer

EIP contains the memory location of the next instruction to be executed

NOP

nop - no operation - does nothing

When issued, execution simply preceeds to the next instruction

The Stack

Memory for functions, local variables and flow control is stored in the stack (LIFO)

ESP - Stack pointer, points to the memory address at the top of the stack

EBP - Stack base pointer, stays consistent within a function. The program can use it as a placeholder to keep track of the location of local variables

Function Calls

Main code calls and temporarily transfers execution to functions before returning to the main code.

Many functions contain a prologue and an epilogue

• The prologue is a few lines of code at the start of the function which prepares the stack and registers for use within the function
• The epilogue is at the end of the function and restores the stack and registers to their state before the function was called

When a function is called:

1. Arguments are placed on the stack using push instructions
2. A function called using memory_location which changes EIP to the address of the first instruction in the function and returns EIP to main code once the function is finished
3. The function prologue pushes local variables, parameters and EBP onto the stack
4. The function executes
5. The function epilogue restores the stack, ESP is adjusted to free local variables, and EBP is restored so that the calling function can address its variables.
   • The leave instruction sets ESP equal to EBP and pops EBP off the stack
6. The function returns by calling ret, this pops the return address off the stack into EIP
7. The stack is adjusted to remove sent arguments

Passing Arguments

c functions and windows api calls, functions are called differently.

There are two things to think about

1. Who’s responsible for cleaning up the stack after the function has run, the caller or the callee
2. Which order do you put the arguments on the stack

The three most common calling conventions are cdecl, stdcall and fastcall.

Example pseudo code

int test(int x, int y, int z);
int a, b, c, ret;

ret = test (a, b, c);

cdecl

• In cdecl parameters are pushed onto the stack from right to left

• The caller cleans up the stack when the function is complete

• Return value stored in EAX

• push c
  push b
  push a
  call test
  add esp, 12
  mov ret, eax
  

• Note line 5 is the caller cleaning up the stack

• Function names have an underscore prefix to denote cdecl used.

stdcall

• stdcall is similar to cdecl however the callee is required to clean the stack
• Therefore line 5 in the previous example would not be needed in stdcall
• stdcall is used for Windows API functions
• Functions have underscore prefix, name followed by @ and length of arguments

fastcall

• In fastcall the first few arguments (typically first two) are passed in registers EDX and ECX
• Additional arguments are loaded right to left
• Calling function is responsible for cleaning the stack
• This is quicker as less data needs to be pushed to and retrived from the stack

• Functions have underscore prefix, name followed by @ and length of arguments

When debugging windows functions, you can look at EBP to retrace the route the program took through the code

Conditionals