Assembly 2: Calling convention and control flow

Overview

We discuss calling conventions and control flow in machine code.

Calling convention

• Some aspects of machine code are fixed by the processor manufacturer
• Intel decided 0xc3 is the representation of ret
• Some aspects of machine code are set by agreement among compiler and operating system developers
• Intel did not decide which register holds return values
• We call this agreement the calling convention since it governs function calls
• Think Geneva Convention, not Comic Convention
• Different conventions can exist for the same processor (e.g., Unix vs. Windows)
• Only codes with the same conventions can safely interact

Elements of a calling convention

• Function arguments
• Function return values
• Local variable storage
• Stack alignment
• Memory and processor state when the program begins

Arguments

• Argument registers are %rdi, %rsi, %rdx, %rcx, %r8, %r9, in that order
• Large objects are passed in up to 2 registers if they fit, stack otherwise

Return values

• Return register is %rax
• Large objects are returned in %rax+%rdx if they fit, otherwise first argument points to space for return value
• The name of a C++ function encodes the types of its arguments
• This makes C++ compilations safer and supports overloading (functions with different behavior based on argument types)
• Example: f(int)_Z1fi
• _Z: This is a mangled name
• 1: Function name is 1 character long
• f: Actual function name
• i: First argument is int
• To demangle, try c++filt MANGLEDNAME

Function entry and exit sequence

• Call sequence: Steps required before calling a function
• Set up arguments
• Entry sequence: Steps required to enter a function
• Prepare environment for callee function
• Exit sequence: Steps required to exit a function
• Set up return value
• Restore environment for caller function
• Return sequence: Steps required after function returns
• Clean up any argument space

subq $8, %rsp movq %NEXT_rip, (%rsp) movq FUNCTION, %rip  Action of ret instruction addq$8, %rsp
movq -8(%rsp), %rip


Local variables

• Local variables are stored in the stack
• The function entry sequence reserves enough space for locals
• Small local variables may be stored in the red zone
• 128 bytes below %rsp
• Does not need explicit reservation
• Generally seen in leaf functions (functions that do not call other functions)