System Calls

Module 3.9 — User-space to kernel-space transition, syscall table lookup

Speed:

Step 1 / 9

USER SPACE

User Code

Application calls open("/etc/hosts", O_RDONLY)

User Space (Ring 3)

Application

open("/etc/hosts", O_RDONLY, 0644)

libc Wrapper

mov rax, 2
syscall

Kernel Space (Ring 0)

sys_call_table[]

[0] read

[1] write

[2] open

[3] close

[57] fork

[59] execve

[60] exit

VFS Layer

Inode Lookup

File Descriptors

Execution Timeline

open()

Open a file descriptor. Traverses VFS, finds inode, allocates fd.

Syscall #2

pathname"/etc/hosts"

flagsO_RDONLY

mode0644

CPU Registers

RAX—

RDI"/etc/hosts"

RSIO_RDONLY

RDX0644

CPL3 (Ring 3)

System Call Flow

App calls libc wrapper (e.g., open())

Wrapper puts syscall # in RAX, args in RDI/RSI/RDX

SYSCALL instruction: Ring 3 → Ring 0

Kernel looks up handler in sys_call_table

Handler executes, result goes in RAX

SYSRET: Ring 0 → Ring 3, resume user code

x86-64 Syscall Convention

RAX = syscall number

RDI = arg1, RSI = arg2

RDX = arg3, R10 = arg4

R8 = arg5, R9 = arg6

Return value in RAX

Negative = -errno