Operand type mismatch when using "jmp *%esp"

Question

I have this snippet in my code

void jmp_esp()
{
    __asm__("jmp *%esp");
}

when compiling with gcc

gcc aslr.c -o aslr -ggdb -fno-stack-protector -z execstack

i get this error.

aslr.c: Assembler messages:
aslr.c:6: Error: operand type mismatch for `jmp'

Why this line is failing to compile although the assembly instruction is valid ?

I've read about DEP (Data Execution Prevention). could it be that this feature is creating this compilation error ? if so, how to disable it ?

Answer 1

The instruction jmp *%esp is available only in 16 and 32 bit modes. In 64 bit mode, jmp r/m32 cannot be encoded. Depending on what your intent is, there are two ways to fix your code:

• if your intent is to write a 32 bit x86 program, compile and link with -m32 to make the compiler emit 32 bit code.
• if your intent is to write a 64 bit x86 program, change the instruction to jmp *%rsp to jump to the address contained in the rsp register instead.

Note that this is independent of DEP. DEP prevents the execution of memory regions not specifically marked as executable. This happens at runtime, not at compile time.