What causes this program to segmentation fault?

Question

I am attempting to calculate the size of pushed data onto the stack. When attempting this I receive a segmentation fault. My objective is to point ebp into esp before pushing any data onto the stack. After I push data onto the stack I am measuring the size of said data in bytes ebp - esp and storing in ebx and printing to stdout using printf.

For example:

; compile with:
; nasm -f elf test.asm && gcc -m32 -o test test.o
section .text
global  main
extern printf

main:
    ; set the frame pointer to the beginning of the stack before-
    ; data is pushed.
    push  ebp
    mov   ebp, esp

    push  ebx
    push  0x00       ; <- null terminating byte/string truncation
    push  0x64636261 ; <- data

    mov   ebx, ebp
    sub   ebx, esp   ; start of stack - end of stack  = sizeof(data) stored in ebx

    push  ebx
    push  format_str
    call  printf
    add   esp, 8

    pop   ebp
    pop   ebx
    ret

section .data
format_str db "%s", 2, 0

When compiling this code I receive the output:

Segmentation fault (core dumped)

Expected output:

5

Answer 1

For a start, items should be popped in the reverse order to which they were pushed, if you want them back in their original registers. What you have here (with irrelevant lines removed):

push ebp
push ebx
pop ebp
pop ebx

will not restore ebp to its previous value, and this is very likely to cause problems moving up through the stack frames.

---

Additionally, you may be better off following the normal practice of restoring esp from ebp rather than blindly adding eight. That would be something like this at the end:

; add esp, 8 ; not the normal way.
mov   esp, ebp
pop   ebp

Doing it this way removes the need for you to manually calculate how many bytes you need to take off the stack, a calculation that you actually got wrong since you didn't take into account everything you pushed.

---

And, finally, before you do that, you have to make sure the esp is in the right place so that the pop ebp will work. That means popping everything you pushed (other than ebp itself). Since you pushed (after ebp) ebx, 0x00000000, 0c64636261, ebx, and format_str, you should make sure all of those are off the stack before attempting to pop ebp.

Taking all that into account gives you something like:

main:
    push  ebp
    mov   ebp, esp

    push  ebx           ; (+1)
    push  0x00          ; (+2)
    push  0x64636261    ; (+3)

    mov   ebx, ebp
    sub   ebx, esp

    push  ebx           ; (+4)
    push  format_str    ; (+5)
    call  printf
    add   esp, 16       ; (-5, -4, -3, -2)
    pop   ebx           ; (-1)

    mov   esp, ebp
    pop   ebp
    ret

Each of those (+N) comments represent a 32-bit value that has been pushed on the stack, and the (-N) comments indicate which instructions reverse those pushes. The add esp, 16 reverses four of them (at four bytes apiece), and is done that way since we don't care what happens to those items. That leaves the final pop to recover the original ebx (which we do care about).

That final reload of esp is, I think, unnecessary in this case since it's been restored to the correct value by previous steps. Whether you leave it in for prolog/epilog consistency is up to you.