memcmp ignores characters at odd positions

Question

I am calling memcmp from fasm (version 1.71.51) code, and I obtain strange results.

It seems like memcmp only compares characters that are at odd position. Code:

format ELF64

section '.text' executable

    public _start

    extrn memcmp
    extrn exit

_start:
    push    rbp     ; Align the stack to 16 bytes

    mov     rdi, str1
    mov     rsi, str2
    mov     rdx, BUFF_LEN
    call    memcmp

    mov     rdi, rax
    call    exit
    pop     rbp


section '.data' writeable

str1                db '1509487271'
BUFF_LEN = $ - str1
str2                db '1509487273'

I am running Ubuntu:

$ uname -a
Linux freedom 4.13.0-43-generic #48~16.04.1-Ubuntu SMP Thu May 17 12:56:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

To assemble and run the code above, I use the following commands:

$ fasm strange_memcmp.asm && ld -dynamic-linker /lib64/ld-linux-x86-64.so.2 -lc -melf_x86_64 strange_memcmp.o -o strange_memcmp.out && ./strange_memcmp.out 
flat assembler  version 1.71.51  (16384 kilobytes memory)
3 passes, 803 bytes.
$ echo $?
0

I expect that memcmp will return a value that is not 0, because the buffers are different.

I tried changing the two buffers. If I set:

str1 = '2509487271'
str2 = '1509487273'

I obtain a return value of 1.

If I set:

str1 = '1409487271'
str2 = '1509487273'

I obtain a return value of 0. It seems like memcmp cares only about characters at even positions: 0, 2, 4, ... etc.

I have a strange feeling that I am calling memcmp incorrectly. This might have something to do with Linux x64 abi, but I can't find out what could be the problem. Any ideas are appreciated!

EDIT: According to Raymond's answer, the return value from the process is anded with a mask.

To fix the code above, I added the two instructions:

neg     rax
sbb     rax, rax

right after the call to memcmp.

Answer 1

You are passing the 64-bit value returned from memcmp directly to exit, but the value is ANDed with 0xFF before being returned to the parent. memcmp is probably returning a nonzero value like 512, but the system truncates it to 0.