llvm dev - Feb 2017 - sanitize=cfi-icall in real life application on x86 and ARM

Hi,

After going through small samples showing that "icall" is working, I
wanted to give a try with more complex stuff.
I decided to use nginx. As first, I tried with x86 platform. It went quite
smoothly, there is one runtime error reported by CFI, I fixed that the server
was working fine, without any issues.
Then I switched to the destination platform, ARMv7 based. This time nginx with
cfi-icall enabled flag became completely unusable. Worker processes crashed just
after spawning them.
I checked with different "sanitize" flags - like SafeStack, etc. - in
all these cases it was fine. So only with "icall" there were problems.
Started debugging it:
===GDB OUTPUT==Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0007e2cc in ngx_execute_proc.cfi ()

(gdb) backtrace
#0  0x0007e2cc in ngx_execute_proc.cfi ()
#1  0x0007e05c in ngx_spawn_process ()
#2  0x0007f8f0 in ngx_master_process_cycle.cfi ()
#3  0x000585dc in main ()
======
My questions are:
* What exactly is this add-on ".cfi", I can't see it when
compiling without "icall" flag
* Any ideas how to continue? I started disabling source folders from cfi
checking, as first, with "os", but then segmentation fault occurred in
other place - so I would have to blacklist 6 folder out of 7.

I know that "icall" for ARM is quite recent implementation, so might
be still something to improve there.

I tried on 2 different ARMx7 based platforms - i.MX6 and Raspberry Pi 2. Results
were the same as one might expect.

Regards,
Michal

After more debugging I found that the problem is with passing a function pointer
as parameter and calling it.
So in case of nginx it looks following:

ngx_start_worker_processes(ngx_cycle_t *cycle, ngx_int_t n, ngx_int_t type)
{
    ngx_int_t      i;
    ngx_channel_t  ch;

    ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "start worker
processes");

    ngx_memzero(&ch, sizeof(ngx_channel_t));

    ch.command = NGX_CMD_OPEN_CHANNEL;

    for (i = 0; i < n; i++) {

        ngx_spawn_process(cycle, ngx_worker_process_cycle,
                          (void *) (intptr_t) i, "worker process",
type);

        ch.pid = ngx_processes[ngx_process_slot].pid;
        ch.slot = ngx_process_slot;
        ch.fd = ngx_processes[ngx_process_slot].channel[0];

        ngx_pass_open_channel(cycle, &ch);
    }
}

ngx_pid_t
ngx_spawn_process(ngx_cycle_t *cycle, ngx_spawn_proc_pt proc, void *data,
    char *name, ngx_int_t respawn)
{
      ...
      proc(cycle, data);
      ...
}

inside nginx_spawn_process instead of calling ngx_worker_process_cycle, there is
another function invoked - ngx_execute_proc, which expects different data
(definitely not 0,1,2,3,...). As results nginx crashes.
And again, this happens ONLY with CFI (icall) enabled.

Could you please help?

regards,
Michal


________________________________________
From: Kurek, Michal
Sent: Friday, February 10, 2017 1:14 PM
To: llvm-dev at lists.llvm.org
Subject: sanitize=cfi-icall in real life application on x86 and ARM

Hi,

After going through small samples showing that “icall” is working, I wanted to
give a try with more complex stuff.
I decided to use nginx. As first, I tried with x86 platform. It went quite
smoothly, there is one runtime error reported by CFI, I fixed that the server
was working fine, without any issues.
Then I switched to the destination platform, ARMv7 based. This time nginx with
cfi-icall enabled flag became completely unusable. Worker processes crashed just
after spawning them.
I checked with different “sanitize” flags – like SafeStack, etc. - in all these
cases it was fine. So only with “icall” there were problems.
Started debugging it:
===GDB OUTPUT==Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0007e2cc in ngx_execute_proc.cfi ()

(gdb) backtrace
#0  0x0007e2cc in ngx_execute_proc.cfi ()
#1  0x0007e05c in ngx_spawn_process ()
#2  0x0007f8f0 in ngx_master_process_cycle.cfi ()
#3  0x000585dc in main ()
======
My questions are:
• What exactly is this add-on “.cfi”, I can’t see it when compiling without
“icall” flag
• Any ideas how to continue? I started disabling source folders from cfi
checking, as first, with “os”, but then segmentation fault occurred in other
place – so I would have to blacklist 6 folder out of 7.

I know that “icall” for ARM is quite recent implementation, so might be still
something to improve there.

I tried on 2 different ARMx7 based platforms – i.MX6 and Raspberry Pi 2. Results
were the same as one might expect.

Regards,
Michal