llvm dev - Mar 2019 - llvm symbolizer not able to parse debuginfo files

I am trying to run NFS-Ganesha with ASAN in our setup. I am having
difficulties to make llvm symbolizer print symbol names from the
.debug binaries/libraries, once ASAN shows any error;

bash-4.2# /opt/rh/llvm-toolset-7/root/usr/bin/llvm-symbolizer --version
LLVM (http://llvm.org/):
  LLVM version 5.0.1
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: nocona

I am getting the following error (The addr2line is working ok, also
gdb is able to parse these files):
python asan_symbolize.py  < asan.log.173664
==================================================================173664==
ERROR: AddressSanitizer: heap-use-after-free on address
0x60040002ff90 at pc 0x7f83d22f91bb bp 0x7f83c17f3e40 sp
0x7f83c17f3e30
WRITE of size 4 at 0x60040002ff90 thread T12
    #0 0x7f83d22f91ba
(/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug+0x251ba)

/opt/rh/llvm-toolset-7/root/usr/bin/llvm-symbolizer
--use-symbol-table=true --demangle=False --inlining=true
--default-arch=i386
"/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug"
0x251ba
LLVMSymbolizer: error reading file: Invalid data was encountered while
parsing the file
addr2line -fi -e /usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug
"/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug"
0x251ba
    #0 0x7f83d22f91ba in minerva_mkdir
/usr/src/debug/nfs-ganesha-2.7.1/FSAL/FSAL_MINERVA/handle.c:747

How can I fix this?

I googled a bit about using the debug binaries/libraries with
llvm-symbolizer/asan_symbolize.py, I found that there was a patch
proposed:
https://reviews.llvm.org/D4902
But this was discarded saying:
"
This logic is already implemented in the symbolization tools, if you
want them to work with custom sysroot (so that debug versions of
binaries could be found at /path/to/my/sysroot/usr/lib/debug), just
add the --sysroot flag to the tools
"
Apart from the parsing error mentioned above, I have couple of other concerns:
a) I have to manually append .debug to all the binaries/libraries for
which I have a corresponding .debug image in the asan.logfile
b) The sysroot parameter cannot be used when all the libraries used by
the applications does not have corresponding .debug versions, for
example, in our setup libc does not have .debug version, in that case
I have to replace the entire path of entities which have .debug images
otherwise the symbolizer fails with no such file for libc.

Is there a way to do this more easily? (I hope I am asking the
question on correct list, please point me to
the correct list otherwise.)
Thanks!
Ashish

*ping*

On Mon, Mar 18, 2019 at 3:59 PM Ashish Sangwan <ashishsangwan2 at
gmail.com> wrote:
>
> I am trying to run NFS-Ganesha with ASAN in our setup. I am having
> difficulties to make llvm symbolizer print symbol names from the
> .debug binaries/libraries, once ASAN shows any error;
>
> bash-4.2# /opt/rh/llvm-toolset-7/root/usr/bin/llvm-symbolizer --version
> LLVM (http://llvm.org/):
>   LLVM version 5.0.1
>   Optimized build.
>   Default target: x86_64-unknown-linux-gnu
>   Host CPU: nocona
>
> I am getting the following error (The addr2line is working ok, also
> gdb is able to parse these files):
> python asan_symbolize.py  < asan.log.173664
> ================================================================>
==173664== ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60040002ff90 at pc 0x7f83d22f91bb bp 0x7f83c17f3e40 sp
> 0x7f83c17f3e30
> WRITE of size 4 at 0x60040002ff90 thread T12
>     #0 0x7f83d22f91ba
> (/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug+0x251ba)
>
> /opt/rh/llvm-toolset-7/root/usr/bin/llvm-symbolizer
> --use-symbol-table=true --demangle=False --inlining=true
> --default-arch=i386
> "/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug"
0x251ba
> LLVMSymbolizer: error reading file: Invalid data was encountered while
> parsing the file
> addr2line -fi -e
/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug
> "/usr/lib/debug/usr/lib64/ganesha/libfsalminerva.so.4.2.0.debug"
0x251ba
>     #0 0x7f83d22f91ba in minerva_mkdir
> /usr/src/debug/nfs-ganesha-2.7.1/FSAL/FSAL_MINERVA/handle.c:747
>
> How can I fix this?
>
> I googled a bit about using the debug binaries/libraries with
> llvm-symbolizer/asan_symbolize.py, I found that there was a patch
> proposed:
> https://reviews.llvm.org/D4902
> But this was discarded saying:
> "
> This logic is already implemented in the symbolization tools, if you
> want them to work with custom sysroot (so that debug versions of
> binaries could be found at /path/to/my/sysroot/usr/lib/debug), just
> add the --sysroot flag to the tools
> "
> Apart from the parsing error mentioned above, I have couple of other
concerns:
> a) I have to manually append .debug to all the binaries/libraries for
> which I have a corresponding .debug image in the asan.logfile
> b) The sysroot parameter cannot be used when all the libraries used by
> the applications does not have corresponding .debug versions, for
> example, in our setup libc does not have .debug version, in that case
> I have to replace the entire path of entities which have .debug images
> otherwise the symbolizer fails with no such file for libc.
>
> Is there a way to do this more easily? (I hope I am asking the
> question on correct list, please point me to
> the correct list otherwise.)
> Thanks!
> Ashish