llvm dev - Apr 2013 - [LLVMdev] RFC: Using zlib to decompress debug info sections.

Just in case - do we want to link with libz.so installed in the system, or
be self-contained and copy sources to LLVM repo?


On Tue, Apr 16, 2013 at 10:48 PM, Eric Christopher <echristo at
gmail.com>wrote:
> On Tue, Apr 16, 2013 at 9:37 AM, Alexey Samsonov <samsonov at
google.com>
> wrote:
> >
> > On Tue, Apr 16, 2013 at 8:31 PM, Michael Spencer <bigcheesegs at
gmail.com>
> > wrote:
> >>
> >> On Tue, Apr 16, 2013 at 2:37 AM, Alexey Samsonov <samsonov at
google.com>
> >> wrote:
> >>>
> >>> Hi!
> >>>
> >>> TL;DR WDYT of adding zlib decompression capabilities to
LLVMObject
> >>> library?
> >>>
> >>>
> >>> ld.gold from GNU binutils has --compress-debug-sections=zlib
option,
> >>> which uses zlib to compress .debug_xxx sections and renames
them to
> >>> .zdebug_xxx.
> >>> binutils (and GDB) support this properly, while LLVM command
line tools
> >>> don't:
> >>>
> >>> $ ld --version
> >>> GNU gold (GNU Binutils for Ubuntu 2.22) 1.11
> >>> $ ./bin/clang++ -g a.cc -Wl,--compress-debug-sections=zlib
> >>> $ objdump -h a.out | grep debug
> >>>  26 .debug_info   00000066  0000000000000000  0000000000000000
>  00002010
> >>> 2**0
> >>>  27 .debug_abbrev 00000048  0000000000000000  0000000000000000
>  00002068
> >>> 2**0
> >>>  28 .debug_aranges 00000000  0000000000000000 
0000000000000000
>  000020bb
> >>> 2**0
> >>>  29 .debug_macinfo 00000000  0000000000000000 
0000000000000000
>  000020cf
> >>> 2**0
> >>>  30 .debug_line   00000053  0000000000000000  0000000000000000
>  000020e3
> >>> 2**0
> >>>  31 .debug_loc    00000000  0000000000000000  0000000000000000
>  0000213e
> >>> 2**0
> >>>  32 .debug_pubtypes 00000000  0000000000000000 
0000000000000000
> >>> 00002152  2**0
> >>>  33 .debug_str    00000069  0000000000000000  0000000000000000
>  00002166
> >>> 2**0
> >>>  34 .debug_ranges 00000000  0000000000000000  0000000000000000
>  000021d9
> >>> 2**0
> >>> $ ./bin/llvm-objdump -h a.out | grep debug
> >>>  27 .zdebug_info  00000058 0000000000000000
> >>>  28 .zdebug_abbrev 00000053 0000000000000000
> >>>  29 .zdebug_aranges 00000014 0000000000000000
> >>>  30 .zdebug_macinfo 00000014 0000000000000000
> >>>  31 .zdebug_line  0000005b 0000000000000000
> >>>  32 .zdebug_loc   00000014 0000000000000000
> >>>  33 .zdebug_pubtypes 00000014 0000000000000000
> >>>  34 .zdebug_str   00000073 0000000000000000
> >>>  35 .zdebug_ranges 00000014 0000000000000000
> >>>
> >>> Decompression and proper handling of debug info sections may
be needed
> >>> in llvm-dwarfdump and llvm-symbolizer tools. We can implement
this by:
> >>> 1) Checking if zlib is present in the system during
configuration.
> >>> 2) Adding zlib decompression to llvm::MemoryBuffer, and
section
> >>> decompression to LLVMObject (this would require optional
linking with
> -lz).
> >>> 3) Using the methods in LLVM tools where needed.
> >>>
> >>> Does this make sense to you?
> >>>
> >>> --
> >>> Alexey Samsonov, MSK
> >>
> >>
> >> I'm not sure MemoryBuffer is the right place to do this
either. I'm also
> >> not sure if we want debug info decompression to be transparent in
> LLVMObject
> >> or not. I'm leaning towards no since it's not part of the
standard yet,
> >
> >
> > Yeah, I also think that decompression should be explicitly requested
by
> the
> > user of LLVMObject.
> >
> >>
> >> unless gold is actually using the SHF_COMPRESSED flag.
> >>
> >>
> >> I think it should be part of Object, but as an external API that
is used
> >> when you find a section you know from external factors (the name
matches
> >> some list) is compressed.
> >>
>
> Definitely want the feature :)
>
> I don't see SHF_COMPRESSED (unless readelf just isn't showing it to
> me), but it wouldn't be too hard to get binutils to mark them as such.
> Right now the convention is .z<foo> are compressed, but that's
not as
> precise as we'd like it to be. There's been some talk on the
binutils
> list about it, but it hasn't been implemented yet.
>
> -eric
>


-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130416/e7bcd553/attachment.html>

Historically we've done the former. The latter would require Chris
wanting to do that.

-eric

On Tue, Apr 16, 2013 at 11:52 AM, Alexey Samsonov <samsonov at google.com>
wrote:
> Just in case - do we want to link with libz.so installed in the system, or
> be self-contained and copy sources to LLVM repo?
>
>
> On Tue, Apr 16, 2013 at 10:48 PM, Eric Christopher <echristo at
gmail.com>
> wrote:
>>
>> On Tue, Apr 16, 2013 at 9:37 AM, Alexey Samsonov <samsonov at
google.com>
>> wrote:
>> >
>> > On Tue, Apr 16, 2013 at 8:31 PM, Michael Spencer <bigcheesegs
at gmail.com>
>> > wrote:
>> >>
>> >> On Tue, Apr 16, 2013 at 2:37 AM, Alexey Samsonov <samsonov
at google.com>
>> >> wrote:
>> >>>
>> >>> Hi!
>> >>>
>> >>> TL;DR WDYT of adding zlib decompression capabilities to
LLVMObject
>> >>> library?
>> >>>
>> >>>
>> >>> ld.gold from GNU binutils has
--compress-debug-sections=zlib option,
>> >>> which uses zlib to compress .debug_xxx sections and
renames them to
>> >>> .zdebug_xxx.
>> >>> binutils (and GDB) support this properly, while LLVM
command line
>> >>> tools
>> >>> don't:
>> >>>
>> >>> $ ld --version
>> >>> GNU gold (GNU Binutils for Ubuntu 2.22) 1.11
>> >>> $ ./bin/clang++ -g a.cc -Wl,--compress-debug-sections=zlib
>> >>> $ objdump -h a.out | grep debug
>> >>>  26 .debug_info   00000066  0000000000000000 
0000000000000000
>> >>> 00002010
>> >>> 2**0
>> >>>  27 .debug_abbrev 00000048  0000000000000000 
0000000000000000
>> >>> 00002068
>> >>> 2**0
>> >>>  28 .debug_aranges 00000000  0000000000000000 
0000000000000000
>> >>> 000020bb
>> >>> 2**0
>> >>>  29 .debug_macinfo 00000000  0000000000000000 
0000000000000000
>> >>> 000020cf
>> >>> 2**0
>> >>>  30 .debug_line   00000053  0000000000000000 
0000000000000000
>> >>> 000020e3
>> >>> 2**0
>> >>>  31 .debug_loc    00000000  0000000000000000 
0000000000000000
>> >>> 0000213e
>> >>> 2**0
>> >>>  32 .debug_pubtypes 00000000  0000000000000000 
0000000000000000
>> >>> 00002152  2**0
>> >>>  33 .debug_str    00000069  0000000000000000 
0000000000000000
>> >>> 00002166
>> >>> 2**0
>> >>>  34 .debug_ranges 00000000  0000000000000000 
0000000000000000
>> >>> 000021d9
>> >>> 2**0
>> >>> $ ./bin/llvm-objdump -h a.out | grep debug
>> >>>  27 .zdebug_info  00000058 0000000000000000
>> >>>  28 .zdebug_abbrev 00000053 0000000000000000
>> >>>  29 .zdebug_aranges 00000014 0000000000000000
>> >>>  30 .zdebug_macinfo 00000014 0000000000000000
>> >>>  31 .zdebug_line  0000005b 0000000000000000
>> >>>  32 .zdebug_loc   00000014 0000000000000000
>> >>>  33 .zdebug_pubtypes 00000014 0000000000000000
>> >>>  34 .zdebug_str   00000073 0000000000000000
>> >>>  35 .zdebug_ranges 00000014 0000000000000000
>> >>>
>> >>> Decompression and proper handling of debug info sections
may be needed
>> >>> in llvm-dwarfdump and llvm-symbolizer tools. We can
implement this by:
>> >>> 1) Checking if zlib is present in the system during
configuration.
>> >>> 2) Adding zlib decompression to llvm::MemoryBuffer, and
section
>> >>> decompression to LLVMObject (this would require optional
linking with
>> >>> -lz).
>> >>> 3) Using the methods in LLVM tools where needed.
>> >>>
>> >>> Does this make sense to you?
>> >>>
>> >>> --
>> >>> Alexey Samsonov, MSK
>> >>
>> >>
>> >> I'm not sure MemoryBuffer is the right place to do this
either. I'm
>> >> also
>> >> not sure if we want debug info decompression to be transparent
in
>> >> LLVMObject
>> >> or not. I'm leaning towards no since it's not part of
the standard yet,
>> >
>> >
>> > Yeah, I also think that decompression should be explicitly
requested by
>> > the
>> > user of LLVMObject.
>> >
>> >>
>> >> unless gold is actually using the SHF_COMPRESSED flag.
>> >>
>> >>
>> >> I think it should be part of Object, but as an external API
that is
>> >> used
>> >> when you find a section you know from external factors (the
name
>> >> matches
>> >> some list) is compressed.
>> >>
>>
>> Definitely want the feature :)
>>
>> I don't see SHF_COMPRESSED (unless readelf just isn't showing
it to
>> me), but it wouldn't be too hard to get binutils to mark them as
such.
>> Right now the convention is .z<foo> are compressed, but
that's not as
>> precise as we'd like it to be. There's been some talk on the
binutils
>> list about it, but it hasn't been implemented yet.
>>
>> -eric
>
>
>
>
> --
> Alexey Samsonov, MSK

On Apr 16, 2013, at 11:53 AM, Eric Christopher <echristo at gmail.com>
wrote:
> Historically we've done the former. The latter would require Chris
> wanting to do that.
This case isn't so clearcut.  We like to include libraries in the source to
make it easy to get up and running without having to install a ton of
dependencies.  However, this has license implications and is generally annoying.

Given that zlib is so widely available by default, and that the compiler can
generate correct (albeit uncompressed) debug info, I think the best thing is to
*not* include a copy in llvm.  Just detect and use it if we can find it, but
otherwise generate uncompressed output.

-Chris