llvm dev - Apr 2013 - [LLVMdev] RFC: Using zlib to decompress debug info sections.

On Tue, Apr 16, 2013 at 8:31 PM, Michael Spencer <bigcheesegs at
gmail.com>wrote:
> On Tue, Apr 16, 2013 at 2:37 AM, Alexey Samsonov <samsonov at
google.com>wrote:
>
>> Hi!
>>
>> TL;DR WDYT of adding zlib decompression capabilities to LLVMObject
>> library?
>>
>
>> ld.gold from GNU binutils has --compress-debug-sections=zlib option,
>>  which uses zlib to compress .debug_xxx sections and renames them to
>> .zdebug_xxx.
>> binutils (and GDB) support this properly, while LLVM command line tools
>> don't:
>>
>> $ ld --version
>> GNU gold (GNU Binutils for Ubuntu 2.22) 1.11
>> $ ./bin/clang++ -g a.cc -Wl,--compress-debug-sections=zlib
>> $ objdump -h a.out | grep debug
>>  26 .debug_info   00000066  0000000000000000  0000000000000000 
00002010
>>  2**0
>>  27 .debug_abbrev 00000048  0000000000000000  0000000000000000 
00002068
>>  2**0
>>  28 .debug_aranges 00000000  0000000000000000  0000000000000000 
000020bb
>>  2**0
>>  29 .debug_macinfo 00000000  0000000000000000  0000000000000000 
000020cf
>>  2**0
>>  30 .debug_line   00000053  0000000000000000  0000000000000000 
000020e3
>>  2**0
>>  31 .debug_loc    00000000  0000000000000000  0000000000000000 
0000213e
>>  2**0
>>  32 .debug_pubtypes 00000000  0000000000000000  0000000000000000
>>  00002152  2**0
>>  33 .debug_str    00000069  0000000000000000  0000000000000000 
00002166
>>  2**0
>>  34 .debug_ranges 00000000  0000000000000000  0000000000000000 
000021d9
>>  2**0
>> $ ./bin/llvm-objdump -h a.out | grep debug
>>  27 .zdebug_info  00000058 0000000000000000
>>  28 .zdebug_abbrev 00000053 0000000000000000
>>  29 .zdebug_aranges 00000014 0000000000000000
>>  30 .zdebug_macinfo 00000014 0000000000000000
>>  31 .zdebug_line  0000005b 0000000000000000
>>  32 .zdebug_loc   00000014 0000000000000000
>>  33 .zdebug_pubtypes 00000014 0000000000000000
>>  34 .zdebug_str   00000073 0000000000000000
>>  35 .zdebug_ranges 00000014 0000000000000000
>>
>> Decompression and proper handling of debug info sections may be needed
>> in llvm-dwarfdump and llvm-symbolizer tools. We can implement this by:
>> 1) Checking if zlib is present in the system during configuration.
>> 2) Adding zlib decompression to llvm::MemoryBuffer, and section
>> decompression to LLVMObject (this would require optional linking with
-lz).
>> 3) Using the methods in LLVM tools where needed.
>>
>> Does this make sense to you?
>>
>> --
>> Alexey Samsonov, MSK
>>
>
> I'm not sure MemoryBuffer is the right place to do this either. I'm
also
> not sure if we want debug info decompression to be transparent in
> LLVMObject or not. I'm leaning towards no since it's not part of
the
> standard yet,
>
Yeah, I also think that decompression should be explicitly requested by the
user of LLVMObject.

> unless gold is actually using the SHF_COMPRESSED flag.
>
> I think it should be part of Object, but as an external API that is used
> when you find a section you know from external factors (the name matches
> some list) is compressed.
>
> - Michael Spencer
>
>

-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130416/db863f0a/attachment.html>

On Tue, Apr 16, 2013 at 9:37 AM, Alexey Samsonov <samsonov at google.com>
wrote:
>
> On Tue, Apr 16, 2013 at 8:31 PM, Michael Spencer <bigcheesegs at
gmail.com>
> wrote:
>>
>> On Tue, Apr 16, 2013 at 2:37 AM, Alexey Samsonov <samsonov at
google.com>
>> wrote:
>>>
>>> Hi!
>>>
>>> TL;DR WDYT of adding zlib decompression capabilities to LLVMObject
>>> library?
>>>
>>>
>>> ld.gold from GNU binutils has --compress-debug-sections=zlib
option,
>>> which uses zlib to compress .debug_xxx sections and renames them to
>>> .zdebug_xxx.
>>> binutils (and GDB) support this properly, while LLVM command line
tools
>>> don't:
>>>
>>> $ ld --version
>>> GNU gold (GNU Binutils for Ubuntu 2.22) 1.11
>>> $ ./bin/clang++ -g a.cc -Wl,--compress-debug-sections=zlib
>>> $ objdump -h a.out | grep debug
>>>  26 .debug_info   00000066  0000000000000000  0000000000000000 
00002010
>>> 2**0
>>>  27 .debug_abbrev 00000048  0000000000000000  0000000000000000 
00002068
>>> 2**0
>>>  28 .debug_aranges 00000000  0000000000000000  0000000000000000 
000020bb
>>> 2**0
>>>  29 .debug_macinfo 00000000  0000000000000000  0000000000000000 
000020cf
>>> 2**0
>>>  30 .debug_line   00000053  0000000000000000  0000000000000000 
000020e3
>>> 2**0
>>>  31 .debug_loc    00000000  0000000000000000  0000000000000000 
0000213e
>>> 2**0
>>>  32 .debug_pubtypes 00000000  0000000000000000  0000000000000000
>>> 00002152  2**0
>>>  33 .debug_str    00000069  0000000000000000  0000000000000000 
00002166
>>> 2**0
>>>  34 .debug_ranges 00000000  0000000000000000  0000000000000000 
000021d9
>>> 2**0
>>> $ ./bin/llvm-objdump -h a.out | grep debug
>>>  27 .zdebug_info  00000058 0000000000000000
>>>  28 .zdebug_abbrev 00000053 0000000000000000
>>>  29 .zdebug_aranges 00000014 0000000000000000
>>>  30 .zdebug_macinfo 00000014 0000000000000000
>>>  31 .zdebug_line  0000005b 0000000000000000
>>>  32 .zdebug_loc   00000014 0000000000000000
>>>  33 .zdebug_pubtypes 00000014 0000000000000000
>>>  34 .zdebug_str   00000073 0000000000000000
>>>  35 .zdebug_ranges 00000014 0000000000000000
>>>
>>> Decompression and proper handling of debug info sections may be
needed
>>> in llvm-dwarfdump and llvm-symbolizer tools. We can implement this
by:
>>> 1) Checking if zlib is present in the system during configuration.
>>> 2) Adding zlib decompression to llvm::MemoryBuffer, and section
>>> decompression to LLVMObject (this would require optional linking
with -lz).
>>> 3) Using the methods in LLVM tools where needed.
>>>
>>> Does this make sense to you?
>>>
>>> --
>>> Alexey Samsonov, MSK
>>
>>
>> I'm not sure MemoryBuffer is the right place to do this either.
I'm also
>> not sure if we want debug info decompression to be transparent in
LLVMObject
>> or not. I'm leaning towards no since it's not part of the
standard yet,
>
>
> Yeah, I also think that decompression should be explicitly requested by the
> user of LLVMObject.
>
>>
>> unless gold is actually using the SHF_COMPRESSED flag.
>>
>>
>> I think it should be part of Object, but as an external API that is
used
>> when you find a section you know from external factors (the name
matches
>> some list) is compressed.
>>
Definitely want the feature :)

I don't see SHF_COMPRESSED (unless readelf just isn't showing it to
me), but it wouldn't be too hard to get binutils to mark them as such.
Right now the convention is .z<foo> are compressed, but that's not as
precise as we'd like it to be. There's been some talk on the binutils
list about it, but it hasn't been implemented yet.

-eric

Just in case - do we want to link with libz.so installed in the system, or
be self-contained and copy sources to LLVM repo?


On Tue, Apr 16, 2013 at 10:48 PM, Eric Christopher <echristo at
gmail.com>wrote:
> On Tue, Apr 16, 2013 at 9:37 AM, Alexey Samsonov <samsonov at
google.com>
> wrote:
> >
> > On Tue, Apr 16, 2013 at 8:31 PM, Michael Spencer <bigcheesegs at
gmail.com>
> > wrote:
> >>
> >> On Tue, Apr 16, 2013 at 2:37 AM, Alexey Samsonov <samsonov at
google.com>
> >> wrote:
> >>>
> >>> Hi!
> >>>
> >>> TL;DR WDYT of adding zlib decompression capabilities to
LLVMObject
> >>> library?
> >>>
> >>>
> >>> ld.gold from GNU binutils has --compress-debug-sections=zlib
option,
> >>> which uses zlib to compress .debug_xxx sections and renames
them to
> >>> .zdebug_xxx.
> >>> binutils (and GDB) support this properly, while LLVM command
line tools
> >>> don't:
> >>>
> >>> $ ld --version
> >>> GNU gold (GNU Binutils for Ubuntu 2.22) 1.11
> >>> $ ./bin/clang++ -g a.cc -Wl,--compress-debug-sections=zlib
> >>> $ objdump -h a.out | grep debug
> >>>  26 .debug_info   00000066  0000000000000000  0000000000000000
>  00002010
> >>> 2**0
> >>>  27 .debug_abbrev 00000048  0000000000000000  0000000000000000
>  00002068
> >>> 2**0
> >>>  28 .debug_aranges 00000000  0000000000000000 
0000000000000000
>  000020bb
> >>> 2**0
> >>>  29 .debug_macinfo 00000000  0000000000000000 
0000000000000000
>  000020cf
> >>> 2**0
> >>>  30 .debug_line   00000053  0000000000000000  0000000000000000
>  000020e3
> >>> 2**0
> >>>  31 .debug_loc    00000000  0000000000000000  0000000000000000
>  0000213e
> >>> 2**0
> >>>  32 .debug_pubtypes 00000000  0000000000000000 
0000000000000000
> >>> 00002152  2**0
> >>>  33 .debug_str    00000069  0000000000000000  0000000000000000
>  00002166
> >>> 2**0
> >>>  34 .debug_ranges 00000000  0000000000000000  0000000000000000
>  000021d9
> >>> 2**0
> >>> $ ./bin/llvm-objdump -h a.out | grep debug
> >>>  27 .zdebug_info  00000058 0000000000000000
> >>>  28 .zdebug_abbrev 00000053 0000000000000000
> >>>  29 .zdebug_aranges 00000014 0000000000000000
> >>>  30 .zdebug_macinfo 00000014 0000000000000000
> >>>  31 .zdebug_line  0000005b 0000000000000000
> >>>  32 .zdebug_loc   00000014 0000000000000000
> >>>  33 .zdebug_pubtypes 00000014 0000000000000000
> >>>  34 .zdebug_str   00000073 0000000000000000
> >>>  35 .zdebug_ranges 00000014 0000000000000000
> >>>
> >>> Decompression and proper handling of debug info sections may
be needed
> >>> in llvm-dwarfdump and llvm-symbolizer tools. We can implement
this by:
> >>> 1) Checking if zlib is present in the system during
configuration.
> >>> 2) Adding zlib decompression to llvm::MemoryBuffer, and
section
> >>> decompression to LLVMObject (this would require optional
linking with
> -lz).
> >>> 3) Using the methods in LLVM tools where needed.
> >>>
> >>> Does this make sense to you?
> >>>
> >>> --
> >>> Alexey Samsonov, MSK
> >>
> >>
> >> I'm not sure MemoryBuffer is the right place to do this
either. I'm also
> >> not sure if we want debug info decompression to be transparent in
> LLVMObject
> >> or not. I'm leaning towards no since it's not part of the
standard yet,
> >
> >
> > Yeah, I also think that decompression should be explicitly requested
by
> the
> > user of LLVMObject.
> >
> >>
> >> unless gold is actually using the SHF_COMPRESSED flag.
> >>
> >>
> >> I think it should be part of Object, but as an external API that
is used
> >> when you find a section you know from external factors (the name
matches
> >> some list) is compressed.
> >>
>
> Definitely want the feature :)
>
> I don't see SHF_COMPRESSED (unless readelf just isn't showing it to
> me), but it wouldn't be too hard to get binutils to mark them as such.
> Right now the convention is .z<foo> are compressed, but that's
not as
> precise as we'd like it to be. There's been some talk on the
binutils
> list about it, but it hasn't been implemented yet.
>
> -eric
>


-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130416/e7bcd553/attachment.html>