llvm dev - May 2019 - [LLD] Should --compress_debug_sections be enabled (=zlib) by default ?

Hi,

In the file lld/ELF/Driver.cpp in function getCompressDebugSections we can see
that the current default for lld is no debug section compression. It looks like
tools like gdb, valgrind, elfutils, gcc's backtrace lib currently support
compressed symbols. Since perf can use libdw from elfutils, I guess it supports
it too.

Do you think it's time to enable compressed debug section by default ? On
some of my big binaries (several hundreds of MB) this could save 50% of the
overall binary size.

Cheers,
Romain

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Romain GEISSLER via llvm-dev
> Sent: Friday, May 03, 2019 12:53 PM
> To: llvm-dev at lists.llvm.org
> Subject: [llvm-dev] [LLD] Should --compress_debug_sections be enabled
> (=zlib) by default ?
> 
> Hi,
> 
> In the file lld/ELF/Driver.cpp in function getCompressDebugSections we can
> see that the current default for lld is no debug section compression. It
> looks like tools like gdb, valgrind, elfutils, gcc's backtrace lib
> currently support compressed symbols. Since perf can use libdw from
> elfutils, I guess it supports it too.
> 
> Do you think it's time to enable compressed debug section by default ?
On
> some of my big binaries (several hundreds of MB) this could save 50% of
> the overall binary size.
I think we'd want a way to make this customizable for different platforms.
--paulr
> 
> Cheers,
> Romain
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

We can compress debug sections in compiler-generated object files as well
as linker-generated executables. So the question whether we should compress
debug sections is not only relevant to lld but to clang, and I think it's
actually a whole toolchain-level decision.

If we decide to compress debug sections for some target, we should make a
change to the compiler driver to flip the default for a compiler flag to
turn on compression. lld is invoked by the compiler driver, so a change for
the linker would be added to the compiler driver too.

I think I'm mildly in favor of making such change -- I guess the cost of
compress doesn't too high, and that might even make things *faster* by
reducing the amount of IO. I'd guess people wouldn't even care whether
debug sections are compressed or not, as their tools already support it.
But I don't have an evidence that the latter is actually the case.

*From: *Romain GEISSLER via llvm-dev <llvm-dev at lists.llvm.org>
*Date: *Mon, May 6, 2019 at 2:18 AM
*To: *llvm-dev at lists.llvm.org

Hi,
>
> In the file lld/ELF/Driver.cpp in function getCompressDebugSections we can
> see that the current default for lld is no debug section compression. It
> looks like tools like gdb, valgrind, elfutils, gcc's backtrace lib
> currently support compressed symbols. Since perf can use libdw from
> elfutils, I guess it supports it too.
>
> Do you think it's time to enable compressed debug section by default ?
On
> some of my big binaries (several hundreds of MB) this could save 50% of the
> overall binary size.
>
> Cheers,
> Romain
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190507/0425e12e/attachment.html>

Le 7 mai 2019 à 09:03, Rui Ueyama <ruiu at google.com<mailto:ruiu at
google.com>> a écrit :

Hi,


We can compress debug sections in compiler-generated object files as well as
linker-generated executables. So the question whether we should compress debug
sections is not only relevant to lld but to clang, and I think it's actually
a whole toolchain-level decision.

If we assume the compiler is clang. However in my case it would be gcc (I agree
this is not a very ordinary mix of compiler/linker). When you use the full GNU
toolchain, this choice of default value is done purely in binutils (you can
configure it with the flag
--enable-compressed-debug-sections={all,gas,gold,ld,none}).

Today as far as I know in binutils, to reduce IO, it's enabled by default in
"gas" but not in "ld.bfd" (and by the way, it's not even
implemented in "ld.gold", I have to submit a patch for that). This may
be true for some target only, I am considering Linux x64. The fear in enabling
it in "ld.bfd" is that it might break tools. For the tool I use (gdb,
valgrind, (gcc) sanitizers (with gcc's libbacktrace for the stack printing))
it works. There might be other tools broken by this change. So yes it make sense
to make the change at the whole toolchain level, in both the assembler + the
linker. I don't know if clang uses it's own LLVM assembler or GNU gas.

I think I'm mildly in favor of making such change -- I guess the cost of
compress doesn't too high, and that might even make things faster by
reducing the amount of IO. I'd guess people wouldn't even care whether
debug sections are compressed or not, as their tools already support it. But I
don't have an evidence that the latter is actually the case.

Ok. I will just patch lld myself then, only for my own toolchain.

Thanks,
Romain
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190507/815a7017/attachment.html>

On Fri, 03 May 2019 18:52:48 +0200, Romain GEISSLER via llvm-dev
wrote:
> Do you think it's time to enable compressed debug section by default ?
It depends whether you use index (Apple index or .gdb_index or .debug_names).
If not then compressed debug is probably always a win.
.gdb_index is not going to be supported by LLDB (it is supported by GDB),
.debug_names (at LLD level) is going to be supported by LLDB in some time.

If you do use index then loading such file will be slower as this type of
compression cannot decompress only small section of the file addressed by the
index. Then some filesystem level (or block device level) compressed can be
a better compromise between disk size and debug info access performance.

But it sure needs more numbers.


Jan