llvm dev - Aug 2018 - Default compression level for -compress-debug-info=zlib?

Folks,

I'd like to get expert's opinion on which compression level is suitable
for
lld's -compress-debug-section=zlib option, which let the linker compress
.debug_* sections using zlib.

Currently, lld uses compression level 9 which produces the smallest output
in exchange for a longer link time. My question is, is this what people
actually want? We didn't consciously choose compression level 9. That was
just the default compression level for zlib::compress function.

For an experiment, I created a patch to use compression level 1 instead of
9 and linked clang using that modified lld. By default, lld takes 1m4s to
link clang with --compress-debug-sections=zlib. With that patch, it took
only 31s.

Here is a comparison of clang executable size with various configurations:

no debug sections:    275 MB
level 9 compression:  855 MB
level 1 compression:  922 MB
no compression:      2044 MB

Given that the best compression takes significantly longer time than the
fastest compression, we probably should change the default to level 1. Any
objections?

I wonder what is the best compression level when -O2 is passed to lld. We
could use level 9 when -O2 is passed, but is there any reason to compress
debug sections that hard in the first place?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180801/7c57e5c0/attachment.html>

I don't claim to be an expert, but I did some zlib compression
benchmarks in the past. IIRC, my conclusion from that was that the
"DEFAULT" zlib level (6) is indeed a very good default for a lot of
cases -- it does not generate much larger outputs, while being
significantly faster than the max level. This all depends on the data
set and what you intend to do with the resulting data, of course, but
I guess my point is you don't have to choose only between 1 and 9. I
think it would be interesting to at least get the data for the default
level before making choice.

cheers,
pl
On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Folks,
>
> I'd like to get expert's opinion on which compression level is
suitable for lld's -compress-debug-section=zlib option, which let the linker
compress .debug_* sections using zlib.
>
> Currently, lld uses compression level 9 which produces the smallest output
in exchange for a longer link time. My question is, is this what people actually
want? We didn't consciously choose compression level 9. That was just the
default compression level for zlib::compress function.
>
> For an experiment, I created a patch to use compression level 1 instead of
9 and linked clang using that modified lld. By default, lld takes 1m4s to link
clang with --compress-debug-sections=zlib. With that patch, it took only 31s.
>
> Here is a comparison of clang executable size with various configurations:
>
> no debug sections:    275 MB
> level 9 compression:  855 MB
> level 1 compression:  922 MB
> no compression:      2044 MB
>
> Given that the best compression takes significantly longer time than the
fastest compression, we probably should change the default to level 1. Any
objections?
>
> I wonder what is the best compression level when -O2 is passed to lld. We
could use level 9 when -O2 is passed, but is there any reason to compress debug
sections that hard in the first place?
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Also not an expert, but would it make sense for this to be configurable at
a fine-grained level, perhaps with another option, or an extension to the
compress-debug-sections switch interface? That way users who care about the
finer details can configure it themselves. And we should pick sensible
options for the default.

James

On 2 August 2018 at 11:08, Pavel Labath via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I don't claim to be an expert, but I did some zlib compression
> benchmarks in the past. IIRC, my conclusion from that was that the
> "DEFAULT" zlib level (6) is indeed a very good default for a lot
of
> cases -- it does not generate much larger outputs, while being
> significantly faster than the max level. This all depends on the data
> set and what you intend to do with the resulting data, of course, but
> I guess my point is you don't have to choose only between 1 and 9. I
> think it would be interesting to at least get the data for the default
> level before making choice.
>
> cheers,
> pl
> On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Folks,
> >
> > I'd like to get expert's opinion on which compression level is
suitable
> for lld's -compress-debug-section=zlib option, which let the linker
> compress .debug_* sections using zlib.
> >
> > Currently, lld uses compression level 9 which produces the smallest
> output in exchange for a longer link time. My question is, is this what
> people actually want? We didn't consciously choose compression level 9.
> That was just the default compression level for zlib::compress function.
> >
> > For an experiment, I created a patch to use compression level 1
instead
> of 9 and linked clang using that modified lld. By default, lld takes 1m4s
> to link clang with --compress-debug-sections=zlib. With that patch, it
> took only 31s.
> >
> > Here is a comparison of clang executable size with various
> configurations:
> >
> > no debug sections:    275 MB
> > level 9 compression:  855 MB
> > level 1 compression:  922 MB
> > no compression:      2044 MB
> >
> > Given that the best compression takes significantly longer time than
the
> fastest compression, we probably should change the default to level 1. Any
> objections?
> >
> > I wonder what is the best compression level when -O2 is passed to lld.
> We could use level 9 when -O2 is passed, but is there any reason to
> compress debug sections that hard in the first place?
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180802/002a7d8e/attachment.html>

Here is the results of linking clang (with debug info) with various
compression levels. Apparently, the current compression level 9 seems
overkill.

Level   Time            Size
0       0m17.128s       2045081496   Z_NO_COMPRESSION
1       0m31.471s       922618584    Z_BEST_SPEED
2       0m32.659s       903642376
3       0m36.749s       890805856
4       0m41.532s       876697184
5       0m48.383s       862778576
6       1m3.176s        855251640    Z_DEFAULT_COMPRESSION
7       1m15.335s       853755920
8       2m0.561s        852497560
9       2m33.972s       852397408    Z_BEST_COMPRESSION

On Thu, Aug 2, 2018 at 3:09 AM Pavel Labath <labath at google.com> wrote:
> I don't claim to be an expert, but I did some zlib compression
> benchmarks in the past. IIRC, my conclusion from that was that the
> "DEFAULT" zlib level (6) is indeed a very good default for a lot
of
> cases -- it does not generate much larger outputs, while being
> significantly faster than the max level. This all depends on the data
> set and what you intend to do with the resulting data, of course, but
> I guess my point is you don't have to choose only between 1 and 9. I
> think it would be interesting to at least get the data for the default
> level before making choice.
>
> cheers,
> pl
> On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Folks,
> >
> > I'd like to get expert's opinion on which compression level is
suitable
> for lld's -compress-debug-section=zlib option, which let the linker
> compress .debug_* sections using zlib.
> >
> > Currently, lld uses compression level 9 which produces the smallest
> output in exchange for a longer link time. My question is, is this what
> people actually want? We didn't consciously choose compression level 9.
> That was just the default compression level for zlib::compress function.
> >
> > For an experiment, I created a patch to use compression level 1
instead
> of 9 and linked clang using that modified lld. By default, lld takes 1m4s
> to link clang with --compress-debug-sections=zlib. With that patch, it took
> only 31s.
> >
> > Here is a comparison of clang executable size with various
> configurations:
> >
> > no debug sections:    275 MB
> > level 9 compression:  855 MB
> > level 1 compression:  922 MB
> > no compression:      2044 MB
> >
> > Given that the best compression takes significantly longer time than
the
> fastest compression, we probably should change the default to level 1. Any
> objections?
> >
> > I wonder what is the best compression level when -O2 is passed to lld.
> We could use level 9 when -O2 is passed, but is there any reason to
> compress debug sections that hard in the first place?
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180802/a190a229/attachment.html>