llvm dev - Jan 2016 - LLD benchmark results for all commits

Speed is one of the most important goals for the new ELF linker. I ran a
benchmark for every commit to LLD/ELF that was made after the last October,
and I'd like to share the result. The motivation is to see if the linker is
getting slower as we add more features, and if so, which commits are heavy
ones in terms of performance. I measured time to link the same Clang
executable.

Here is the result. This table contains a commit message, a SVN revision
number, time to link, size of the resulting Clang binary in each row.

https://docs.google.com/spreadsheets/d/1QlQZ51VQUna6gX0qXj19v_ZEII5vGaHrUhBKq-Z_9uc/edit?usp=sharing

As you can see, the linker is faster than the original. A few key findings
are listed below:

 - The linker tend to be gradually getting slower over time in general.
 - There are a few commits that increased link time by a few percent or
more.
 - However, we also committed a few performance improvement changes to
offset them.

The commit that increased link time most (+15%) is r251212: Add support for
merging string from SHF_STRINGS sections. This does not come as a surprise
since string merging is a computationally heavy operation.

The commit that decreased link time most (-29%) is r250315: ELF2: Merge
.{text,rodata,data,bss}.* sections. This is because the commit
significantly decreased number of sections in the output.

The following commits are also noteworthy.

 - r257017: ELF: Improve performance of string table construction (-14.0%)
 - r251332: Optimize StringTableBuilder (-7.8%)

Ideally, we should have a perf bot to run a benchmark for every commit, so
that we do not accidentally commit changes that hurt performance.

Overall, I'm satisfied with the result. It found no commit that increases
link time for no justifiable reason.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160115/fb72ef2b/attachment.html>

Cool - thanks for sharing!

Perhaps you could add a graph of the times in another sheet/beside the
table?

On Fri, Jan 15, 2016 at 2:42 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Speed is one of the most important goals for the new ELF linker. I ran a
> benchmark for every commit to LLD/ELF that was made after the last October,
> and I'd like to share the result. The motivation is to see if the
linker is
> getting slower as we add more features, and if so, which commits are heavy
> ones in terms of performance. I measured time to link the same Clang
> executable.
>
> Here is the result. This table contains a commit message, a SVN revision
> number, time to link, size of the resulting Clang binary in each row.
>
>
>
https://docs.google.com/spreadsheets/d/1QlQZ51VQUna6gX0qXj19v_ZEII5vGaHrUhBKq-Z_9uc/edit?usp=sharing
>
> As you can see, the linker is faster than the original. A few key findings
> are listed below:
>
>  - The linker tend to be gradually getting slower over time in general.
>  - There are a few commits that increased link time by a few percent or
> more.
>  - However, we also committed a few performance improvement changes to
> offset them.
>
> The commit that increased link time most (+15%) is r251212: Add support
> for merging string from SHF_STRINGS sections. This does not come as a
> surprise since string merging is a computationally heavy operation.
>
> The commit that decreased link time most (-29%) is r250315: ELF2: Merge
> .{text,rodata,data,bss}.* sections. This is because the commit
> significantly decreased number of sections in the output.
>
> The following commits are also noteworthy.
>
>  - r257017: ELF: Improve performance of string table construction (-14.0%)
>  - r251332: Optimize StringTableBuilder (-7.8%)
>
> Ideally, we should have a perf bot to run a benchmark for every commit, so
> that we do not accidentally commit changes that hurt performance.
>
> Overall, I'm satisfied with the result. It found no commit that
increases
> link time for no justifiable reason.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160115/627fc2fe/attachment.html>

Because the graph hides some part of the table?

On Fri, Jan 15, 2016 at 2:54 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> Cool - thanks for sharing!
>
> Perhaps you could add a graph of the times in another sheet/beside the
> table?
>
> On Fri, Jan 15, 2016 at 2:42 PM, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Speed is one of the most important goals for the new ELF linker. I ran
a
>> benchmark for every commit to LLD/ELF that was made after the last
October,
>> and I'd like to share the result. The motivation is to see if the
linker is
>> getting slower as we add more features, and if so, which commits are
heavy
>> ones in terms of performance. I measured time to link the same Clang
>> executable.
>>
>> Here is the result. This table contains a commit message, a SVN
revision
>> number, time to link, size of the resulting Clang binary in each row.
>>
>>
>>
https://docs.google.com/spreadsheets/d/1QlQZ51VQUna6gX0qXj19v_ZEII5vGaHrUhBKq-Z_9uc/edit?usp=sharing
>>
>> As you can see, the linker is faster than the original. A few key
>> findings are listed below:
>>
>>  - The linker tend to be gradually getting slower over time in general.
>>  - There are a few commits that increased link time by a few percent or
>> more.
>>  - However, we also committed a few performance improvement changes to
>> offset them.
>>
>> The commit that increased link time most (+15%) is r251212: Add support
>> for merging string from SHF_STRINGS sections. This does not come as a
>> surprise since string merging is a computationally heavy operation.
>>
>> The commit that decreased link time most (-29%) is r250315: ELF2: Merge
>> .{text,rodata,data,bss}.* sections. This is because the commit
>> significantly decreased number of sections in the output.
>>
>> The following commits are also noteworthy.
>>
>>  - r257017: ELF: Improve performance of string table construction
>> (-14.0%)
>>  - r251332: Optimize StringTableBuilder (-7.8%)
>>
>> Ideally, we should have a perf bot to run a benchmark for every commit,
>> so that we do not accidentally commit changes that hurt performance.
>>
>> Overall, I'm satisfied with the result. It found no commit that
>> increases link time for no justifiable reason.
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160115/85bbff8d/attachment.html>

Thanks a lot for doing this.

Yes, a perf bot that would just link clang over and over in a quiet
environment would be awesome.

Cheers,
Rafael


On 15 January 2016 at 17:42, Rui Ueyama via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Speed is one of the most important goals for the new ELF linker. I ran a
> benchmark for every commit to LLD/ELF that was made after the last October,
> and I'd like to share the result. The motivation is to see if the
linker is
> getting slower as we add more features, and if so, which commits are heavy
> ones in terms of performance. I measured time to link the same Clang
> executable.
>
> Here is the result. This table contains a commit message, a SVN revision
> number, time to link, size of the resulting Clang binary in each row.
>
>
https://docs.google.com/spreadsheets/d/1QlQZ51VQUna6gX0qXj19v_ZEII5vGaHrUhBKq-Z_9uc/edit?usp=sharing
>
> As you can see, the linker is faster than the original. A few key findings
> are listed below:
>
>  - The linker tend to be gradually getting slower over time in general.
>  - There are a few commits that increased link time by a few percent or
> more.
>  - However, we also committed a few performance improvement changes to
> offset them.
>
> The commit that increased link time most (+15%) is r251212: Add support for
> merging string from SHF_STRINGS sections. This does not come as a surprise
> since string merging is a computationally heavy operation.
>
> The commit that decreased link time most (-29%) is r250315: ELF2: Merge
> .{text,rodata,data,bss}.* sections. This is because the commit
significantly
> decreased number of sections in the output.
>
> The following commits are also noteworthy.
>
>  - r257017: ELF: Improve performance of string table construction (-14.0%)
>  - r251332: Optimize StringTableBuilder (-7.8%)
>
> Ideally, we should have a perf bot to run a benchmark for every commit, so
> that we do not accidentally commit changes that hurt performance.
>
> Overall, I'm satisfied with the result. It found no commit that
increases
> link time for no justifiable reason.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>