llvm dev - Dec 2017 - [RFC] - Deduplication of debug information in linkers (LLD).

?But could not we for example do split dwarf, but for example do dedup of types
?

I do not mean right now, but in a theory ?


Best regards,
George | Developer | Access Softek, Inc
________________________________
От: David Blaikie <dblaikie at gmail.com>
Отправлено: 16 декабря 2017 г. 22:25
Кому: George Rimar
Копия: Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael Espindola
Тема: Re: [llvm-dev] [RFC] - Deduplication of debug information in linkers
(LLD).



On Sat, Dec 16, 2017 at 4:08 AM George Rimar <grimar at
accesssoftek.com<mailto:grimar at accesssoftek.com>> wrote:
>Wasn't our (lld/ELF's) position on debug info size that we should
focus on providing a great split-dwarf workflow and not try go too far out of
our way to deduplicate >or otherwise reduce debug info size inside LLD? I
recall there being some patches that made linking of large debug binaries like
1.5GB+ clang faster, but we decided to >reject those changes because
split-dwarf was the "right" solution.
>
>Rafael, Rui?
>
>(I even recall Rafael saying at one point that a great split-dwarf workflow
was one of the key things he considered as necessary for him to consider LLD
"done")
>
>-- Sean Silva
I probably not the right person to suggest (still mostly learning here for now,
so would like to be on a fence in general),
but it looks for me that splitting DWARF and deduplicating DWARF is a bit
othogonal things.
It feels for me that there is a room to do both things and have a benefit from
combinatiion ?

The two features/directions don't really compose - if the DWARF is split,
then the linker never sees the DWARF (it's not in the object files), so has
no deduplication to do. (llvm-dwp might see it, so the deduplication can happen
there)


George.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171216/a82b9005/attachment.html>

Or following workflow:

Split dwarf is used to make linker to proccess less, like relocations, right ?

What about (I think I heard this somewhere, not sure idea is mine, it was a year
ago I think).

What about something that combines output to a linker output, so it could do
optimizations

of DWARF data (probably no need to proccess relocations at all, so it would be
fast probably).

And combine everything to a single file.

Some kind of mix. Not sure it makes sence. Just wondering.


Best regards,
George | Developer | Access Softek, Inc
________________________________
От: George Rimar
Отправлено: 16 декабря 2017 г. 22:27
Кому: David Blaikie
Копия: Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael Espindola
Тема: Re: [llvm-dev] [RFC] - Deduplication of debug information in linkers
(LLD).


?But could not we for example do split dwarf, but for example do dedup of types
?

I do not mean right now, but in a theory ?


Best regards,
George | Developer | Access Softek, Inc
________________________________
От: David Blaikie <dblaikie at gmail.com>
Отправлено: 16 декабря 2017 г. 22:25
Кому: George Rimar
Копия: Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael Espindola
Тема: Re: [llvm-dev] [RFC] - Deduplication of debug information in linkers
(LLD).



On Sat, Dec 16, 2017 at 4:08 AM George Rimar <grimar at
accesssoftek.com<mailto:grimar at accesssoftek.com>> wrote:
>Wasn't our (lld/ELF's) position on debug info size that we should
focus on providing a great split-dwarf workflow and not try go too far out of
our way to deduplicate >or otherwise reduce debug info size inside LLD? I
recall there being some patches that made linking of large debug binaries like
1.5GB+ clang faster, but we decided to >reject those changes because
split-dwarf was the "right" solution.
>
>Rafael, Rui?
>
>(I even recall Rafael saying at one point that a great split-dwarf workflow
was one of the key things he considered as necessary for him to consider LLD
"done")
>
>-- Sean Silva
I probably not the right person to suggest (still mostly learning here for now,
so would like to be on a fence in general),
but it looks for me that splitting DWARF and deduplicating DWARF is a bit
othogonal things.
It feels for me that there is a room to do both things and have a benefit from
combinatiion ?

The two features/directions don't really compose - if the DWARF is split,
then the linker never sees the DWARF (it's not in the object files), so has
no deduplication to do. (llvm-dwp might see it, so the deduplication can happen
there)


George.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171216/37f95ba2/attachment.html>

On Sat, Dec 16, 2017 at 11:27 AM George Rimar <grimar at accesssoftek.com>
wrote:
> ​But could not we for example do split dwarf, but for example do dedup of
> types ?
>
Yep, that is already supported - but the type deduplication happens in the
DWP tool/generation, not in the linker. The linker inputs (.o files) don't
contain any of the type information, only the .dwo files contain the type
information, etc, in a Fission build.
> I do not mean right now, but in a theory ?
>
>
> Best regards,
> George | Developer | Access Softek, Inc
> ------------------------------
> *От:* David Blaikie <dblaikie at gmail.com>
> *Отправлено:* 16 декабря 2017 г. 22:25
> *Кому:* George Rimar
> *Копия:* Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael
Espindola
> *Тема:* Re: [llvm-dev] [RFC] - Deduplication of debug information in
> linkers (LLD).
>
>
>
> On Sat, Dec 16, 2017 at 4:08 AM George Rimar <grimar at
accesssoftek.com>
> wrote:
>
>> >Wasn't our (lld/ELF's) position on debug info size that we
should focus
>> on providing a great split-dwarf workflow and not try go too far out of
our
>> way to deduplicate >or otherwise reduce debug info size inside LLD?
I
>> recall there being some patches that made linking of large debug
binaries
>> like 1.5GB+ clang faster, but we decided to >reject those changes
because
>> split-dwarf was the "right" solution.
>> >
>> >Rafael, Rui?
>> >
>> >(I even recall Rafael saying at one point that a great split-dwarf
>> workflow was one of the key things he considered as necessary for him
to
>> consider LLD "done")
>> >
>> >-- Sean Silva
>>
>> I probably not the right person to suggest (still mostly learning here
>> for now, so would like to be on a fence in general),
>> but it looks for me that splitting DWARF and deduplicating DWARF is a
>> bit othogonal things.
>> It feels for me that there is a room to do both things and have a
benefit
>> from combinatiion ?
>>
>
> The two features/directions don't really compose - if the DWARF is
split,
> then the linker never sees the DWARF (it's not in the object files), so
has
> no deduplication to do. (llvm-dwp might see it, so the deduplication can
> happen there)
>
>
>>
>> George.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171217/b1ce2276/attachment.html>

On Sat, Dec 16, 2017 at 11:40 AM George Rimar <grimar at accesssoftek.com>
wrote:
> Or following workflow:
>
> Split dwarf is used to make linker to proccess less, like relocations,
> right ?
>
Partly, though the main motivation as far as I know, was to have to provide
fewer bytes to the linker at all. That's why something like Apple's
scheme
(leave the debug info in the object files, but have the linker ignore them
- then merge the debug info separately in dsymutil) wasn't applicable -
because that still leaves large object files. For a distributed build
system like Google's, with very large binaries, the presence of the bytes,
even if they're ignored/not processed by the linker, was problematic.
> What about (I think I heard this somewhere, not sure idea is mine, it was
> a year ago I think).
>
> What about something that combines output to a linker output, so it could
> do optimizations
>
Not sure I understand what you're suggesting here, sorry :/
> of DWARF data (probably no need to proccess relocations at all, so it
> would be fast probably).
>
> And combine everything to a single file.
>
> Some kind of mix. Not sure it makes sence. Just wondering.
>
>
> Best regards,
> George | Developer | Access Softek, Inc
> ------------------------------
> *От:* George Rimar
> *Отправлено:* 16 декабря 2017 г. 22:27
> *Кому:* David Blaikie
>
> *Копия:* Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael
Espindola
> *Тема:* Re: [llvm-dev] [RFC] - Deduplication of debug information in
> linkers (LLD).
>
> ​But could not we for example do split dwarf, but for example do dedup of
> types ?
>
> I do not mean right now, but in a theory ?
>
>
> Best regards,
> George | Developer | Access Softek, Inc
> ------------------------------
> *От:* David Blaikie <dblaikie at gmail.com>
> *Отправлено:* 16 декабря 2017 г. 22:25
> *Кому:* George Rimar
> *Копия:* Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael
Espindola
> *Тема:* Re: [llvm-dev] [RFC] - Deduplication of debug information in
> linkers (LLD).
>
>
>
> On Sat, Dec 16, 2017 at 4:08 AM George Rimar <grimar at
accesssoftek.com>
> wrote:
>
>> >Wasn't our (lld/ELF's) position on debug info size that we
should focus
>> on providing a great split-dwarf workflow and not try go too far out of
our
>> way to deduplicate >or otherwise reduce debug info size inside LLD?
I
>> recall there being some patches that made linking of large debug
binaries
>> like 1.5GB+ clang faster, but we decided to >reject those changes
because
>> split-dwarf was the "right" solution.
>> >
>> >Rafael, Rui?
>> >
>> >(I even recall Rafael saying at one point that a great split-dwarf
>> workflow was one of the key things he considered as necessary for him
to
>> consider LLD "done")
>> >
>> >-- Sean Silva
>>
>> I probably not the right person to suggest (still mostly learning here
>> for now, so would like to be on a fence in general),
>> but it looks for me that splitting DWARF and deduplicating DWARF is a
>> bit othogonal things.
>> It feels for me that there is a room to do both things and have a
benefit
>> from combinatiion ?
>>
>
> The two features/directions don't really compose - if the DWARF is
split,
> then the linker never sees the DWARF (it's not in the object files), so
has
> no deduplication to do. (llvm-dwp might see it, so the deduplication can
> happen there)
>
>
>>
>> George.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171217/1a4ad215/attachment.html>