llvm dev - Dec 2017 - [RFC] - Deduplication of debug information in linkers (LLD).

On Sat, Dec 16, 2017 at 11:40 AM George Rimar <grimar at accesssoftek.com>
wrote:
> Or following workflow:
>
> Split dwarf is used to make linker to proccess less, like relocations,
> right ?
>
Partly, though the main motivation as far as I know, was to have to provide
fewer bytes to the linker at all. That's why something like Apple's
scheme
(leave the debug info in the object files, but have the linker ignore them
- then merge the debug info separately in dsymutil) wasn't applicable -
because that still leaves large object files. For a distributed build
system like Google's, with very large binaries, the presence of the bytes,
even if they're ignored/not processed by the linker, was problematic.
> What about (I think I heard this somewhere, not sure idea is mine, it was
> a year ago I think).
>
> What about something that combines output to a linker output, so it could
> do optimizations
>
Not sure I understand what you're suggesting here, sorry :/
> of DWARF data (probably no need to proccess relocations at all, so it
> would be fast probably).
>
> And combine everything to a single file.
>
> Some kind of mix. Not sure it makes sence. Just wondering.
>
>
> Best regards,
> George | Developer | Access Softek, Inc
> ------------------------------
> *От:* George Rimar
> *Отправлено:* 16 декабря 2017 г. 22:27
> *Кому:* David Blaikie
>
> *Копия:* Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael
Espindola
> *Тема:* Re: [llvm-dev] [RFC] - Deduplication of debug information in
> linkers (LLD).
>
> ​But could not we for example do split dwarf, but for example do dedup of
> types ?
>
> I do not mean right now, but in a theory ?
>
>
> Best regards,
> George | Developer | Access Softek, Inc
> ------------------------------
> *От:* David Blaikie <dblaikie at gmail.com>
> *Отправлено:* 16 декабря 2017 г. 22:25
> *Кому:* George Rimar
> *Копия:* Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael
Espindola
> *Тема:* Re: [llvm-dev] [RFC] - Deduplication of debug information in
> linkers (LLD).
>
>
>
> On Sat, Dec 16, 2017 at 4:08 AM George Rimar <grimar at
accesssoftek.com>
> wrote:
>
>> >Wasn't our (lld/ELF's) position on debug info size that we
should focus
>> on providing a great split-dwarf workflow and not try go too far out of
our
>> way to deduplicate >or otherwise reduce debug info size inside LLD?
I
>> recall there being some patches that made linking of large debug
binaries
>> like 1.5GB+ clang faster, but we decided to >reject those changes
because
>> split-dwarf was the "right" solution.
>> >
>> >Rafael, Rui?
>> >
>> >(I even recall Rafael saying at one point that a great split-dwarf
>> workflow was one of the key things he considered as necessary for him
to
>> consider LLD "done")
>> >
>> >-- Sean Silva
>>
>> I probably not the right person to suggest (still mostly learning here
>> for now, so would like to be on a fence in general),
>> but it looks for me that splitting DWARF and deduplicating DWARF is a
>> bit othogonal things.
>> It feels for me that there is a room to do both things and have a
benefit
>> from combinatiion ?
>>
>
> The two features/directions don't really compose - if the DWARF is
split,
> then the linker never sees the DWARF (it's not in the object files), so
has
> no deduplication to do. (llvm-dwp might see it, so the deduplication can
> happen there)
>
>
>>
>> George.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171217/1a4ad215/attachment.html>

>Not sure I understand what you're suggesting here, sorry :/
Ah, looks it is that:
"that's why something like Apple's scheme (leave the debug info in
the object files, but have the linker ignore them -
then merge the debug info separately in dsymutil) wasn't applicable".

So instead of splitting debug info into separate files, info could be still
included into linker output.
?But difference with current traditional flow is that linker would not process
debug relocations
(because technically info can be (probably) close/the same as split debug has),
what could (probably)
resolve issues with relocations processing (too large values). Could keep the
single output file, but save time
that linker normally spends on resolving relocations. And allows to do things
like deduplications inside linker
(what can be convinent). Though increases output size (not sure it is still a
problem nowadays).

Have to run away now, sorry.

George.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171217/5e5bd455/attachment.html>

>>Not sure I understand what you're suggesting here, sorry :/
>
>Ah, looks it is that:
>"that's why something like Apple's scheme (leave the debug info
in the object files, but have the linker ignore them -
>then merge the debug info separately in dsymutil) wasn't
applicable".
>
So instead of splitting debug info into separate files, info could be still
included into linker output.
?>But difference with current traditional flow is that linker would not
process debug relocations
>(because technically info can be (probably) close/the same as split debug
has), what could (probably)
>resolve issues with relocations processing (too large values). Could keep
the single output file, but save time
>that linker normally spends on resolving relocations. And allows to do
things like deduplications inside linker
>(what can be convinent). Though increases output size (not sure it is still
a problem nowadays).
>
>Have to run away now, sorry.
>
>George.
Ok. That above looks sound wierd. I think I just need to read more about how all
DWARF things work then.
(particularry about split debug output details and dsymutil, though their logic
looks transparent for me atm).
Or that brief description I mentioned I saw somewhere was dead
from start or it most likely it was just something else/different suggested,
probably with involving
a compiler/debugger side changes too as well. Can only guess now :/

George.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171218/b9a20393/attachment.html>