llvm dev - Sep 2021 - [LLD] Relocation overflows and .nv_fatbin

Sorry I wasn't clear. I am not talking about moving .nv_fatbin in its
entirety. Although we are also doing it with linker script INSERT AFTER .bss.
I was referring to re-arranging input sections within the .nv_fatbin output
section.

So, we have in .text*
relocation into .nv_fatbin input section from foo3.o

In output .nv_fatbin without any changes we will have
foo1.o input section (some cuda code)
foo2.o input section (some cuda code)
foo3.o input section (has relocation in to)

With this layout we get relocation overflow.

if we shuffle things
foo3.o (has relocation in to from .text*)
foo1.o (some cuda code)
foo2.o (some cuda code)

It shortens the distance from src to dst fo relocation, and all other cuda
sections can grow.

Hopefully, this clarifies things.
Alex





________________________________
From: Fangrui Song <maskray at google.com>
Sent: Wednesday, September 8, 2021 12:01 PM
To: Alexander Yermolovich <ayermolo at fb.com>
Cc: llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] [LLD] Relocation overflows and .nv_fatbin

On 2021-09-08, Alexander Yermolovich via llvm-dev wrote:
>Hello
>
>I am seeing relocation overflows from .text section in to .nv_fatbin. The
whole thing, nv_fatbin, is a bit of a black box, but there does appear to be
only one. We have a downstream patch in LLD, that moves .nv_fatbin section(s)
that have relocations in to to the "top". Looking around at what's
in .nv_fartbin the rest of the code should be bunch of cuda stuff. So, in theory
that can grow, and we shouldn't get any more relocation overflows. At least
due to the size of .nv_fatbin.
>
>I was wondering if there is a better way of doing it. Maybe with a linker
script? I investigated it, and that answer seems to be no, but I am not an
expert in linker scripts.
>
>Thank You
>Alex
I implemented INSERT [AFTER|BEFORE] for orphan sections in
https://reviews.llvm.org/D74375
You may consider moving .nv* and __nv* sections after .bss

But linker synthesized etext/_etext may be in a weird position.
To fix that, use the OVERWRITE_SECTIONS feature I added for LLD 13.0.0

OVERWRITE_SECTIONS {
   .tdata : { etext = .; _etext = .; *(.tdata) }
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210908/169ed845/attachment.html>

On Wed, Sep 8, 2021 at 12:44 PM Alexander Yermolovich <ayermolo at fb.com>
wrote:
>
> Sorry I wasn't clear. I am not talking about moving .nv_fatbin in its
entirety. Although we are also doing it with linker script INSERT AFTER .bss.
> I was referring to re-arranging input sections within the .nv_fatbin output
section.
>
> So, we have in .text*
> relocation into .nv_fatbin input section from foo3.o
>
> In output .nv_fatbin without any changes we will have
> foo1.o input section (some cuda code)
> foo2.o input section (some cuda code)
> foo3.o input section (has relocation in to)
>
> With this layout we get relocation overflow.
>
> if we shuffle things
> foo3.o (has relocation in to from .text*)
> foo1.o (some cuda code)
> foo2.o (some cuda code)
>
> It shortens the distance from src to dst fo relocation, and all other cuda
sections can grow.
>
> Hopefully, this clarifies things.
> Alex
>
Reordering output sections is much more effective.

The ELF port has an option --symbol-ordering-file which reorders input
sections within one output section, but I doubt you can find something
as a marker.
The option was originally conceived to improve performance (by
optimizing for instruction cache/iTLB locality), not to mitigate
relocation overflows.
(macOS ld64 has a similar but more powerful -order_file which can
specify input filenames.)

Reordering input sections automatically has some small value but it
would break phase ordering and cause more maintenance burden. So I
very strongly object to that.
>
>
>
>
> ________________________________
> From: Fangrui Song <maskray at google.com>
> Sent: Wednesday, September 8, 2021 12:01 PM
> To: Alexander Yermolovich <ayermolo at fb.com>
> Cc: llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] [LLD] Relocation overflows and .nv_fatbin
>
> On 2021-09-08, Alexander Yermolovich via llvm-dev wrote:
> >Hello
> >
> >I am seeing relocation overflows from .text section in to .nv_fatbin.
The whole thing, nv_fatbin, is a bit of a black box, but there does appear to be
only one. We have a downstream patch in LLD, that moves .nv_fatbin section(s)
that have relocations in to to the "top". Looking around at what's
in .nv_fartbin the rest of the code should be bunch of cuda stuff. So, in theory
that can grow, and we shouldn't get any more relocation overflows. At least
due to the size of .nv_fatbin.
> >
> >I was wondering if there is a better way of doing it. Maybe with a
linker script? I investigated it, and that answer seems to be no, but I am not
an expert in linker scripts.
> >
> >Thank You
> >Alex
>
> I implemented INSERT [AFTER|BEFORE] for orphan sections in
https://reviews.llvm.org/D74375
> You may consider moving .nv* and __nv* sections after .bss
>
> But linker synthesized etext/_etext may be in a weird position.
> To fix that, use the OVERWRITE_SECTIONS feature I added for LLD 13.0.0
>
> OVERWRITE_SECTIONS {
>    .tdata : { etext = .; _etext = .; *(.tdata) }
> }


-- 
宋方睿