llvm dev - Nov 2011 - [LLVMdev] Problems with lazy linking change

r143524 changed ModuleLinker such that when linking a Module B into a Module A,
any internal/private/... functions in B are only linked into A if there is a use
of them in A.  This is problematic for my usage of module linking, where I'm
basically linking a few modules together (some of which include internal
functions) and then generating additional code that may use some of the internal
functions--so I'd like everything to be linked in at the start, so that all
of those functions are available.  I don't want to not mark those functions
as internal, since if they're not used, I'd like them to be eliminated
when I'm done with compilation.  With this change, those functions are now
no longer available.

If there's a different way I should be doing this (i.e. if I'm doing the
wrong thing in the above), I'd be interested to hear a suggestion. 
Alternatively, as a potential fix, I've attached a patch that adds an option
to LinkModule() and LinkInModules() that enables/disables lazy linking.

Thanks,
-matt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvm.patch
Type: application/octet-stream
Size: 5861 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111104/b7d1e82e/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: clang.patch
Type: application/octet-stream
Size: 588 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111104/b7d1e82e/attachment-0001.obj>

On Fri, Nov 4, 2011 at 10:46 AM, Matt Pharr <matt.pharr at gmail.com>
wrote:
> r143524 changed ModuleLinker such that when linking a Module B into a
Module A, any internal/private/... functions in B are only linked into A if
there is a use of them in A.  This is problematic for my usage of module
linking, where I'm basically linking a few modules together (some of which
include internal functions) and then generating additional code that may use
some of the internal functions--so I'd like everything to be linked in at
the start, so that all of those functions are available.  I don't want to
not mark those functions as internal, since if they're not used, I'd
like them to be eliminated when I'm done with compilation.  With this
change, those functions are now no longer available.
>
> If there's a different way I should be doing this (i.e. if I'm
doing the wrong thing in the above), I'd be interested to hear a suggestion.
One way to do something like this is to mark the relevant functions
external, then use the -internalize pass once everything is linked
together.  That should be much less sensitive to how exactly linking
works.

-Eli

On Nov 4, 2011, at 10:55 AM, Eli Friedman wrote:
> On Fri, Nov 4, 2011 at 10:46 AM, Matt Pharr <matt.pharr at gmail.com>
wrote:
>> r143524 changed ModuleLinker such that when linking a Module B into a
Module A, any internal/private/... functions in B are only linked into A if
there is a use of them in A.  This is problematic for my usage of module
linking, where I'm basically linking a few modules together (some of which
include internal functions) and then generating additional code that may use
some of the internal functions--so I'd like everything to be linked in at
the start, so that all of those functions are available.  I don't want to
not mark those functions as internal, since if they're not used, I'd
like them to be eliminated when I'm done with compilation.  With this
change, those functions are now no longer available.
>> 
>> If there's a different way I should be doing this (i.e. if I'm
doing the wrong thing in the above), I'd be interested to hear a suggestion.
> 
> One way to do something like this is to mark the relevant functions
> external, then use the -internalize pass once everything is linked
> together.  That should be much less sensitive to how exactly linking
> works.
Ah, thanks for the pointer to the internalize pass.  I ended up implementing the
trivial variant of that that takes a given set of function names to mark as
internal, rather than a set of functions to mark as external like it does.
>From an API generality/flexibility standpoint, I do think that it's a a
step backwards to no longer have a Linker method available that will just
absolutely link everything in the two given modules together.
Thanks,
-matt

On 11/04/2011 01:46 PM, Matt Pharr wrote:
> r143524 changed ModuleLinker such that when linking a Module B into a
> Module A, any internal/private/... functions in B are only linked
> into A if there is a use of them in A.  This is problematic for my
> usage of module linking, where I'm basically linking a few modules
> together (some of which include internal functions) and then
> generating additional code that may use some of the internal
> functions--so I'd like everything to be linked in at the start, so
> that all of those functions are available.  I don't want to not mark
> those functions as internal, since if they're not used, I'd like
them
> to be eliminated when I'm done with compilation.  With this change,
> those functions are now no longer available.
> 
This is an interesting regression that also affects LTO. I have reported
llvm.org/pr11367 for tracking it.

Thanks for finding it!
> Thanks, -matt
Cheers,
Rafael