Neil Henning via llvm-dev
2016-Apr-21 09:18 UTC
[llvm-dev] Lazily Loaded Modules and Linker::LinkOnlyNeeded
Hey all, For LinkModules, /*dest*/ is a fully materialized module, /*src*/ is a lazily loaded module. From what I understood, getLinkedToGlobal() is finding the function in /*src*/ that matches some function declaration in /*dest*/, and given that /*src*/ is lazily loaded it could be un-materialized. The functions I need brought in from /*src*//**/ into /*dest*/ are always declarations in /*dest*/. The problem is that (for some reason) the combination of Linker::LinkOnlyNeeded and a function that is not materialized will not copy the function body from /*src*/ into /*dest*/. Cheers, -Neil. On 20/04/16 20:39, Teresa Johnson wrote:> > > On Wed, Apr 20, 2016 at 12:28 PM, Rafael Espíndola > <rafael.espindola at gmail.com <mailto:rafael.espindola at gmail.com>> wrote: > > > I understood from his description that he reversed the > destination and source so that destination is the user code. > I assumed it was not lazy loaded, but that would explain the > question then :) > > Neil: can you clarify? If Teresa is right, why aren't you > materializing the destination module entirely? > > > > I don't think it has ever been tried to use a lazy destination. > Having said that, I don't think isMaterializable should return > true for a declaration. > > > Looking at isMaterializable, I'm now a little confused about Neil's > case - the materializable bit is set to true only when > the MODULE_CODE_FUNCTION indicated that it was !isproto, which means > it should have a definition in that module. So I agree with you that > isMaterializable shouldn't be returning true when the symbol is only > available as a declaration. > > Neil - what case are you trying to handle here? If there is a > materializable definition in the dest module, wouldn't you want to use > that instead of linking one in from the source module? > > Teresa > > > > Even materializing functions from the source module on the fly > isn't supported right now, is it? > > It is. > > Neil, the flag is linked to llvm-link's -only-needed command line > option. At least for simple cases it seems to be working. Given > > declare void @g() > define void @f() { > call void @g() > ret void > } > > and > > define void @g() { > ret void > } > define void @h() { > ret void > } > > > linking with "llvm-link -only-needed test1.bc test2.bc" will bring > in g, but not h. Can you write a testcase showing what you were > expecting it to do but it is not? > > Cheers, > Rafael > > > > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com > <mailto:tejohnson at google.com> | 408-460-2413 >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160421/d037c799/attachment.html>
Neil Henning via llvm-dev
2016-Apr-21 10:40 UTC
[llvm-dev] Lazily Loaded Modules and Linker::LinkOnlyNeeded
So (post morning-coffee) - I did some more digging. It seems that it will correctly bring over a function from /*src*/ that is explicitly called in /*dest*/, but it won't bring over any functions that are called by a /*src*/ function from another /*src */function. I may have dug too deep in the rabbit hole when blaming lazily loaded modules for the fault - is it perhaps as simple as LinkOnlyNeeded does not work when importing a function that calls another function? -Neil. On 21/04/16 10:18, Neil Henning via llvm-dev wrote:> Hey all, > > For LinkModules, /*dest*/ is a fully materialized module, /*src*/ is a > lazily loaded module. > > From what I understood, getLinkedToGlobal() is finding the function in > /*src*/ that matches some function declaration in /*dest*/, and given > that /*src*/ is lazily loaded it could be un-materialized. > > The functions I need brought in from /*src*/ into /*dest*/ are always > declarations in /*dest*/. The problem is that (for some reason) the > combination of Linker::LinkOnlyNeeded and a function that is not > materialized will not copy the function body from /*src*/ into /*dest*/. > > Cheers, > -Neil. > > On 20/04/16 20:39, Teresa Johnson wrote: >> >> >> On Wed, Apr 20, 2016 at 12:28 PM, Rafael Espíndola >> <rafael.espindola at gmail.com> wrote: >> >> >> I understood from his description that he reversed the >> destination and source so that destination is the user code. >> I assumed it was not lazy loaded, but that would explain the >> question then :) >> >> Neil: can you clarify? If Teresa is right, why aren't you >> materializing the destination module entirely? >> >> >> >> I don't think it has ever been tried to use a lazy destination. >> Having said that, I don't think isMaterializable should return >> true for a declaration. >> >> >> Looking at isMaterializable, I'm now a little confused about Neil's >> case - the materializable bit is set to true only when >> the MODULE_CODE_FUNCTION indicated that it was !isproto, which means >> it should have a definition in that module. So I agree with you that >> isMaterializable shouldn't be returning true when the symbol is only >> available as a declaration. >> >> Neil - what case are you trying to handle here? If there is a >> materializable definition in the dest module, wouldn't you want to >> use that instead of linking one in from the source module? >> >> Teresa >> >> >> >> Even materializing functions from the source module on the >> fly isn't supported right now, is it? >> >> It is. >> >> Neil, the flag is linked to llvm-link's -only-needed command line >> option. At least for simple cases it seems to be working. Given >> >> declare void @g() >> define void @f() { >> call void @g() >> ret void >> } >> >> and >> >> define void @g() { >> ret void >> } >> define void @h() { >> ret void >> } >> >> >> linking with "llvm-link -only-needed test1.bc test2.bc" will >> bring in g, but not h. Can you write a testcase showing what you >> were expecting it to do but it is not? >> >> Cheers, >> Rafael >> >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson at google.com >> <mailto:tejohnson at google.com> | 408-460-2413 >> > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160421/af79c6fa/attachment.html>
Teresa Johnson via llvm-dev
2016-Apr-21 12:57 UTC
[llvm-dev] Lazily Loaded Modules and Linker::LinkOnlyNeeded
On Thu, Apr 21, 2016 at 2:18 AM, Neil Henning <llvm at duskborn.com> wrote:> Hey all, > > For LinkModules, *dest* is a fully materialized module, *src* is a lazily > loaded module. >It's not clear to me then why your change to materialize the dest copy is helping then.> > From what I understood, getLinkedToGlobal() is finding the function in > *src* that matches some function declaration in *dest*, and given that > *src* is lazily loaded it could be un-materialized. >It does the reverse (finds function in dest that matches function in src): /// Given a global in the source module, return the global in the /// destination module that is being linked to, if any. GlobalValue *getLinkedToGlobal(const GlobalValue *SrcGV) { The functions I need brought in from *src* into *dest* are always> declarations in *dest*. The problem is that (for some reason) the > combination of Linker::LinkOnlyNeeded and a function that is not > materialized will not copy the function body from *src* into *dest*. >>From your subsequent email: > So (post morning-coffee) - I did some more digging. > > It seems that it will correctly bring over a function from *src* that isexplicitly called in *dest*, but it won't bring over any functions> that are called by a *src* function from another *src *function. > I may have dug too deep in the rabbit hole when blaming lazily loadedmodules for the fault - is it perhaps as simple as> LinkOnlyNeeded does not work when importing a function that calls anotherfunction? So in that case linkIfNeeded will return false since you have LinkOnlyNeeded and there is no declaration in the dest module (it isn't needed by the dest module). I'm not completely sure what should happen here with LinkOnlyNeeded. There is handling in the IRMover to lazily add references to link that are encountered when copying over bodies, however, this only happens when the reference value is LinkOnce (since it could be discarded in the source module). It may be that in the case LinkOnlyNeeded is designed for this functionality wasn't needed - it was specific to a special situation in linking CUDA, see http://reviews.llvm.org/D12459. I'm guessing to handle your case you would want to change ModuleLinker::addLazyFor() to be more aggressive in the LinkOnlyNeeded case. Teresa> > Cheers, > -Neil. > > > On 20/04/16 20:39, Teresa Johnson wrote: > > > > On Wed, Apr 20, 2016 at 12:28 PM, Rafael Espíndola < > <rafael.espindola at gmail.com>rafael.espindola at gmail.com> wrote: > >> >>> I understood from his description that he reversed the destination and >>> source so that destination is the user code. >>> I assumed it was not lazy loaded, but that would explain the question >>> then :) >>> >>> Neil: can you clarify? If Teresa is right, why aren't you materializing >>> the destination module entirely? >>> >>> >> >> I don't think it has ever been tried to use a lazy destination. Having >> said that, I don't think isMaterializable should return true for a >> declaration. >> > > Looking at isMaterializable, I'm now a little confused about Neil's case - > the materializable bit is set to true only when the MODULE_CODE_FUNCTION > indicated that it was !isproto, which means it should have a definition in > that module. So I agree with you that isMaterializable shouldn't be > returning true when the symbol is only available as a declaration. > > Neil - what case are you trying to handle here? If there is a > materializable definition in the dest module, wouldn't you want to use that > instead of linking one in from the source module? > > Teresa > > >> >>> Even materializing functions from the source module on the fly isn't >>> supported right now, is it? >>> >>> >> It is. >> >> Neil, the flag is linked to llvm-link's -only-needed command line option. >> At least for simple cases it seems to be working. Given >> >> declare void @g() >> define void @f() { >> call void @g() >> ret void >> } >> >> and >> >> define void @g() { >> ret void >> } >> define void @h() { >> ret void >> } >> >> >> linking with "llvm-link -only-needed test1.bc test2.bc" will bring in g, >> but not h. Can you write a testcase showing what you were expecting it to >> do but it is not? >> >> Cheers, >> Rafael >> >> > > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com | > 408-460-2413 > > >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160421/f33a162f/attachment.html>
Rafael Espíndola via llvm-dev
2016-Apr-21 14:29 UTC
[llvm-dev] Lazily Loaded Modules and Linker::LinkOnlyNeeded
On 21 April 2016 at 06:40, Neil Henning via llvm-dev < llvm-dev at lists.llvm.org> wrote:> So (post morning-coffee) - I did some more digging. > > It seems that it will correctly bring over a function from *src* that is > explicitly called in *dest*, but it won't bring over any functions that > are called by a *src* function from another *src *function. >I can reproduce that. Taking a look. Cheers, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160421/dcd94adc/attachment.html>
Rafael Espíndola via llvm-dev
2016-Apr-21 15:06 UTC
[llvm-dev] Lazily Loaded Modules and Linker::LinkOnlyNeeded
> > So in that case linkIfNeeded will return false since you have LinkOnlyNeeded and there is no declaration in the dest module (it isn't needed by the dest module). I'm not completely sure what should happen here with LinkOnlyNeeded. There is handling in the IRMover to lazily add references to link that are encountered when copying over bodies, however, this only happens when the reference value is LinkOnce (since it could be discarded in the source module). It may be that in the case LinkOnlyNeeded is designed for this functionality wasn't needed - it was specific to a special situation in linking CUDA, see http://reviews.llvm.org/D12459. I'm guessing to handle your case you would want to change ModuleLinker::addLazyFor() to be more aggressive in the LinkOnlyNeeded case.Yes, that was the problem. Fixed in r266995. Neil, can you give that a try? Cheers, Rafael