Anna Thomas via llvm-dev
2016-Dec-23 20:42 UTC
[llvm-dev] ThinLTO promotion is ending up with "invalid" IR before IR-Linking
We had the same problem with verification when merging/linking 2 modules: a
compile unit is not listed in named metadata (llvm.dbg.cu),
We worked around this problem by using the DebugInfoFinder processModule
function, which identifies all the compile units in the new module, by visiting
all the subprogram scopes and the already available compile units. However, this
succeeds verification only if the unnamed/missing compile units have children
sub programs. This is not a strong guarantee.
Anna
On Dec 23, 2016, at 2:33 PM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
I just hit another issue: before of ODR type uniquing on bitcode loading, we can
end-up with a metadata graph that include metadata from the destination module,
including a DICompileUnit.
Unfortunately this compile-unit is not listed in llvm.dbg.cu and this does not
pass the verifier.
CC a few folks more aware of debug info that I am: is there a plan to make
llvm.dbg.cu disappear?
—
Mehdi
On Dec 23, 2016, at 11:04 AM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
On Dec 23, 2016, at 9:34 AM, Teresa Johnson <tejohnson at
google.com<mailto:tejohnson at google.com>> wrote:
On Thu, Dec 22, 2016 at 8:55 PM, Mehdi Amini <mehdi.amini at
apple.com<mailto:mehdi.amini at apple.com>> wrote:
Hey,
As I’m playing with Metadata lazy-loading, I added a verifier right before
running the IRLinker in FunctionImport.cpp, and it does not pass (on current
trunk) in a few cases. One that I looked at ended up with aliases pointing to
available_externally functions for instance.
How do these look after IRLinking (in the dest module)? I looked at the logic in
renameModuleForThinLTO and all the conversions to available_externally are
predicated on it being in the provided GlobalsToImport set. But in
FunctionImport.cpp selectCallee() we specifically prevent importing of aliases
that would result in the aliasee becoming available_externally. Presumably the
resulting IRLinked dest module looks legit, otherwise we would have later
verifier failures.
So the source module is:
@weakalias = weak alias void (...), bitcast (void ()* @globalfunc1 to void
(...)*)
define void @globalfunc1() #0 {
entry:
ret void
}
But we turn globalfunc1 into available_externally in renameModuleForThinLTO(),
which make the alias invalid.
We don’t import the IR so the destination module is OK.
36
I’m hesitant about breaking the IR verifier like that before calling the
IRLinker. The alternative I can see now would be:
- to perform any handling that requires breaking the verifier as part of the
IRLinker process,
- or to perform the verifier-breaking changes that we do as a pre-process step
to the IRLinker as a post-process step on the linked module.
Wouldn't this second option result in verification errors on the resulting
dest module?
Not sure why? Since currently the destination module is valid, the post-process
should be OK as well?
—
Mehdi
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161223/4f0a9220/attachment.html>
Mehdi Amini via llvm-dev
2016-Dec-23 20:54 UTC
[llvm-dev] ThinLTO promotion is ending up with "invalid" IR before IR-Linking
Hi Anna, Can you clarify if you observed this post-linking? — Mehdi> On Dec 23, 2016, at 12:42 PM, Anna Thomas <anna at azul.com> wrote: > > We had the same problem with verification when merging/linking 2 modules: a compile unit is not listed in named metadata (llvm.dbg.cu), > > We worked around this problem by using the DebugInfoFinder processModule function, which identifies all the compile units in the new module, by visiting all the subprogram scopes and the already available compile units. However, this succeeds verification only if the unnamed/missing compile units have children sub programs. This is not a strong guarantee. > > > Anna > > >> On Dec 23, 2016, at 2:33 PM, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> I just hit another issue: before of ODR type uniquing on bitcode loading, we can end-up with a metadata graph that include metadata from the destination module, including a DICompileUnit. >> Unfortunately this compile-unit is not listed in llvm.dbg.cu and this does not pass the verifier. >> >> CC a few folks more aware of debug info that I am: is there a plan to make llvm.dbg.cu disappear? >> >> — >> Mehdi >> >>> On Dec 23, 2016, at 11:04 AM, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> >>>> On Dec 23, 2016, at 9:34 AM, Teresa Johnson <tejohnson at google.com <mailto:tejohnson at google.com>> wrote: >>>> >>>> >>>> >>>> On Thu, Dec 22, 2016 at 8:55 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote: >>>> Hey, >>>> >>>> As I’m playing with Metadata lazy-loading, I added a verifier right before running the IRLinker in FunctionImport.cpp, and it does not pass (on current trunk) in a few cases. One that I looked at ended up with aliases pointing to available_externally functions for instance. >>>> >>>> How do these look after IRLinking (in the dest module)? I looked at the logic in renameModuleForThinLTO and all the conversions to available_externally are predicated on it being in the provided GlobalsToImport set. But in FunctionImport.cpp selectCallee() we specifically prevent importing of aliases that would result in the aliasee becoming available_externally. Presumably the resulting IRLinked dest module looks legit, otherwise we would have later verifier failures. >>> >>> So the source module is: >>> >>> @weakalias = weak alias void (...), bitcast (void ()* @globalfunc1 to void (...)*) >>> define void @globalfunc1() #0 { >>> entry: >>> ret void >>> } >>> >>> But we turn globalfunc1 into available_externally in renameModuleForThinLTO(), which make the alias invalid. >>> >>> We don’t import the IR so the destination module is OK. >>> 36 >>> >>> >>>> >>>> >>>> I’m hesitant about breaking the IR verifier like that before calling the IRLinker. The alternative I can see now would be: >>>> - to perform any handling that requires breaking the verifier as part of the IRLinker process, >>>> - or to perform the verifier-breaking changes that we do as a pre-process step to the IRLinker as a post-process step on the linked module. >>>> >>>> Wouldn't this second option result in verification errors on the resulting dest module? >>> >>> Not sure why? Since currently the destination module is valid, the post-process should be OK as well? >>> >>> — >>> Mehdi >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161223/9b659d3c/attachment.html>
Anna Thomas via llvm-dev
2017-Jan-03 16:29 UTC
[llvm-dev] ThinLTO promotion is ending up with "invalid" IR before IR-Linking
Hi Mehdi
We observed this verification failure after our (downstream)
'mini-linker' links together two modules, not from the main llvm linker.
This mini-linker is at our front end and the newly generated module is passed to
the llvm optimizer.
In our downstream case, we ‘know’ about the characteristics of the modules
generated in our front-end, so we can enforce the guarantee “A compile unit has
at least one child” at the mini-linker phase. This avoids the verification
failure around the named DICompileUnit metadata. I thought about avoiding
verification of the debug info metadata (verifyModule has a flag for this),
until a point where we can enforce correct debug metadata. However, correct
complete verification at various phases is important, so this partial
verification idea was the last resort.
Thanks,
Anna
On Dec 23, 2016, at 3:54 PM, Mehdi Amini <mehdi.amini at
apple.com<mailto:mehdi.amini at apple.com>> wrote:
Hi Anna,
Can you clarify if you observed this post-linking?
—
Mehdi
On Dec 23, 2016, at 12:42 PM, Anna Thomas <anna at azul.com<mailto:anna at
azul.com>> wrote:
We had the same problem with verification when merging/linking 2 modules: a
compile unit is not listed in named metadata (llvm.dbg.cu),
We worked around this problem by using the DebugInfoFinder processModule
function, which identifies all the compile units in the new module, by visiting
all the subprogram scopes and the already available compile units. However, this
succeeds verification only if the unnamed/missing compile units have children
sub programs. This is not a strong guarantee.
Anna
On Dec 23, 2016, at 2:33 PM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
I just hit another issue: before of ODR type uniquing on bitcode loading, we can
end-up with a metadata graph that include metadata from the destination module,
including a DICompileUnit.
Unfortunately this compile-unit is not listed in llvm.dbg.cu and this does not
pass the verifier.
CC a few folks more aware of debug info that I am: is there a plan to make
llvm.dbg.cu disappear?
—
Mehdi
On Dec 23, 2016, at 11:04 AM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
On Dec 23, 2016, at 9:34 AM, Teresa Johnson <tejohnson at
google.com<mailto:tejohnson at google.com>> wrote:
On Thu, Dec 22, 2016 at 8:55 PM, Mehdi Amini <mehdi.amini at
apple.com<mailto:mehdi.amini at apple.com>> wrote:
Hey,
As I’m playing with Metadata lazy-loading, I added a verifier right before
running the IRLinker in FunctionImport.cpp, and it does not pass (on current
trunk) in a few cases. One that I looked at ended up with aliases pointing to
available_externally functions for instance.
How do these look after IRLinking (in the dest module)? I looked at the logic in
renameModuleForThinLTO and all the conversions to available_externally are
predicated on it being in the provided GlobalsToImport set. But in
FunctionImport.cpp selectCallee() we specifically prevent importing of aliases
that would result in the aliasee becoming available_externally. Presumably the
resulting IRLinked dest module looks legit, otherwise we would have later
verifier failures.
So the source module is:
@weakalias = weak alias void (...), bitcast (void ()* @globalfunc1 to void
(...)*)
define void @globalfunc1() #0 {
entry:
ret void
}
But we turn globalfunc1 into available_externally in renameModuleForThinLTO(),
which make the alias invalid.
We don’t import the IR so the destination module is OK.
36
I’m hesitant about breaking the IR verifier like that before calling the
IRLinker. The alternative I can see now would be:
- to perform any handling that requires breaking the verifier as part of the
IRLinker process,
- or to perform the verifier-breaking changes that we do as a pre-process step
to the IRLinker as a post-process step on the linked module.
Wouldn't this second option result in verification errors on the resulting
dest module?
Not sure why? Since currently the destination module is valid, the post-process
should be OK as well?
—
Mehdi
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170103/40c5183b/attachment.html>