llvm dev - Jan 2018 - llvm::UpgradeDebugInfo does a full verification

Op 24-1-2018 om 18:26 schreef Adrian Prantl:
> 
> 
>> On Jan 23, 2018, at 11:01 PM, Carlo Kok via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> Hi,
>>
>> I noticed that since recently UpgradeDebugInfo (Which is called for
loading any bitcode from disk/memory) does a full bitcode verification. Is this
on purpose or is this a mistake? Seems to have a fair amount of overhead.
> 
> That is be design, though there are bugs in the LTO pipeline where we run
the Verifier more than once. "Upgrading" debug info means checking for
a malformed metadata graph (for example, produced by older, buggy versions of
LLVM, or buggy frontends) that would crash LLVM. The "upgrade"
consists of stripping the debug info form the module.
Yeah I read that. My problem is that I use these calls to feed llvm my 
generated bitcode (I don't use the llvm apis, I write bitcode and give 
the endresult to llvm); now there's a fairly noticable overhead in 
loading this code into llvm and there doesn't seem to be a "load
bitcode
and skips checks" call I can use instead.

> On Jan 24, 2018, at 10:47 PM, Carlo Kok <ck at remobjects.com> wrote:
> 
> 
> 
> Op 24-1-2018 om 18:26 schreef Adrian Prantl:
>>> On Jan 23, 2018, at 11:01 PM, Carlo Kok via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>> 
>>> Hi,
>>> 
>>> I noticed that since recently UpgradeDebugInfo (Which is called for
loading any bitcode from disk/memory) does a full bitcode verification. Is this
on purpose or is this a mistake? Seems to have a fair amount of overhead.
>> That is be design, though there are bugs in the LTO pipeline where we
run the Verifier more than once. "Upgrading" debug info means checking
for a malformed metadata graph (for example, produced by older, buggy versions
of LLVM, or buggy frontends) that would crash LLVM. The "upgrade"
consists of stripping the debug info form the module.
> 
> Yeah I read that. My problem is that I use these calls to feed llvm my
generated bitcode (I don't use the llvm apis, I write bitcode and give the
endresult to llvm); now there's a fairly noticable overhead in loading this
code into llvm and there doesn't seem to be a "load bitcode and skips
checks" call I can use instead.
For testing purposes, opt and llc provide an option to not run the Verifier, but
want to strongly discourage using this for anything besides writing testcases.
IMHO whenever data is passed to LLVM, LLVM should run the verifier once. Does
the overhead of the verifier show up in the profile? If so, is there a specific
part that is slowing things down? The verifier checks should be mostly O(n) with
the occasional DenseMap lookup. If improving the performance of the Verifier is
an option I would rather like to go down this route.

-- adrian

On January 25, 2018 5:56:15 PM GMT+01:00, Adrian Prantl <aprantl at
apple.com> wrote:
>
>
>> On Jan 24, 2018, at 10:47 PM, Carlo Kok <ck at remobjects.com>
wrote:
>> 
>> 
>> 
>> Op 24-1-2018 om 18:26 schreef Adrian Prantl:
>>>> On Jan 23, 2018, at 11:01 PM, Carlo Kok via llvm-dev
><llvm-dev at lists.llvm.org> wrote:
>> 
>> Yeah I read that. My problem is that I use these calls to feed llvm
>my generated bitcode (I don't use the llvm apis, I write bitcode and
>give the endresult to llvm); now there's a fairly noticable overhead in
>loading this code into llvm and there doesn't seem to be a "load
>bitcode and skips checks" call I can use instead.
>
>For testing purposes, opt and llc provide an option to not run the
>Verifier, but want to strongly discourage using this for anything
>besides writing testcases. IMHO whenever data is passed to LLVM, LLVM
>should run the verifier once. Does the overhead of the verifier show up
>in the profile? If so, is there a specific part that is slowing things
>down? The verifier checks should be mostly O(n) with the occasional
>DenseMap lookup. If improving the performance of the Verifier is an
>option I would rather like to go down this route.
>
I understand the need for this from an external tools POV. I suppose my use case
is way different. My compiler and llvm used match exactly and in debug mode I
verify myself.  I didn't profile it yet as I haven't found out a good
profiler for this just yet. Just noticed quite noticable difference after
upgrading. I'll try to get exact numbers shortly.

-- 
Carlo Kok