llvm dev - Nov 2014 - [LLVMdev] Using the unused "version" field in the bitcode wrapper (redux)

On Mon, Nov 24, 2014 at 7:00 PM, Robinson, Paul <
Paul_Robinson at playstation.sony.com> wrote:
>  It should report that the
> particular new feature is not support. In this case, something like
> "unknown 'g' flag in datalayout".
>
>  Currently that would hit an llvm_unreachable().  How should we
> re-engineer this to provide sensible error recovery in an LTO context,
> while continuing to provide the die-horribly fallback behavior that it
> currently (quite reasonably) provides in all other contexts?
>
That is just a bug and has nothing to do with LTO or compatibility. We
should never hit llvm_unreachable due to a bad input. In release builds
llvm_unreachable turns into nasal demons.

>
>
> And that's just one example
>
Please provide another. The above example is invalid w.r.t. LTO or
compatibility so it is hard to see your point from it.

> of a place that would have to be re-engineered this way, to provide the
> necessary degree of future-proofing across some indeterminate number of
> modules within LLVM that never expected to see this particular problem.
>
>
>
> Again, *any* solution involving the bitcode wrapper is not interesting
> to the open source project since it is not required.
>
>
>
> I fail to see why all that re-engineering effort IS "required" to
support
> the use-cases we've provided, and why future-proofing a large body of
> software that quite honestly was never designed for it and has genuinely
> very few reasons why it should be.
>
>
>
> If we're going to support bitcode as a long-term storage format,
> everything that pulls information out of bitcode MUST be future-proofed.
> Doing it piece by piece is a LOT of engineering effort to make LLVM
> user-friendly in this way; it was never intended to have that degree of
> self-defense.
>
I don't understand what you mean by "future-proofed" in this
context. If
you mean "never crash on bad user input", then your point doesn't
make
sense to me. LLVM is engineered to never crash on bad user input in the
same sense that it is engineered to correctly compile code; neither is
allowed, and if either happens it is a bug.

-- Sean Silva

>   The wrapper provides that as a simple low-cost high-reliability
> mechanism, and allows the LLVM community to continue not having to worry
> about future-proofing.  It adds the necessary layer that provides the
> necessary protection, to those very few vendors who actually need this kind
> of feature.  And it costs *nothing* for anyone else, because nobody else
> will actually have the wrapper.
>
>
>
> If you're asserting that "bitcode as a long-term storage
format" has been
> completely rejected by the open source project, well, that's a
different
> statement, and not one that I have been hearing.
>
>
>
> Sony is definitely moving in the direction of supporting bitcode as a
> long-term format, therefore we DO require future-proofing, and the wrapper
> is the cheapest engineering solution to get that.
>
> --paulr
>
>
>
> *From:* Sean Silva [mailto:chisophugis at gmail.com]
> *Sent:* Monday, November 24, 2014 4:19 PM
> *To:* Rafael Espíndola
> *Cc:* Robinson, Paul; Bruce Hoult; Yung, Douglas; llvmdev at cs.uiuc.edu
> *Subject:* Re: [LLVMdev] Using the unused "version" field in the
bitcode
> wrapper (redux)
>
>
>
>
>
>
>
> On Mon, Nov 24, 2014 at 7:53 AM, Rafael Espíndola <
> rafael.espindola at gmail.com> wrote:
>
> > Right, that version number is used to resolve *ambiguities* in how to
> > interpret some chunk of bitcode.  It is not a generic bitcode version
> > scheme, because most bitcode format changes involve things like adding
> > new operands or opcodes, which are easily identified without needing
> > an explicit version number.
>
> That is what it is used for at the moment. It is is just a number, we
> can increment it as often as we want.
>
> > The scenario I am most concerned about is this:
> >
> > - We as a vendor publish toolchain #12 based on SVN r250000.
> > - During subsequent LLVM development, changes happen (!).
> >   For example, a new key letter 'g' in the Data Layout.  This
is
> >   not a bitcode ambiguity so MODULE_CODE_VERSION is unchanged.
> > - We as a vendor publish toolchain #13 based on SVN r300000.
> > - Some middleware provider publishes libIncrediblyUseful.bc built
> >   using spiffy new toolchain #13.
> > - Some hapless game developer tries to use libIncrediblyUseful.bc
> >   but is still on toolchain #12. This causes an error during some
> >   LTO build phase, of course; the question is, what kind of error
> >   and how does Hapless Game Developer know what to do?
>
> In summary. An old tool reading new bitcode. It should report that the
> particular new feature is not support. In this case, something like
> "unknown 'g' flag in datalayout".
>
> > This does *nothing* for Hapless Game Developer.  HGD wants to see
> > "this bitcode file was generated by a newer version, I don't
understand
> > how to interpret it" because _that's_ the actual problem.
>
> We can probably add a note saying "newer or corrupt BC".
>
>
>
> This makes sense to me. I think it is pretty safe to assume that any
> "invalid bitcode" that an end-user would get their hands on is
just because
> the bitcode is from a newer version.
>
>
>
> -- Sean Silva
>
>
>
>
> > Proposed solution:
> >
> > Whether to emit a bitcode wrapper becomes a target-dependent
predicate.
>
> Again, *any* solution involving the bitcode wrapper is not interesting
> to the open source project since it is not required.
>
> Cheers,
> Rafael
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141124/c3fea6f6/attachment.html>

On Mon, Nov 24, 2014 at 9:41 PM, Sean Silva <chisophugis at gmail.com>
wrote:
> I fail to see why all that re-engineering effort IS "required" to
support
>> the use-cases we've provided, and why future-proofing a large body
of
>> software that quite honestly was never designed for it and has
genuinely
>> very few reasons why it should be.
>>
>>
>>
>> If we're going to support bitcode as a long-term storage format,
>> everything that pulls information out of bitcode MUST be
future-proofed.
>> Doing it piece by piece is a LOT of engineering effort to make LLVM
>> user-friendly in this way; it was never intended to have that degree of
>> self-defense.
>>
>
> I don't understand what you mean by "future-proofed" in this
context. If
> you mean "never crash on bad user input", then your point
doesn't make
> sense to me. LLVM is engineered to never crash on bad user input in the
> same sense that it is engineered to correctly compile code; neither is
> allowed, and if either happens it is a bug.
>
This is true, but we can say from experience that there are a lot of these
kinds of bugs and it will take a lot of effort to fix them all. Personally,
I think this worth doing. We should be using the new diagnostic handler on
the LLVMContext, and most of LLVM should have a way of returning without
exploding.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141125/3df13e0a/attachment.html>

The fundamental problem is that IR has never been treated as "user
input" before.  It has always been an ephemeral format, and if some
component comes along and sees something it doesn't recognize, that is ipso
facto a compiler bug, not a user input error, and it's perfectly okay to
crash on a compiler bug.  Changing that fundamental assumption would be pretty
pervasive.

I will say that the bitcode reader itself is pretty good about identifying
bitcode it doesn't recognize; it's the components deeper inside LLVM
that are more worrisome.
--paulr

From: Reid Kleckner [mailto:rnk at google.com]
Sent: Tuesday, November 25, 2014 11:43 AM
To: Sean Silva
Cc: Robinson, Paul; Bruce Hoult; llvmdev at cs.uiuc.edu; Yung, Douglas
Subject: Re: [LLVMdev] Using the unused "version" field in the bitcode
wrapper (redux)

On Mon, Nov 24, 2014 at 9:41 PM, Sean Silva <chisophugis at
gmail.com<mailto:chisophugis at gmail.com>> wrote:
I fail to see why all that re-engineering effort IS "required" to
support the use-cases we've provided, and why future-proofing a large body
of software that quite honestly was never designed for it and has genuinely very
few reasons why it should be.

If we're going to support bitcode as a long-term storage format, everything
that pulls information out of bitcode MUST be future-proofed.  Doing it piece by
piece is a LOT of engineering effort to make LLVM user-friendly in this way; it
was never intended to have that degree of self-defense.

I don't understand what you mean by "future-proofed" in this
context. If you mean "never crash on bad user input", then your point
doesn't make sense to me. LLVM is engineered to never crash on bad user
input in the same sense that it is engineered to correctly compile code; neither
is allowed, and if either happens it is a bug.

This is true, but we can say from experience that there are a lot of these kinds
of bugs and it will take a lot of effort to fix them all. Personally, I think
this worth doing. We should be using the new diagnostic handler on the
LLVMContext, and most of LLVM should have a way of returning without exploding.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141125/333ad809/attachment.html>