llvm dev - Nov 2013 - [LLVMdev] bit code file incompatibility due to debug info changes

On Mon, Nov 18, 2013 at 10:37 AM, Chandler Carruth <chandlerc at
google.com> wrote:
>
> On Mon, Nov 18, 2013 at 10:35 AM, Eric Christopher <echristo at
gmail.com>
> wrote:
>>
>> >> At a minimum, it seems like we need a version number in the
debug info
>> >> metadata so we can detect this situation and avoid crashing.
>> >
>> >
>> > Or to put it in the terms of the IR: we need to autoupgrade the
debug
>> > info
>> > metadata just like we do intrinsics. With debug info this might
(at the
>> > worst) involve dropping old metadata.
>> >
>>
>> The verifier is probably the best route here. I think if we fail debug
>> info verification we can just strip it and continue.
>
>
> I'm slightly worried about relying on verification based
"autoupgrade"
> (where by upgrade I mean discard old cruft since it doesn't impact
> correctness) due to compile time costs, but if the verification is fast
> enough to be "always on", then cool.
Agreed, it's not really that fast either. The problem with versions is
that it makes small changes that only require a bit of modification to
the testcase to be pretty heinous upgrade paths and a lot of work -
you can see that from the work any of us have had to do when changing
the format. We'd like to keep the testcases and upgrade those as well
because otherwise we're just dropping testcases like mad constantly.
It's a lot of work, but the format has been changing frequently enough
that we'd be on version 1000 or something by now and it wouldn't
really accomplish much. The verifier seems to be the best route for
that. It isn't too fast, but then reading in bitcode files from disk
isn't particularly speedy either.

-eric

On Mon, Nov 18, 2013 at 10:45 AM, Eric Christopher <echristo at
gmail.com>wrote:
> On Mon, Nov 18, 2013 at 10:37 AM, Chandler Carruth <chandlerc at
google.com>
> wrote:
> >
> > On Mon, Nov 18, 2013 at 10:35 AM, Eric Christopher <echristo at
gmail.com>
> > wrote:
> >>
> >> >> At a minimum, it seems like we need a version number in
the debug
> info
> >> >> metadata so we can detect this situation and avoid
crashing.
> >> >
> >> >
> >> > Or to put it in the terms of the IR: we need to autoupgrade
the debug
> >> > info
> >> > metadata just like we do intrinsics. With debug info this
might (at
> the
> >> > worst) involve dropping old metadata.
> >> >
> >>
> >> The verifier is probably the best route here. I think if we fail
debug
> >> info verification we can just strip it and continue.
> >
> >
> > I'm slightly worried about relying on verification based
"autoupgrade"
> > (where by upgrade I mean discard old cruft since it doesn't impact
> > correctness) due to compile time costs, but if the verification is
fast
> > enough to be "always on", then cool.
>
> Agreed, it's not really that fast either. The problem with versions is
> that it makes small changes that only require a bit of modification to
> the testcase to be pretty heinous upgrade paths and a lot of work -
> you can see that from the work any of us have had to do when changing
> the format. We'd like to keep the testcases and upgrade those as well
> because otherwise we're just dropping testcases like mad constantly.
> It's a lot of work, but the format has been changing frequently enough
> that we'd be on version 1000 or something by now and it wouldn't
> really accomplish much. The verifier seems to be the best route for
> that. It isn't too fast, but then reading in bitcode files from disk
> isn't particularly speedy either.

It depends a bit, also, on what kind of guarantees we need to offer. If the
guarantee when reading IR from disk is "will not crash" then
there's
nothing for it but to run full debug info verification.

On the other hand, if we can assume that some specific metadata implies the
correctness of some other metadata, then all we need to do is check a known
debug info version number. If it's the current number, do nothing (assume
the rest of the debug info is up to date and well formed), otherwise do all
the work to find the debug info and dump it (no need to verify it - we just
have to do the work to find it, so we don't go following bad links later on
- that should be as easy as dropping the llvm.dbg.cu named node and
removing all debug intrinsics and the instruction metadata line
references). But this latter scheme isn't robust against arbitrary metadata
(that could, coincidentally, have the right version number and arbitrary
metadata that breaks all our debug info metadata assumptions)

If the latter is sufficient for everyone's needs/principles, great.

- David
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131118/fd48508c/attachment.html>

On Mon, Nov 18, 2013 at 10:55 AM, David Blaikie <dblaikie at gmail.com>
wrote:
> It depends a bit, also, on what kind of guarantees we need to offer. If
> the guarantee when reading IR from disk is "will not crash" then
there's
> nothing for it but to run full debug info verification.
>
> On the other hand, if we can assume that some specific metadata implies
> the correctness of some other metadata, then all we need to do is check a
> known debug info version number. If it's the current number, do nothing
> (assume the rest of the debug info is up to date and well formed),
> otherwise do all the work to find the debug info and dump it (no need to
> verify it - we just have to do the work to find it, so we don't go
> following bad links later on - that should be as easy as dropping the
> llvm.dbg.cu named node and removing all debug intrinsics and the
> instruction metadata line references). But this latter scheme isn't
robust
> against arbitrary metadata (that could, coincidentally, have the right
> version number and arbitrary metadata that breaks all our debug info
> metadata assumptions)
>
> If the latter is sufficient for everyone's needs/principles, great.
>
This makes sense to me, but I see Eric's fundamental concern with upgrading
test cases.


One other possible idea: we could artificially force coarseness on metadata
while things are still iterating rapidly by just having a version number
that rotates every "release". (Even non-open-source-releases could be
handled by letting anyone, any time they need to rev it, update what the
current version is and update every test case to reference that version.)
If the version is nicely factored, it should at least be an obvious diff if
still huge.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131118/4c20cf71/attachment.html>