llvm dev - Nov 2014 - [LLVMdev] Using the unused "version" field in the bitcode wrapper (redux)

Hi,

The conversation has drifted slightly, so I wanted to bring it back to the
version field in the bitcode wrapper.

Currently in the toolchain which we ship and support, we use a proprietary
linker. That linker is unable to read bitcode files and we do not have any plans
to enable it to as far as I’m aware. Because of this, we need a way of
identifying the version of a bitcode file without needing to read/understand the
bitcode itself. The bitcode wrapper is perfectly suited for this task as it is
simple to parse without the linker needing to understand bitcode, is already
defined and already includes a version field.

If we could get the compiler to use that version field that is already present,
it would be a simple solution to our problem. So I guess my question is whether
there is any objection to actually using a field which is already allocated for
this kind of information?

Douglas Yung

From: Sean Silva [mailto:chisophugis at gmail.com]
Sent: Friday, October 31, 2014 21:13
To: llvmdev at cs.uiuc.edu
Cc: Yung, Douglas
Subject: Re: Using the unused "version" field in the bitcode wrapper
(redux)



On Fri, Oct 31, 2014 at 6:21 PM, Sean Silva <chisophugis at
gmail.com<mailto:chisophugis at gmail.com>> wrote:
Hi all,

Doug Yung started a discussion earlier
(http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html) about
using the unused "version" field in the bitcode wrapper, and I think
there was some misunderstanding. I'd like to clarify the motivation.

The reason we want to add the version field is to easily identify
"old" bitcode. It is only LLVM version granularity, but that is good
enough for us. The obvious thing is that we offer compatibility, so why do we
need to know the bitcode version? There are really two reasons:

1. In the short term, LLVM does theoretically provide compatibility. However, we
at Sony are under a much stronger commitment to our customers than the open
source project here, so until the test infrastructure is beefed up quite a bit
to improve this confidence in the backwards compatibility promise, the version
field is a quick unobtrusive way for us to behave correctly in the short term.
Beefing up the compatibility testing is a separate discussion that everybody
realizes is a much larger long-term endeavor; we at Sony are glad to help with
that. As we prepare for our first official SDK release with LTO, where our
customers are officially sanctioned to feed bitcode to our tools, a solution is
needed though. I think that existing toolchains with LTO will also benefit from
this in the near-term.

2. In the long term, it *will* break across major releases. Currently I
don't think we have a way to identify this. This is a bit longer-term
perspective, but it will eventually come up and this fixes it.


Also, is there a reason why the bitcode wrapper is Darwin-specific?

Rafael gave me some of the backstory on this. Basically it is to work around
some buggy behavior in the Darwin ar. Adding that on the front of the bitcode
file just to get a version doesn't seem like a very clean thing to do.

Doug, what other alternatives did you guys consider before settling on this?

As for #2 above, the non-universality of the wrapper makes using the wrapper as
a version indicator sort of a non-starter.

Looks like I've taken my second U-turn on this proposal :/


Can we just always use it and simplify the code path?

-- Sean Silva

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141103/85d99854/attachment.html>

On Mon, Nov 3, 2014 at 2:40 PM, Yung, Douglas <
douglas_yung at playstation.sony.com> wrote:
>  Hi,
>
>
>
> The conversation has drifted slightly, so I wanted to bring it back to the
> version field in the bitcode wrapper.
>
>
>
> Currently in the toolchain which we ship and support, we use a proprietary
> linker. That linker is unable to read bitcode files and we do not have any
> plans to enable it to as far as I’m aware. Because of this, we need a way
> of identifying the version of a bitcode file without needing to
> read/understand the bitcode itself.
>
You haven't established that you really need this. AFAIK Apple's linker
doesn't need this version information and they have shipped LTO for a while
now.

> The bitcode wrapper is perfectly suited for this task as it is simple to
> parse without the linker needing to understand bitcode, is already defined
> and already includes a version field.
>
>
>
> If we could get the compiler to use that version field that is already
> present, it would be a simple solution to our problem.
>
What is this "problem"?

> So I guess my question is whether there is any objection to actually using
> a field which is already allocated for this kind of information?
>
That is not what is being discussed. We're discussing the forest and not
the trees; this patch implies a whole lot of stuff.

-- Sean Silva

>
>
> Douglas Yung
>
>
>
> *From:* Sean Silva [mailto:chisophugis at gmail.com]
> *Sent:* Friday, October 31, 2014 21:13
> *To:* llvmdev at cs.uiuc.edu
> *Cc:* Yung, Douglas
> *Subject:* Re: Using the unused "version" field in the bitcode
wrapper
> (redux)
>
>
>
>
>
>
>
> On Fri, Oct 31, 2014 at 6:21 PM, Sean Silva <chisophugis at
gmail.com> wrote:
>
> Hi all,
>
>
>
> Doug Yung started a discussion earlier (
> http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html)
> about using the unused "version" field in the bitcode wrapper,
and I think
> there was some misunderstanding. I'd like to clarify the motivation.
>
>
>
> The reason we want to add the version field is to easily identify
"old"
> bitcode. It is only LLVM version granularity, but that is good enough for
> us. The obvious thing is that we offer compatibility, so why do we need to
> know the bitcode version? There are really two reasons:
>
>
>
> 1. In the short term, LLVM does theoretically provide compatibility.
> However, we at Sony are under a much stronger commitment to our customers
> than the open source project here, so until the test infrastructure is
> beefed up quite a bit to improve this confidence in the backwards
> compatibility promise, the version field is a quick unobtrusive way for us
> to behave correctly in the short term. Beefing up the compatibility testing
> is a separate discussion that everybody realizes is a much larger long-term
> endeavor; we at Sony are glad to help with that. As we prepare for our
> first official SDK release with LTO, where our customers are officially
> sanctioned to feed bitcode to our tools, a solution is needed though. I
> think that existing toolchains with LTO will also benefit from this in the
> near-term.
>
>
>
> 2. In the long term, it *will* break across major releases. Currently I
> don't think we have a way to identify this. This is a bit longer-term
> perspective, but it will eventually come up and this fixes it.
>
>
>
>
>
> Also, is there a reason why the bitcode wrapper is Darwin-specific?
>
>
>
> Rafael gave me some of the backstory on this. Basically it is to work
> around some buggy behavior in the Darwin ar. Adding that on the front of
> the bitcode file just to get a version doesn't seem like a very clean
thing
> to do.
>
>
>
> Doug, what other alternatives did you guys consider before settling on
> this?
>
>
>
> As for #2 above, the non-universality of the wrapper makes using the
> wrapper as a version indicator sort of a non-starter.
>
>
>
> Looks like I've taken my second U-turn on this proposal :/
>
>
>
>
>
>   Can we just always use it and simplify the code path?
>
>
>
> -- Sean Silva
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141103/6d8d860d/attachment.html>

> Currently in the toolchain which we ship and support, we use a proprietary
> linker. That linker is unable to read bitcode files and we do not have any
> plans to enable it to as far as I’m aware. Because of this, we need a way
of
> identifying the version of a bitcode file without needing to
read/understand
> the bitcode itself. The bitcode wrapper is perfectly suited for this task
as
> it is simple to parse without the linker needing to understand bitcode, is
> already defined and already includes a version field.
There are a few issues in this statement:

First: it is very specific to the current state of a specific toolchain.

Second: even in the specific case mentioned, the conclusion that the
version must readable without understanding bitcode seems a non
sequitur. The gold linker doesn't link with llvm and ld64 does so only
indirectly. The entire LLVM specific knowledge is placed in
LLVMgold.so or libLTO.dylib. Any linker that supports LTO with LLVM
will have a component or related tool that knows LLVM and could check
version information and raise the alarm. It must be able to report an
error because two GlobalVariables could not be merged, why is
producing an error on the version harder?

Third: even the conclusion that a version is needed seems a non
sequitur. The current policy gives us the *option* of dropping support
for old bitcode once we get to 4.1. That means that *if* we decide to
drop something, *and* that something cannot be easily identified by
the bitcode structure itself, *then* we will need to add a version
number in 4.0.

There can always be bugs, but we should just test for compatibility
and fix when issues are found.
>
>
> If we could get the compiler to use that version field that is already
> present, it would be a simple solution to our problem. So I guess my
> question is whether there is any objection to actually using a field which
> is already allocated for this kind of information?
As noted, the backwards compatibility guarantees that we offer require
all bitcode formats to be treated equally, so depending of the wrapper
for this is not a good idea.

Cheers,
Rafael

> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu] On Behalf Of Sean Silva
>
> You haven't established that you really need this. AFAIK Apple's
linker
> doesn't need this version information and they have shipped LTO for a 
> while now.
Does Apple support library/middleware providers shipping bitcode instead
of object code?  That's the most nervous-making scenario for compatibility.
LTO by itself needs bitcode only as an ephemeral stage between source and
object; it's supporting bitcode as a long-lived on-disk format that keeps
us awake at night.

I acknowledge the compatibility promise but I've been whacked upside the
head too often by QA over the years to take an unverified promise at
face value.  I would like worked examples and industry experience reports.
--paulr

On 11/04/2014 07:04 AM, Robinson, Paul wrote:
>> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu] On Behalf Of Sean Silva
>>
>> You haven't established that you really need this. AFAIK
Apple's linker
>> doesn't need this version information and they have shipped LTO for
a
>> while now.
> Does Apple support library/middleware providers shipping bitcode instead
> of object code?  That's the most nervous-making scenario for
compatibility.
> LTO by itself needs bitcode only as an ephemeral stage between source and
> object; it's supporting bitcode as a long-lived on-disk format that
keeps
> us awake at night.
>
> I acknowledge the compatibility promise but I've been whacked upside
the
> head too often by QA over the years to take an unverified promise at
> face value.  I would like worked examples and industry experience reports.
> --paulr
I've been bitten by trying to do this sort of thing recently. The 
released LLVM 3.4 and 3.5, and the Xcode 5.x tools, generate debug info 
marked with the module flag:

     !49 = metadata !{i32 2, metadata !"Debug Info Version", i32 1}

The tools included with Xcode 6.x generate debug info marked as:

     !49 = metadata !{i32 2, metadata !"Debug Info Version", i32
600054001}

and will discard debug info marked with any other version.

I had originally hoped that our (Open Dylan) compiler could output 
bitcode that people could compile and link using the standard 
/usr/bin/clang installed with Xcode, but it's looking more like we may 
want to provide users with a build of vanilla LLVM/Clang for this 
platform instead.

Hopefully the debug info improvements currently under way will 
incorporate a compatibility guarantee that Apple-distributed tools will 
stick to as well.

-Peter-

On Tue Nov 04 2014 at 12:43:05 PM Peter S. Housel <housel at acm.org>
wrote:
> On 11/04/2014 07:04 AM, Robinson, Paul wrote:
> >> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu]
> On Behalf Of Sean Silva
> >>
> >> You haven't established that you really need this. AFAIK
Apple's linker
> >> doesn't need this version information and they have shipped
LTO for a
> >> while now.
> > Does Apple support library/middleware providers shipping bitcode
instead
> > of object code?  That's the most nervous-making scenario for
> compatibility.
> > LTO by itself needs bitcode only as an ephemeral stage between source
and
> > object; it's supporting bitcode as a long-lived on-disk format
that keeps
> > us awake at night.
> >
> > I acknowledge the compatibility promise but I've been whacked
upside the
> > head too often by QA over the years to take an unverified promise at
> > face value.  I would like worked examples and industry experience
> reports.
> > --paulr
>
> I've been bitten by trying to do this sort of thing recently. The
> released LLVM 3.4 and 3.5, and the Xcode 5.x tools, generate debug info
> marked with the module flag:
>
>      !49 = metadata !{i32 2, metadata !"Debug Info Version", i32
1}
>
> The tools included with Xcode 6.x generate debug info marked as:
>
>      !49 = metadata !{i32 2, metadata !"Debug Info Version", i32
600054001}
>
>
This is an apple bug. You should report it.

> and will discard debug info marked with any other version.
>
>
This is correct behavior.

> I had originally hoped that our (Open Dylan) compiler could output
> bitcode that people could compile and link using the standard
> /usr/bin/clang installed with Xcode, but it's looking more like we may
> want to provide users with a build of vanilla LLVM/Clang for this
> platform instead.
>
> Hopefully the debug info improvements currently under way will
> incorporate a compatibility guarantee that Apple-distributed tools will
> stick to as well.
>
>
Unlikely for now though I don't speak for Apple.

-eric

> -Peter-
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141104/378ae035/attachment.html>