llvm dev - Nov 2014 - [LLVMdev] Using the unused "version" field in the bitcode wrapper (redux)

Hi all,

Doug Yung started a discussion earlier (
http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html)
about using the unused "version" field in the bitcode wrapper, and I
think
there was some misunderstanding. I'd like to clarify the motivation.

The reason we want to add the version field is to easily identify
"old"
bitcode. It is only LLVM version granularity, but that is good enough for
us. The obvious thing is that we offer compatibility, so why do we need to
know the bitcode version? There are really two reasons:

1. In the short term, LLVM does theoretically provide compatibility.
However, we at Sony are under a much stronger commitment to our customers
than the open source project here, so until the test infrastructure is
beefed up quite a bit to improve this confidence in the backwards
compatibility promise, the version field is a quick unobtrusive way for us
to behave correctly in the short term. Beefing up the compatibility testing
is a separate discussion that everybody realizes is a much larger long-term
endeavor; we at Sony are glad to help with that. As we prepare for our
first official SDK release with LTO, where our customers are officially
sanctioned to feed bitcode to our tools, a solution is needed though. I
think that existing toolchains with LTO will also benefit from this in the
near-term.

2. In the long term, it *will* break across major releases. Currently I
don't think we have a way to identify this. This is a bit longer-term
perspective, but it will eventually come up and this fixes it.


Also, is there a reason why the bitcode wrapper is Darwin-specific? Can we
just always use it and simplify the code path?

-- Sean Silva
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141031/bd0e6d2e/attachment.html>

On Fri, Oct 31, 2014 at 6:21 PM, Sean Silva <chisophugis at gmail.com>
wrote:
> Hi all,
>
> Doug Yung started a discussion earlier (
> http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html)
> about using the unused "version" field in the bitcode wrapper,
and I think
> there was some misunderstanding. I'd like to clarify the motivation.
>
> The reason we want to add the version field is to easily identify
"old"
> bitcode. It is only LLVM version granularity, but that is good enough for
> us. The obvious thing is that we offer compatibility, so why do we need to
> know the bitcode version? There are really two reasons:
>
> 1. In the short term, LLVM does theoretically provide compatibility.
> However, we at Sony are under a much stronger commitment to our customers
> than the open source project here, so until the test infrastructure is
> beefed up quite a bit to improve this confidence in the backwards
> compatibility promise, the version field is a quick unobtrusive way for us
> to behave correctly in the short term. Beefing up the compatibility testing
> is a separate discussion that everybody realizes is a much larger long-term
> endeavor; we at Sony are glad to help with that. As we prepare for our
> first official SDK release with LTO, where our customers are officially
> sanctioned to feed bitcode to our tools, a solution is needed though. I
> think that existing toolchains with LTO will also benefit from this in the
> near-term.
>
> 2. In the long term, it *will* break across major releases. Currently I
> don't think we have a way to identify this. This is a bit longer-term
> perspective, but it will eventually come up and this fixes it.
>
>
> Also, is there a reason why the bitcode wrapper is Darwin-specific?
>
Rafael gave me some of the backstory on this. Basically it is to work
around some buggy behavior in the Darwin ar. Adding that on the front of
the bitcode file just to get a version doesn't seem like a very clean thing
to do.

Doug, what other alternatives did you guys consider before settling on this?

As for #2 above, the non-universality of the wrapper makes using the
wrapper as a version indicator sort of a non-starter.

Looks like I've taken my second U-turn on this proposal :/


> Can we just always use it and simplify the code path?
>
> -- Sean Silva
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141031/acd26e5c/attachment.html>

Hi Sean,
> Rafael gave me some of the backstory on this. Basically it is to work
around some buggy behavior in the Darwin ar. Adding that on the front of the
bitcode file just to get a version doesn't seem
> like a very clean thing to do.
>
> Doug, what other alternatives did you guys consider before settling on
this?
>
> As for #2 above, the non-universality of the wrapper makes using the
wrapper as a version indicator sort of a non-starter.
Using the bitcode wrapper was our main idea as we discovered it fairly early one
when looking for ways to store the version information. Originally we wanted to
propose adding extra fields at the end of the bitcode wrapper, but we found that
we only needed the version to be embedded by the compiler, and the wrapper
already included an unused version field, so we were hoping to use that. We also
liked that the existing bitcode wrapper was documented, implemented and already
worked with most if not all of the existing LLVM tools.

One alternative that was suggested was we could create a version section in the
bitcode that the compiler would then embed the version into. But we did not want
to take that approach because it would have required the linker to parse the
bitcode to just to extract the version. That functionality is not available (to
my knowledge) in our proprietary linker. By using the bitcode wrapper, our
linker is easily able to extract the version and find the bitcode without
needing to understand bitcode at all.

The non-universality of the wrapper could be an issue, but we were going to work
around that by forcing our compiler to always produce the bitcode wrapper for
our target, so in our case that would not be an issue. If versioning of bitcode
files is something that might be useful to other teams, it might make sense to
just apply it on all bitcode files produced by the compiler. In essence, make it
required instead of optional as it currently stands. The bitcode wrapper would
only add 20 bytes to the size of the file and most LLVM tools should be able to
handle them without any changes.

Douglas Yung

From: Sean Silva [mailto:chisophugis at gmail.com]
Sent: Friday, October 31, 2014 21:13
To: llvmdev at cs.uiuc.edu
Cc: Yung, Douglas
Subject: Re: Using the unused "version" field in the bitcode wrapper
(redux)



On Fri, Oct 31, 2014 at 6:21 PM, Sean Silva <chisophugis at
gmail.com<mailto:chisophugis at gmail.com>> wrote:
Hi all,

Doug Yung started a discussion earlier
(http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html) about
using the unused "version" field in the bitcode wrapper, and I think
there was some misunderstanding. I'd like to clarify the motivation.

The reason we want to add the version field is to easily identify
"old" bitcode. It is only LLVM version granularity, but that is good
enough for us. The obvious thing is that we offer compatibility, so why do we
need to know the bitcode version? There are really two reasons:

1. In the short term, LLVM does theoretically provide compatibility. However, we
at Sony are under a much stronger commitment to our customers than the open
source project here, so until the test infrastructure is beefed up quite a bit
to improve this confidence in the backwards compatibility promise, the version
field is a quick unobtrusive way for us to behave correctly in the short term.
Beefing up the compatibility testing is a separate discussion that everybody
realizes is a much larger long-term endeavor; we at Sony are glad to help with
that. As we prepare for our first official SDK release with LTO, where our
customers are officially sanctioned to feed bitcode to our tools, a solution is
needed though. I think that existing toolchains with LTO will also benefit from
this in the near-term.

2. In the long term, it *will* break across major releases. Currently I
don't think we have a way to identify this. This is a bit longer-term
perspective, but it will eventually come up and this fixes it.


Also, is there a reason why the bitcode wrapper is Darwin-specific?

Rafael gave me some of the backstory on this. Basically it is to work around
some buggy behavior in the Darwin ar. Adding that on the front of the bitcode
file just to get a version doesn't seem like a very clean thing to do.

Doug, what other alternatives did you guys consider before settling on this?

As for #2 above, the non-universality of the wrapper makes using the wrapper as
a version indicator sort of a non-starter.

Looks like I've taken my second U-turn on this proposal :/


Can we just always use it and simplify the code path?

-- Sean Silva

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141101/e6ade9ff/attachment.html>

Hi,

The conversation has drifted slightly, so I wanted to bring it back to the
version field in the bitcode wrapper.

Currently in the toolchain which we ship and support, we use a proprietary
linker. That linker is unable to read bitcode files and we do not have any plans
to enable it to as far as I’m aware. Because of this, we need a way of
identifying the version of a bitcode file without needing to read/understand the
bitcode itself. The bitcode wrapper is perfectly suited for this task as it is
simple to parse without the linker needing to understand bitcode, is already
defined and already includes a version field.

If we could get the compiler to use that version field that is already present,
it would be a simple solution to our problem. So I guess my question is whether
there is any objection to actually using a field which is already allocated for
this kind of information?

Douglas Yung

From: Sean Silva [mailto:chisophugis at gmail.com]
Sent: Friday, October 31, 2014 21:13
To: llvmdev at cs.uiuc.edu
Cc: Yung, Douglas
Subject: Re: Using the unused "version" field in the bitcode wrapper
(redux)



On Fri, Oct 31, 2014 at 6:21 PM, Sean Silva <chisophugis at
gmail.com<mailto:chisophugis at gmail.com>> wrote:
Hi all,

Doug Yung started a discussion earlier
(http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html) about
using the unused "version" field in the bitcode wrapper, and I think
there was some misunderstanding. I'd like to clarify the motivation.

The reason we want to add the version field is to easily identify
"old" bitcode. It is only LLVM version granularity, but that is good
enough for us. The obvious thing is that we offer compatibility, so why do we
need to know the bitcode version? There are really two reasons:

1. In the short term, LLVM does theoretically provide compatibility. However, we
at Sony are under a much stronger commitment to our customers than the open
source project here, so until the test infrastructure is beefed up quite a bit
to improve this confidence in the backwards compatibility promise, the version
field is a quick unobtrusive way for us to behave correctly in the short term.
Beefing up the compatibility testing is a separate discussion that everybody
realizes is a much larger long-term endeavor; we at Sony are glad to help with
that. As we prepare for our first official SDK release with LTO, where our
customers are officially sanctioned to feed bitcode to our tools, a solution is
needed though. I think that existing toolchains with LTO will also benefit from
this in the near-term.

2. In the long term, it *will* break across major releases. Currently I
don't think we have a way to identify this. This is a bit longer-term
perspective, but it will eventually come up and this fixes it.


Also, is there a reason why the bitcode wrapper is Darwin-specific?

Rafael gave me some of the backstory on this. Basically it is to work around
some buggy behavior in the Darwin ar. Adding that on the front of the bitcode
file just to get a version doesn't seem like a very clean thing to do.

Doug, what other alternatives did you guys consider before settling on this?

As for #2 above, the non-universality of the wrapper makes using the wrapper as
a version indicator sort of a non-starter.

Looks like I've taken my second U-turn on this proposal :/


Can we just always use it and simplify the code path?

-- Sean Silva

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141103/85d99854/attachment.html>