llvm dev - Nov 2014 - [LLVMdev] Using the unused "version" field in the bitcode wrapper (redux)

On Tue, Nov 4, 2014 at 2:57 PM, Chris Lattner <clattner at apple.com>
wrote:
>
> On Nov 4, 2014, at 7:04 AM, Robinson, Paul <
> Paul_Robinson at playstation.sony.com> wrote:
>
> >> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu]
> On Behalf Of Sean Silva
> >>
> >> You haven't established that you really need this. AFAIK
Apple's linker
> >> doesn't need this version information and they have shipped
LTO for a
> >> while now.
> >
> > Does Apple support library/middleware providers shipping bitcode
instead
> > of object code?
>
> No.
>
Are there ever any plans to do so?
(this question also goes out to every other vendor that is shipping an LTO
toolchain or plans to. Chad?)

I'm just trying to figure out how much of a Sony-specific issue this is.
Our customers are very performance-hungry and so we would like to provide
the option to middleware vendors (e.g. a physics library). If there is a
strong general desire in the community to not support this use case, we
will want to factor this into our internal decisions.

-- Sean Silva

>
> -Chris
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141104/e6f98bf9/attachment.html>

On Wed, Nov 5, 2014 at 2:30 PM, Sean Silva <chisophugis at gmail.com>
wrote:
> > Does Apple support library/middleware providers shipping bitcode
instead
>
>> > of object code?
>>
>> No.
>>
>
> Are there ever any plans to do so?
> (this question also goes out to every other vendor that is shipping an LTO
> toolchain or plans to. Chad?)
>
> I'm just trying to figure out how much of a Sony-specific issue this
is.
>
The new Andoid ART compiler compiles Dalvik bytecode in standard APKs to
native code on the phone at install time.  It also has in the source code
a, possibly still experimental, "portable mode" that compiles to LLVM
bitcode instead.

I assume (but don't know) this means this would happen on the application
developer's host machine and then be distributed in the Play Store (or
otherwise) as bitcode.

Which would raise large and definite bitcode versioning problems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141105/21c366b5/attachment.html>

On 5 Nov 2014, at 01:30, Sean Silva <chisophugis at gmail.com> wrote:
> Are there ever any plans to do so?
> (this question also goes out to every other vendor that is shipping an LTO
toolchain or plans to. Chad?)
We (current hat: FreeBSD) have plans to bundle bitcode with packages and allow
microarchitecture-specific optimisations to run at install time (e.g. ship
generic x86-64 bitcode, codegen with the current host's schedules). 
We're also interested in a few other things in this area:

- If a package installs a shared library that is only used by one dependency, we
want to link the bitcode and do LTO on the result.

- If a package has a known vulnerability, we want to be able to automatically
trigger multicompiler-style diversity (recompile with randomised code layouts,
nop insertion to break gadgets, and so on).

- Have the kernel periodically harvest profiling data and, after a program has
run with its bundled binary a few times, recompile based on that information. 
In particular, recent Intel chips let you (relatively) cheaply log every branch
into an in-memory buffer.  Running this periodically should give some
statistical sampling of code flow within a binary, without too much overhead.

Most of this requires other pieces of infrastructure that aren't in place
yet, but being able to tell cheaply whether the bitcode libraries that we have
are compatible with the installed toolchain (or whether we need to install
another one) would be helpful.  It's not essential for us to put this in the
bitcode itself though, as we can add it to package metadata if required.

David

On Tue, Nov 4, 2014 at 9:10 PM, Bruce Hoult <bruce at hoult.org> wrote:
> On Wed, Nov 5, 2014 at 2:30 PM, Sean Silva <chisophugis at gmail.com>
wrote:
>
>> > Does Apple support library/middleware providers shipping bitcode
instead
>>
>>> > of object code?
>>>
>>> No.
>>>
>>
>> Are there ever any plans to do so?
>> (this question also goes out to every other vendor that is shipping an
>> LTO toolchain or plans to. Chad?)
>>
>> I'm just trying to figure out how much of a Sony-specific issue
this is.
>>
>
> The new Andoid ART compiler compiles Dalvik bytecode in standard APKs to
> native code on the phone at install time.  It also has in the source code
> a, possibly still experimental, "portable mode" that compiles to
LLVM
> bitcode instead.
>
This is completely inaccurate. ART's portable mode used LLVM's IRBuilder
to
construct IR and then lower it immediately, as it is an ahead-of-time
compiler that executes on the device itself. It never stores the IR out to
disk. Applications continue to use dex for portable distribution.

>
> I assume (but don't know) this means this would happen on the
application
> developer's host machine and then be distributed in the Play Store (or
> otherwise) as bitcode.
>
> Which would raise large and definite bitcode versioning problems.
>
On the other hand, Android RenderScript does use LLVM bitcode as its
portable storage format. We have bitcode writers stretching back to 2 funky
versions of LLVM 2.9 (really pre-3.0). The default writer for us is based
on 3.2, and we upconvert any pre-3.x bitcode that we have to 3.2 (or
something more modern if we can). There have definitely been issue in the
past (attribute changes, removal/update of opcodes, etc.), but we have
always found a way to adapt. We are definitely aware that any 4.x change
could break our readers/writers, although we remain hopeful that we will
adapt once again. All of our bitcode conversion/translation tools are
available in the public Android open source project. Note that we only need
to deal with the C subset of things (so no exception handling), and we
currently will drop/ignore any metadata that might break things (so
debugging has been a challenge).

Steve

>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141106/bfcf6aa6/attachment.html>