llvm dev - Feb 2016 - [cfe-dev] [RFC] Embedding Bitcode in Object Files

Hi Peter

It is not currently related because we started the implementation before
Thin-LTO
gets proposed in the community but our "__LLVM, __bitcode" section is
pretty much
the same as ".llvmbc" section. Note ".llvmbc" doesn't
really follow the section
naming convention for MachO objects. I am hoping to unify them during the
upstream
of the implementation.

Thanks

Steven
 
> On Feb 3, 2016, at 10:48 AM, Peter Collingbourne <peter at pcc.me.uk>
wrote:
> 
> Hi Steven,
> 
> Can you please explain how this relates to the existing .llvmbc section
> feature?
> 
> Peter
> 
> On Wed, Feb 03, 2016 at 10:25:32AM -0800, Steven Wu via cfe-dev wrote:
>> Apple has some internal implemenation for embedding bitcode in the
object file
>> that we would like to upstream. It has few changes to clang frontend,
including
>> new clang options, clang driver changes and utilities to embed bitcode
inside
>> object file. We believe upstreaming these implementations will benefit
the
>> people who would like to develop software on Apple platform using open
source
>> LLVM. It also helps the driver compatibility and it aligns with some of
ongoing
>> efforts like Thin-LTO which also has an object wrapper for bitcode.
>> 
>> Embedded Bitcode Design:
>> Embedded Bitcode are designed to enable bitcode distribution without
disturbing
>> normal development flow. When a program is compiled with bitcode, clang
will
>> embed the optimized bitcode in a special section in the object file,
together
>> with the options that is used during the compilation. The object file
will still
>> have the normal TEXT, DATA sections for normal linking. During the
linking,
>> linker will check all the input object files have embedded bitcode and
collect
>> the bitcode into an archive which is embedded in the output. The
archive also
>> contains all the information that is needed to rebuild the linked
binary. All
>> compilation and linking stage can be replayed to generated the final
binary.
>> 
>> There are mainly two parts we would like to upstream first:
>> 1. Clang Driver:
>> Adding -fembed-bitcode option. When this new option is used, it will
split the
>> compilation into 2 stages. The first stage runs the frontend and all
the
>> optimization passes, and the second stage embeds the bitcode from the
first
>> stage then runs the CodeGen passes.  There is also a
-fembed-bitcode-marker
>> option that doesn't split the compilation into 2 stages and it only
puts an 1
>> byte marker into the object file. This is used to speed up the debug
build
>> because bitcode serialization and verification will make
-fembed-bitcode slower
>> especially with -O0 -g. Linker can still check the presence of the
section to
>> provide feedback if any of the object files participated in the linking
is
>> missing bitcode in a full bitcode build.
>> 2. Bitcode Embedding:
>> Several special sections are used by bitcode to mark the presence of
the bitcode
>> in the MachO file.
>> "__LLVM, __bitcode" is used to store the optimized bitcode in
the object file.
>> It can have an 1-byte size as a marker to provide diagnostics in debug
build.
>> "__LLVM, __cmdline" is used to store the clang command-line
options.  There are
>> few options that are not reflected in the bitcode that we would like to
replay in
>> the rebuild. For example, '-O0' option makes us run FastISel
during rebuild.
>> 
>> 
>> Thanks
>> 
>> Steven
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 
> -- 
> Peter

On 3 February 2016 at 14:01, Steven Wu via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Hi Peter
>
> It is not currently related because we started the implementation before
Thin-LTO
> gets proposed in the community but our "__LLVM, __bitcode"
section is pretty much
> the same as ".llvmbc" section. Note ".llvmbc"
doesn't really follow the section
> naming convention for MachO objects. I am hoping to unify them during the
upstream
> of the implementation.
That would be my main request. Seems like a nice feature, but we
should have one implementation of it :-)

BTW, can you explain a bit why you need things like "-O0" recorded? In
case you want to go from bitcode back to object file one file at a
time (no LTO)? Is that orthogonal? That is, should the command line be
included in .bc files too? What is the command line option that is
included, the -cc1 or the driver one?

There was some discussion on the past about which options get run in
clang if given -flto. For example, it seems likely that a more
conservative inlining pass would be a good thing to not remove
opportunities for the link time inlining. What would happen with
"-flto -fembed-bitcode"? Would the bitcode be the same as with just
-flto and the object file less optimized?

Cheers,
Rafael

Steven,

   I would like to echo Rafael's comments.

My general understanding is that given an object file with embedded IR I should
be able to reproduce the same object.
Everything else should be "supporting" that objective... which might
include relevant flags and transformations leading _to_ this IR and _from_ this
IR to the given object code.

Does my understanding matches your overall goal?

Thanks.

Sergei

---
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The
Linux Foundation

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
Rafael
> Espíndola via llvm-dev
> Sent: Thursday, February 04, 2016 4:01 PM
> To: Steven Wu
> Cc: LLVM Developers Mailing List; cfe-dev
> Subject: Re: [llvm-dev] [cfe-dev] [RFC] Embedding Bitcode in Object Files
> 
> On 3 February 2016 at 14:01, Steven Wu via llvm-dev <llvm-
> dev at lists.llvm.org> wrote:
> > Hi Peter
> >
> > It is not currently related because we started the implementation
> > before Thin-LTO gets proposed in the community but our "__LLVM,
> > __bitcode" section is pretty much the same as ".llvmbc"
section. Note
> > ".llvmbc" doesn't really follow the section naming
convention for
> > MachO objects. I am hoping to unify them during the upstream of the
> implementation.
> 
> That would be my main request. Seems like a nice feature, but we should
> have one implementation of it :-)
> 
> BTW, can you explain a bit why you need things like "-O0"
recorded? In case
> you want to go from bitcode back to object file one file at a time (no
LTO)? Is
> that orthogonal? That is, should the command line be included in .bc files
> too? What is the command line option that is included, the -cc1 or the
driver
> one?
> 
> There was some discussion on the past about which options get run in clang
if
> given -flto. For example, it seems likely that a more conservative inlining
pass
> would be a good thing to not remove opportunities for the link time
inlining.
> What would happen with "-flto -fembed-bitcode"? Would the bitcode
be the
> same as with just -flto and the object file less optimized?
> 
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev