On 06/04/2015 07:43 AM, Teresa Johnson wrote:> On Wed, Jun 3, 2015 at 1:29 PM, Sean Silva <chisophugis at gmail.com> wrote: >> >> On Mon, Jun 1, 2015 at 6:34 AM, Teresa Johnson <tejohnson at google.com> wrote: >>> On Fri, May 29, 2015 at 6:15 PM, Sean Silva <chisophugis at gmail.com> wrote: >>>> >>>> On Fri, May 29, 2015 at 8:01 AM, Teresa Johnson <tejohnson at google.com> >>>> wrote: >>>>> On Fri, May 29, 2015 at 6:56 AM, Alex Rosenberg <alexr at leftfield.org> >>>>> wrote: >>>>>> My earlier statement about wrapping things in a native object file >>>>>> held >>>>>> in that it is controversial. It appears to be still central to your >>>>>> design. >>>>>> >>>>>> It may help to look at the problem from a different viewpoint: LLVM >>>>>> is >>>>>> not a compiler. It is a framework that can be used to make >>>>>> compiler-like >>>>>> tools. >>>>>> >>>>>> From that view, it no longer makes sense to discuss "the plugin," or >>>>>> gold, or $AR, because there isn't just one of any of those things. >>>>>> ld64 >>>>>> isn't the only outlier linker to consider. We have our own linker at >>>>>> Sony, >>>>>> for example. From this perspective, then it makes more sense to >>>>>> consider >>>>>> replacing the binary utilities with ones that support bitcode, >>>>>> because from >>>>>> a user-perspective, all of the linkers already transparently support >>>>>> bitcode >>>>>> directly today, as do ar, nm, etc. This has been necessary for the >>>>>> regular >>>>>> LTO process. >>>>> Hi Alex, >>>>> >>>>> It's true that the LLVM versions of these tools support bitcode >>>>> transparently, but not all build systems use LLVM versions of these >>>>> tools, particularly build systems that support a variety of compilers, >>>>> or legacy build systems. >>>> >>>> If a build system can do >>>> CC=clang >>>> why wouldn't it be able to do >>>> AR=llvm-ar >>>> ? >>> That assumes that the LLVM tools are all deployed in the build system, >>> and adds a requirement for using clang in this mode that wasn't there >>> before when using clang for -O2. We are trying to make the transition >>> from clang -O2 to clang -O2 + ThinLTO as seamless as possible. >> >> I'd just like the point out that downthread serious suggestions are being >> fielded to use a nonstandard ELF header or nonstandard bits marking the >> header. This "adds a requirement". >> >> At this point at least 1 (the only?) concrete deployment use case (besides >> yours) that has been brought up in the ThinLTO RFC threads is inconvenienced >> by this design decision. This suggests that native object wrapping doesn't >> offer as much seamlessness as it seems. > One big goal is to make it as painless as possible to transition from > plain -O2 to -O2+thinlto. Users of clang who don't already use LTO > have not had to use/deploy llvm versions of all of these tools > (llvm-nm, llvm-objcopy, llvm-ar, llvm-ranlib, etc), or the plugins for > the native versions of these tools, because they weren't dealing with > bitcode files. That is why we are prioritizing the native wrapped > approach for the initial implementation. For users of the gold linker > (which uses the LTOModule interfaces) this should make it much easier > to enable ThinLTO (from my browsing of ld64 source, it looks like > native-wrapped bitcode should be handled already there too due to the > handling being hidden behind the lto_module API). > > TeresaQuick question: Is the word required to support ThinLTO using llvm's native tools orthogonal to that required to supporting non-llvm tools? If not, would it make sense to start with a deployment of entirely LLVM based tools - since there seems to be general interest in that - and then come back to the non-llvm based tools separately? Personally, I see both sides here. I can understand why you want to minimize build configuration changes - they tend to be painful - but I also am reluctant to design a major enhancement to LLVM under the assumption that LLVM's own tools aren't adequate for the purpose. That seems like it would be majorly problematic from the perspective of the project as a whole. (I realize that LLVM's tools could simply extract the bitcode out of the wrapper file, but that seems unnecessarily complex for an *initial* LLVM only solution.) Philip
On Thu, Jun 4, 2015 at 10:11 AM, Philip Reames <listmail at philipreames.com> wrote:> On 06/04/2015 07:43 AM, Teresa Johnson wrote: >> >> On Wed, Jun 3, 2015 at 1:29 PM, Sean Silva <chisophugis at gmail.com> wrote: >>> >>> >>> On Mon, Jun 1, 2015 at 6:34 AM, Teresa Johnson <tejohnson at google.com> >>> wrote: >>>> >>>> On Fri, May 29, 2015 at 6:15 PM, Sean Silva <chisophugis at gmail.com> >>>> wrote: >>>>> >>>>> >>>>> On Fri, May 29, 2015 at 8:01 AM, Teresa Johnson <tejohnson at google.com> >>>>> wrote: >>>>>> >>>>>> On Fri, May 29, 2015 at 6:56 AM, Alex Rosenberg <alexr at leftfield.org> >>>>>> wrote: >>>>>>> >>>>>>> My earlier statement about wrapping things in a native object file >>>>>>> held >>>>>>> in that it is controversial. It appears to be still central to your >>>>>>> design. >>>>>>> >>>>>>> It may help to look at the problem from a different viewpoint: LLVM >>>>>>> is >>>>>>> not a compiler. It is a framework that can be used to make >>>>>>> compiler-like >>>>>>> tools. >>>>>>> >>>>>>> From that view, it no longer makes sense to discuss "the plugin," or >>>>>>> gold, or $AR, because there isn't just one of any of those things. >>>>>>> ld64 >>>>>>> isn't the only outlier linker to consider. We have our own linker at >>>>>>> Sony, >>>>>>> for example. From this perspective, then it makes more sense to >>>>>>> consider >>>>>>> replacing the binary utilities with ones that support bitcode, >>>>>>> because from >>>>>>> a user-perspective, all of the linkers already transparently support >>>>>>> bitcode >>>>>>> directly today, as do ar, nm, etc. This has been necessary for the >>>>>>> regular >>>>>>> LTO process. >>>>>> >>>>>> Hi Alex, >>>>>> >>>>>> It's true that the LLVM versions of these tools support bitcode >>>>>> transparently, but not all build systems use LLVM versions of these >>>>>> tools, particularly build systems that support a variety of compilers, >>>>>> or legacy build systems. >>>>> >>>>> >>>>> If a build system can do >>>>> CC=clang >>>>> why wouldn't it be able to do >>>>> AR=llvm-ar >>>>> ? >>>> >>>> That assumes that the LLVM tools are all deployed in the build system, >>>> and adds a requirement for using clang in this mode that wasn't there >>>> before when using clang for -O2. We are trying to make the transition >>>> from clang -O2 to clang -O2 + ThinLTO as seamless as possible. >>> >>> >>> I'd just like the point out that downthread serious suggestions are being >>> fielded to use a nonstandard ELF header or nonstandard bits marking the >>> header. This "adds a requirement". >>> >>> At this point at least 1 (the only?) concrete deployment use case >>> (besides >>> yours) that has been brought up in the ThinLTO RFC threads is >>> inconvenienced >>> by this design decision. This suggests that native object wrapping >>> doesn't >>> offer as much seamlessness as it seems. >> >> One big goal is to make it as painless as possible to transition from >> plain -O2 to -O2+thinlto. Users of clang who don't already use LTO >> have not had to use/deploy llvm versions of all of these tools >> (llvm-nm, llvm-objcopy, llvm-ar, llvm-ranlib, etc), or the plugins for >> the native versions of these tools, because they weren't dealing with >> bitcode files. That is why we are prioritizing the native wrapped >> approach for the initial implementation. For users of the gold linker >> (which uses the LTOModule interfaces) this should make it much easier >> to enable ThinLTO (from my browsing of ld64 source, it looks like >> native-wrapped bitcode should be handled already there too due to the >> handling being hidden behind the lto_module API). >> >> Teresa > > Quick question: Is the word required to support ThinLTO using llvm's native > tools orthogonal to that required to supporting non-llvm tools? If not, > would it make sense to start with a deployment of entirely LLVM based tools > - since there seems to be general interest in that - and then come back to > the non-llvm based tools separately? > > Personally, I see both sides here. I can understand why you want to > minimize build configuration changes - they tend to be painful - but I also > am reluctant to design a major enhancement to LLVM under the assumption that > LLVM's own tools aren't adequate for the purpose. That seems like it would > be majorly problematic from the perspective of the project as a whole. > > (I realize that LLVM's tools could simply extract the bitcode out of the > wrapper file, but that seems unnecessarily complex for an *initial* LLVM > only solution.)Hi Phillip, You are right, we can hide most of the details of the input format behind an API, and make most of the ThinLTO implementation blissfully unaware of what format was used. Given that, I am working on figuring out what the right API is first. One other thing about the bitcode-only approach is that we need to define how the function index etc will be represented in bitcode, which will presumably require some IR changes. So in order to test the distributed build integration and deploy more quickly, I was thinking of the following proposal: 1) Define API first 2) Define and implement native object format support (specifically ELF first since I need that to test the distributed build integration) 3) While 2 is in progress, propose bitcode extensions to handle the ThinLTO info in bitcode 4) Implement bitcode writing/reading support for ThinLTO sections 5) Implement support for other native-object formats: COFF/MachO Interspersed with 2-5 is the rest of the ThinLTO support that is independent of the format as it uses the API from 1. Thanks, Teresa> > Philip >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413
> > Quick question: Is the word required to support ThinLTO using llvm'snative tools orthogonal to that required to supporting non-llvm tools? If not, would it make sense to start with a deployment of entirely LLVM based tools - since there seems to be general interest in that - and then come back to the non-llvm based tools separately?> > Personally, I see both sides here. I can understand why you want tominimize build configuration changes - they tend to be painful - but I also am reluctant to design a major enhancement to LLVM under the assumption that LLVM's own tools aren't adequate for the purpose. That seems like it would be majorly problematic from the perspective of the project as a whole.> > (I realize that LLVM's tools could simply extract the bitcode out of thewrapper file, but that seems unnecessarily complex for an *initial* LLVM only solution.)>Really late on this thread, but I wanted to second Philip's position: wrapped bitcode can be interesting and we should consider it, but a bitcode only solution should probably come first. The things thin lto seems to need are generally desirable bitcode features: easy to read symbol table, ability to read a single function, etc. Cheers, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150623/d47431c1/attachment.html>
On Tue, Jun 23, 2015 at 9:51 AM, Rafael Espíndola < rafael.espindola at gmail.com> wrote:> > > > > Quick question: Is the word required to support ThinLTO using llvm's > native tools orthogonal to that required to supporting non-llvm tools? If > not, would it make sense to start with a deployment of entirely LLVM based > tools - since there seems to be general interest in that - and then come > back to the non-llvm based tools separately? > > > > Personally, I see both sides here. I can understand why you want to > minimize build configuration changes - they tend to be painful - but I also > am reluctant to design a major enhancement to LLVM under the assumption > that LLVM's own tools aren't adequate for the purpose. That seems like it > would be majorly problematic from the perspective of the project as a whole. > > > > (I realize that LLVM's tools could simply extract the bitcode out of the > wrapper file, but that seems unnecessarily complex for an *initial* LLVM > only solution.) > > > > Really late on this thread, but I wanted to second Philip's position: > wrapped bitcode can be interesting and we should consider it, but a bitcode > only solution should probably come first. > > The things thin lto seems to need are generally desirable bitcode > features: easy to read symbol table, ability to read a single function, etc. >Sorry for the late reply, this came in the midst of some summer vacations last month. I've come up with some data structures and interfaces that are independent of the underlying file format, which should make the implementation in each format easier and the implementation order less critical hopefully. I will send that out later when it is fully spec'd. I've been focusing more on the symbol linkage and renaming specification, which I emailed out to the list a couple days ago, since that is fundamental to the ThinLTO design. Thanks, Teresa Cheers,> Rafael >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150716/893344d1/attachment.html>