Mehdi AMINI via llvm-dev
2019-Jan-31 23:37 UTC
[llvm-dev] [RFC] Support embedding bitcodes in LLD with LTO
On Thu, Jan 31, 2019 at 11:05 AM Rui Ueyama via llvm-dev < llvm-dev at lists.llvm.org> wrote:> That feature is probably too specific to your project. Most projects that > use LTO are using LTO just because it generates better code. Your project > is special as your program itself can also interpret LLVM bitcode, but > that's not the case for most other programs. >I agree this is specific "compared to the usual expected output of as linker", but on the other hand it also has potential for opening cool project that can be built on top of this! If this could be supported in lld without too much trouble (maintenance, code complexity, etc.), why not accepting the patches? Best, -- Mehdi> > I think the option that's closest to the one you are looking for is > `--plugin-opt=emit-llvm`. That option makes lld to make an output file in > the bitcode file format (so lld doesn't do LTO if the option is given and > instead writes a raw bitcode as an output). With that option, I don't think > it's too hard to embed bitcode file to your executable. Run the linker > twice, with and without `--plugin-opt=emit-llvm`, and embed the generated > bitcode file using objcopy. > > Does that work for you? > > On Thu, Jan 31, 2019 at 4:18 AM Josef Eisl <josef.eisl at oracle.com> wrote: > >> Thanks for your response! >> >> On 30/01/2019 20:18, Rui Ueyama wrote: >> > Hi Josef, >> > >> > Let me clarify my understanding. Do you want to keep original bitcode >> > files in the output executable when doing LTO, so that the resulting >> > executable contains both compiled bitcode (which is in native machine >> > instructions) and original bitcode files? >> >> Exactly! Kind of analogous to what `clang -fembed-bitcode -c` does, but >> for executables. >> >> > >> > Did you try embedding bitcode files into existing ELF files using >> > objcopy or linker option `--format=binary`? >> >> Yes, that is the alternative. However, having support in the linker for >> that would require less tweaking of exiting build systems. Adding an >> option to CFLAGS/LDFLAGS would then be sufficient. >> >> > >> > On Mon, Jan 28, 2019 at 6:41 AM Josef Eisl via llvm-dev >> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> > >> > Hi everybody! >> > >> > I'm Josef and I'm working at Oracle Labs on Sulong [1,2], the LLVM >> IR >> > execution engine in GraalVM [3]. In addition to executing bare >> bitcode >> > files, Sulong also accepts ELF files with embedded bitcode sections. >> > Therefore, it would be great if LLD in (Full)LTO mode would support >> > embedding bitcode sections to the resulting object file. Is that >> > something that would be considered useful and worth contributing? >> > >> > Thanks, >> > Josef >> > >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190131/331711a7/attachment.html>
Rui Ueyama via llvm-dev
2019-Jan-31 23:55 UTC
[llvm-dev] [RFC] Support embedding bitcodes in LLD with LTO
We need to set some threshold to avoid feature creeping. As I wrote, lld already has a command line flag to emit a bitcode file instead of a compiled one, and perhaps you could do a lot of things you think interesting with that option. If you are already using the feature and find it less powerful for your purposes, then we can discuss whether adding a new feature is the way to go. But as a maintainer of the tool, I don't think asking whether or not an existing feature would work as an initial response is not unreasonable. On Thu, Jan 31, 2019 at 3:37 PM Mehdi AMINI <joker.eph at gmail.com> wrote:> > > On Thu, Jan 31, 2019 at 11:05 AM Rui Ueyama via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> That feature is probably too specific to your project. Most projects that >> use LTO are using LTO just because it generates better code. Your project >> is special as your program itself can also interpret LLVM bitcode, but >> that's not the case for most other programs. >> > > I agree this is specific "compared to the usual expected output of as > linker", but on the other hand it also has potential for opening cool > project that can be built on top of this! > If this could be supported in lld without too much trouble (maintenance, > code complexity, etc.), why not accepting the patches? > > Best, > > -- > Mehdi > > > > > >> >> I think the option that's closest to the one you are looking for is >> `--plugin-opt=emit-llvm`. That option makes lld to make an output file in >> the bitcode file format (so lld doesn't do LTO if the option is given and >> instead writes a raw bitcode as an output). With that option, I don't think >> it's too hard to embed bitcode file to your executable. Run the linker >> twice, with and without `--plugin-opt=emit-llvm`, and embed the generated >> bitcode file using objcopy. >> >> Does that work for you? >> >> On Thu, Jan 31, 2019 at 4:18 AM Josef Eisl <josef.eisl at oracle.com> wrote: >> >>> Thanks for your response! >>> >>> On 30/01/2019 20:18, Rui Ueyama wrote: >>> > Hi Josef, >>> > >>> > Let me clarify my understanding. Do you want to keep original bitcode >>> > files in the output executable when doing LTO, so that the resulting >>> > executable contains both compiled bitcode (which is in native machine >>> > instructions) and original bitcode files? >>> >>> Exactly! Kind of analogous to what `clang -fembed-bitcode -c` does, but >>> for executables. >>> >>> > >>> > Did you try embedding bitcode files into existing ELF files using >>> > objcopy or linker option `--format=binary`? >>> >>> Yes, that is the alternative. However, having support in the linker for >>> that would require less tweaking of exiting build systems. Adding an >>> option to CFLAGS/LDFLAGS would then be sufficient. >>> >>> > >>> > On Mon, Jan 28, 2019 at 6:41 AM Josef Eisl via llvm-dev >>> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> > >>> > Hi everybody! >>> > >>> > I'm Josef and I'm working at Oracle Labs on Sulong [1,2], the LLVM >>> IR >>> > execution engine in GraalVM [3]. In addition to executing bare >>> bitcode >>> > files, Sulong also accepts ELF files with embedded bitcode >>> sections. >>> > Therefore, it would be great if LLD in (Full)LTO mode would support >>> > embedding bitcode sections to the resulting object file. Is that >>> > something that would be considered useful and worth contributing? >>> > >>> > Thanks, >>> > Josef >>> > >>> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190131/63a23f9f/attachment.html>
Mehdi AMINI via llvm-dev
2019-Feb-01 00:12 UTC
[llvm-dev] [RFC] Support embedding bitcodes in LLD with LTO
On Thu, Jan 31, 2019 at 3:56 PM Rui Ueyama <ruiu at google.com> wrote:> We need to set some threshold to avoid feature creeping. >I am not sure to understand what you mean by this? Is this is some sort of view of the tool as a "product" where you want a close control on the exposed surface or something like this? I tend to see LLVM as an "open platform" more than a closed "product": it means that, with the goal of enabling future innovations, I personally tend to look at proposed new features/patches in light of the cost/burden they add on the community maintaining the project. -- Mehdi> As I wrote, lld already has a command line flag to emit a bitcode file > instead of a compiled one, and perhaps you could do a lot of things you > think interesting with that option. If you are already using the feature > and find it less powerful for your purposes, then we can discuss whether > adding a new feature is the way to go. But as a maintainer of the tool, I > don't think asking whether or not an existing feature would work as an > initial response is not unreasonable. > > On Thu, Jan 31, 2019 at 3:37 PM Mehdi AMINI <joker.eph at gmail.com> wrote: > >> >> >> On Thu, Jan 31, 2019 at 11:05 AM Rui Ueyama via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >>> That feature is probably too specific to your project. Most projects >>> that use LTO are using LTO just because it generates better code. Your >>> project is special as your program itself can also interpret LLVM bitcode, >>> but that's not the case for most other programs. >>> >> >> I agree this is specific "compared to the usual expected output of as >> linker", but on the other hand it also has potential for opening cool >> project that can be built on top of this! >> If this could be supported in lld without too much trouble (maintenance, >> code complexity, etc.), why not accepting the patches? >> >> Best, >> >> -- >> Mehdi >> >> >> >> >> >>> >>> I think the option that's closest to the one you are looking for is >>> `--plugin-opt=emit-llvm`. That option makes lld to make an output file in >>> the bitcode file format (so lld doesn't do LTO if the option is given and >>> instead writes a raw bitcode as an output). With that option, I don't think >>> it's too hard to embed bitcode file to your executable. Run the linker >>> twice, with and without `--plugin-opt=emit-llvm`, and embed the generated >>> bitcode file using objcopy. >>> >>> Does that work for you? >>> >>> On Thu, Jan 31, 2019 at 4:18 AM Josef Eisl <josef.eisl at oracle.com> >>> wrote: >>> >>>> Thanks for your response! >>>> >>>> On 30/01/2019 20:18, Rui Ueyama wrote: >>>> > Hi Josef, >>>> > >>>> > Let me clarify my understanding. Do you want to keep original bitcode >>>> > files in the output executable when doing LTO, so that the resulting >>>> > executable contains both compiled bitcode (which is in native machine >>>> > instructions) and original bitcode files? >>>> >>>> Exactly! Kind of analogous to what `clang -fembed-bitcode -c` does, but >>>> for executables. >>>> >>>> > >>>> > Did you try embedding bitcode files into existing ELF files using >>>> > objcopy or linker option `--format=binary`? >>>> >>>> Yes, that is the alternative. However, having support in the linker for >>>> that would require less tweaking of exiting build systems. Adding an >>>> option to CFLAGS/LDFLAGS would then be sufficient. >>>> >>>> > >>>> > On Mon, Jan 28, 2019 at 6:41 AM Josef Eisl via llvm-dev >>>> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>>> > >>>> > Hi everybody! >>>> > >>>> > I'm Josef and I'm working at Oracle Labs on Sulong [1,2], the >>>> LLVM IR >>>> > execution engine in GraalVM [3]. In addition to executing bare >>>> bitcode >>>> > files, Sulong also accepts ELF files with embedded bitcode >>>> sections. >>>> > Therefore, it would be great if LLD in (Full)LTO mode would >>>> support >>>> > embedding bitcode sections to the resulting object file. Is that >>>> > something that would be considered useful and worth contributing? >>>> > >>>> > Thanks, >>>> > Josef >>>> > >>>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190131/856845c9/attachment.html>