Marc via llvm-dev
2020-Mar-09 21:25 UTC
[llvm-dev] llvm-link has no --allow-multiple-definition
Hi Eli, I know the page, and the only mentioning about LTO or link time optimization I found is the option PassManagerBuilder::EP_FullLinkTimeOptimizationLast - and nobody knows how this works (and asked about it on this list too before with no reply), and the author who wrote that part in the llvm code at Redhat did not reply to a question how to actually use it. And there is no example code or any code using this option anywhere on the Internet I could find. And loading such a plugin via -Xclang -load -Xclang plugin.so with EP_FullLinkTimeOptimizationLast doesnt work. So if you can shed any light about how to write a plugin that allows running a pass within ld - I would be very greatful! (and a large community of people fuzzing code :) Regards, Marc On 09.03.20 20:24, Eli Friedman wrote:> Despite the name, llvm-link isn't really a proper linker. If you need symbols to get resolved exactly the same way an ELF linker would resolve them, I would recommend using lld. > > It's possible to write a pass plugin that hooks into the LTO pipeline. http://llvm.org/docs/WritingAnLLVMPass.html describes this. > > -Eli > >> -----Original Message----- >> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Marc via llvm- >> dev >> Sent: Sunday, March 8, 2020 10:01 AM >> To: llvm-dev <llvm-dev at lists.llvm.org> >> Subject: [EXT] [llvm-dev] llvm-link has no --allow-multiple-definition >> >> Hello, >> >> I am developing an LTO instrumentation of programs for fuzzing (for >> afl++) and I run into the following problem: >> >> a lot of open source program have multiple definitons of the same >> functions. In normal compile mode this is not a problem, however with >> LTO this only works with the linker option: >> -Wl,--allow-multiple-definition >> >> Now to be able to instrument (it is an "opt" pass") at link time I need >> to merge all bitcode files together, and this works great - unless I run >> into this error when I link them to llvm-link: >> >> error: Linking globals named 'process_arg': symbol multiply defined! >> >> (this is for the bogofilter tools, but happens with other open source >> program too). >> >> My issue is that neither is --allow-multiple-definition an llvm-link >> command line option, nor can I find an equivalent from the --help output. >> >> is there a plan to add this (IMHO) important feature? Thank you! >> >> Regards, >> Marc >> >> -- >> Marc Heuse >> www.mh-sec.de >> >> PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573 >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Marc Heuse Mobil: +49 177 9611560 Fax: +49 30 37309726 www.mh-sec.de Marc Heuse - IT-Security Consulting Winsstr. 68 10405 Berlin Ust.-Ident.-Nr.: DE244222388 PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573
Eli Friedman via llvm-dev
2020-Mar-09 22:41 UTC
[llvm-dev] llvm-link has no --allow-multiple-definition
If you want to load a pass plugin into lld, you need to write something like " -Wl,-mllvm=-load=plugin.so". Granted, that doesn't actually work on trunk at the moment; posted https://reviews.llvm.org/D75879 . Once you actually have the plugin loaded into the linker, EP_FullLinkTimeOptimizationLast should work. -Eli> -----Original Message----- > From: Marc <mh at mh-sec.de> > Sent: Monday, March 9, 2020 2:26 PM > To: Eli Friedman <efriedma at quicinc.com> > Cc: llvm-dev <llvm-dev at lists.llvm.org> > Subject: [EXT] Re: [llvm-dev] llvm-link has no --allow-multiple-definition > > Hi Eli, > > I know the page, and the only mentioning about LTO or link time > optimization I found is the option > PassManagerBuilder::EP_FullLinkTimeOptimizationLast - and nobody knows > how this works (and asked about it on this list too before with no > reply), and the author who wrote that part in the llvm code at Redhat > did not reply to a question how to actually use it. And there is no > example code or any code using this option anywhere on the Internet I > could find. > > And loading such a plugin via -Xclang -load -Xclang plugin.so with > EP_FullLinkTimeOptimizationLast doesnt work. > > So if you can shed any light about how to write a plugin that allows > running a pass within ld - I would be very greatful! (and a large > community of people fuzzing code :) > > Regards, > Marc > > > On 09.03.20 20:24, Eli Friedman wrote: > > Despite the name, llvm-link isn't really a proper linker. If you need symbols to > get resolved exactly the same way an ELF linker would resolve them, I would > recommend using lld. > > > > It's possible to write a pass plugin that hooks into the LTO pipeline. > http://llvm.org/docs/WritingAnLLVMPass.html describes this. > > > > -Eli > > > >> -----Original Message----- > >> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Marc via > llvm- > >> dev > >> Sent: Sunday, March 8, 2020 10:01 AM > >> To: llvm-dev <llvm-dev at lists.llvm.org> > >> Subject: [EXT] [llvm-dev] llvm-link has no --allow-multiple-definition > >> > >> Hello, > >> > >> I am developing an LTO instrumentation of programs for fuzzing (for > >> afl++) and I run into the following problem: > >> > >> a lot of open source program have multiple definitons of the same > >> functions. In normal compile mode this is not a problem, however with > >> LTO this only works with the linker option: > >> -Wl,--allow-multiple-definition > >> > >> Now to be able to instrument (it is an "opt" pass") at link time I need > >> to merge all bitcode files together, and this works great - unless I run > >> into this error when I link them to llvm-link: > >> > >> error: Linking globals named 'process_arg': symbol multiply defined! > >> > >> (this is for the bogofilter tools, but happens with other open source > >> program too). > >> > >> My issue is that neither is --allow-multiple-definition an llvm-link > >> command line option, nor can I find an equivalent from the --help output. > >> > >> is there a plan to add this (IMHO) important feature? Thank you! > >> > >> Regards, > >> Marc > >> > >> -- > >> Marc Heuse > >> www.mh-sec.de > >> > >> PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573 > >> _______________________________________________ > >> LLVM Developers mailing list > >> llvm-dev at lists.llvm.org > >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > -- > Marc Heuse > Mobil: +49 177 9611560 > Fax: +49 30 37309726 > www.mh-sec.de > > Marc Heuse - IT-Security Consulting > Winsstr. 68 > 10405 Berlin > > Ust.-Ident.-Nr.: DE244222388 > PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573
Marc via llvm-dev
2020-Mar-10 07:00 UTC
[llvm-dev] llvm-link has no --allow-multiple-definition
HI Eli, that sounds like the -mllvm option would be a good a mention in the WritingAnLLVMPass document ;-) Two further questions for this and I am keeping this on the list so other people have pointers to this if they struggle with the same thing: 1. Does this require to use lld? (I am sure it is, just to clarify, for the very low chance that it would also work with gnu ld and the llvm gold plugin performce this magic) 1a. Follow up question if this required lld - does lld has feature parity (and same parameter names) with e.g. gcc-9 ld? 2. When you say it does not work with trunk at the moment - did it work previously? e.g. with the llvm 9 release? Or did it never work so far and is in the progress of being fixed? Thanks a lot! Regards, Marc On 09.03.20 23:41, Eli Friedman wrote:> If you want to load a pass plugin into lld, you need to write something like " -Wl,-mllvm=-load=plugin.so". Granted, that doesn't actually work on trunk at the moment; posted https://reviews.llvm.org/D75879 . > > Once you actually have the plugin loaded into the linker, EP_FullLinkTimeOptimizationLast should work. > > -Eli > >> -----Original Message----- >> From: Marc <mh at mh-sec.de> >> Sent: Monday, March 9, 2020 2:26 PM >> To: Eli Friedman <efriedma at quicinc.com> >> Cc: llvm-dev <llvm-dev at lists.llvm.org> >> Subject: [EXT] Re: [llvm-dev] llvm-link has no --allow-multiple-definition >> >> Hi Eli, >> >> I know the page, and the only mentioning about LTO or link time >> optimization I found is the option >> PassManagerBuilder::EP_FullLinkTimeOptimizationLast - and nobody knows >> how this works (and asked about it on this list too before with no >> reply), and the author who wrote that part in the llvm code at Redhat >> did not reply to a question how to actually use it. And there is no >> example code or any code using this option anywhere on the Internet I >> could find. >> >> And loading such a plugin via -Xclang -load -Xclang plugin.so with >> EP_FullLinkTimeOptimizationLast doesnt work. >> >> So if you can shed any light about how to write a plugin that allows >> running a pass within ld - I would be very greatful! (and a large >> community of people fuzzing code :) >> >> Regards, >> Marc >> >> >> On 09.03.20 20:24, Eli Friedman wrote: >>> Despite the name, llvm-link isn't really a proper linker. If you need symbols to >> get resolved exactly the same way an ELF linker would resolve them, I would >> recommend using lld. >>> >>> It's possible to write a pass plugin that hooks into the LTO pipeline. >> http://llvm.org/docs/WritingAnLLVMPass.html describes this. >>> >>> -Eli >>> >>>> -----Original Message----- >>>> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Marc via >> llvm- >>>> dev >>>> Sent: Sunday, March 8, 2020 10:01 AM >>>> To: llvm-dev <llvm-dev at lists.llvm.org> >>>> Subject: [EXT] [llvm-dev] llvm-link has no --allow-multiple-definition >>>> >>>> Hello, >>>> >>>> I am developing an LTO instrumentation of programs for fuzzing (for >>>> afl++) and I run into the following problem: >>>> >>>> a lot of open source program have multiple definitons of the same >>>> functions. In normal compile mode this is not a problem, however with >>>> LTO this only works with the linker option: >>>> -Wl,--allow-multiple-definition >>>> >>>> Now to be able to instrument (it is an "opt" pass") at link time I need >>>> to merge all bitcode files together, and this works great - unless I run >>>> into this error when I link them to llvm-link: >>>> >>>> error: Linking globals named 'process_arg': symbol multiply defined! >>>> >>>> (this is for the bogofilter tools, but happens with other open source >>>> program too). >>>> >>>> My issue is that neither is --allow-multiple-definition an llvm-link >>>> command line option, nor can I find an equivalent from the --help output. >>>> >>>> is there a plan to add this (IMHO) important feature? Thank you! >>>> >>>> Regards, >>>> Marc >>>> >>>> -- >>>> Marc Heuse >>>> www.mh-sec.de >>>> >>>> PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573 >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> llvm-dev at lists.llvm.org >>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> -- >> Marc Heuse >> Mobil: +49 177 9611560 >> Fax: +49 30 37309726 >> www.mh-sec.de >> >> Marc Heuse - IT-Security Consulting >> Winsstr. 68 >> 10405 Berlin >> >> Ust.-Ident.-Nr.: DE244222388 >> PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573-- Marc Heuse Mobil: +49 177 9611560 Fax: +49 30 37309726 www.mh-sec.de Marc Heuse - IT-Security Consulting Winsstr. 68 10405 Berlin Ust.-Ident.-Nr.: DE244222388 PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573