llvm dev - Mar 2020 - llvm-link has no --allow-multiple-definition

HI Eli,

that sounds like the -mllvm option would be a good a mention in the
WritingAnLLVMPass document ;-)

Two further questions for this and I am keeping this on the list so
other people have pointers to this if they struggle with the same thing:

1. Does this require to use lld? (I am sure it is, just to clarify, for
the very low chance that it would also work with gnu ld and the llvm
gold plugin performce this magic)

1a. Follow up question if this required lld - does lld has feature
parity (and same parameter names) with e.g. gcc-9 ld?

2. When you say it does not work with trunk at the moment - did it work
previously? e.g. with the llvm 9 release? Or did it never work so far
and is in the progress of being fixed?

Thanks a lot!

Regards,
Marc


On 09.03.20 23:41, Eli Friedman wrote:
> If you want to load a pass plugin into lld, you need to write something
like " -Wl,-mllvm=-load=plugin.so".  Granted, that doesn't
actually work on trunk at the moment; posted https://reviews.llvm.org/D75879 .
> 
> Once you actually have the plugin loaded into the linker,
EP_FullLinkTimeOptimizationLast should work.
> 
> -Eli
> 
>> -----Original Message-----
>> From: Marc <mh at mh-sec.de>
>> Sent: Monday, March 9, 2020 2:26 PM
>> To: Eli Friedman <efriedma at quicinc.com>
>> Cc: llvm-dev <llvm-dev at lists.llvm.org>
>> Subject: [EXT] Re: [llvm-dev] llvm-link has no
--allow-multiple-definition
>>
>> Hi Eli,
>>
>> I know the page, and the only mentioning about LTO or link time
>> optimization I found is the option
>> PassManagerBuilder::EP_FullLinkTimeOptimizationLast - and nobody knows
>> how this works (and asked about it on this list too before with no
>> reply), and the author who wrote that part in the llvm code at Redhat
>> did not reply to a question how to actually use it. And there is no
>> example code or any code using this option anywhere on the Internet I
>> could find.
>>
>> And loading such a plugin via -Xclang -load -Xclang plugin.so with
>> EP_FullLinkTimeOptimizationLast doesnt work.
>>
>> So if you can shed any light about how to write a plugin that allows
>> running a pass within ld - I would be very greatful! (and a large
>> community of people fuzzing code :)
>>
>> Regards,
>> Marc
>>
>>
>> On 09.03.20 20:24, Eli Friedman wrote:
>>> Despite the name, llvm-link isn't really a proper linker.  If
you need symbols to
>> get resolved exactly the same way an ELF linker would resolve them, I
would
>> recommend using lld.
>>>
>>> It's possible to write a pass plugin that hooks into the LTO
pipeline.
>> http://llvm.org/docs/WritingAnLLVMPass.html describes this.
>>>
>>> -Eli
>>>
>>>> -----Original Message-----
>>>> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On
Behalf Of Marc via
>> llvm-
>>>> dev
>>>> Sent: Sunday, March 8, 2020 10:01 AM
>>>> To: llvm-dev <llvm-dev at lists.llvm.org>
>>>> Subject: [EXT] [llvm-dev] llvm-link has no
--allow-multiple-definition
>>>>
>>>> Hello,
>>>>
>>>> I am developing an LTO instrumentation of programs for fuzzing
(for
>>>> afl++) and I run into the following problem:
>>>>
>>>> a lot of open source program have multiple definitons of the
same
>>>> functions. In normal compile mode this is not a problem,
however with
>>>> LTO this only works with the linker option:
>>>>  -Wl,--allow-multiple-definition
>>>>
>>>> Now to be able to instrument (it is an "opt"
pass") at link time I need
>>>> to merge all bitcode files together, and this works great -
unless I run
>>>> into this error when I link them to llvm-link:
>>>>
>>>> error: Linking globals named 'process_arg': symbol
multiply defined!
>>>>
>>>> (this is for the bogofilter tools, but happens with other open
source
>>>> program too).
>>>>
>>>> My issue is that neither is --allow-multiple-definition an
llvm-link
>>>> command line option, nor can I find an equivalent from the
--help output.
>>>>
>>>> is there a plan to add this (IMHO) important feature? Thank
you!
>>>>
>>>> Regards,
>>>> Marc
>>>>
>>>> --
>>>> Marc Heuse
>>>> www.mh-sec.de
>>>>
>>>> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>> --
>> Marc Heuse
>> Mobil: +49 177 9611560
>> Fax: +49 30 37309726
>> www.mh-sec.de
>>
>> Marc Heuse - IT-Security Consulting
>> Winsstr. 68
>> 10405 Berlin
>>
>> Ust.-Ident.-Nr.: DE244222388
>> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
-- 
Marc Heuse
Mobil: +49 177 9611560
Fax: +49 30 37309726
www.mh-sec.de

Marc Heuse - IT-Security Consulting
Winsstr. 68
10405 Berlin

Ust.-Ident.-Nr.: DE244222388
PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573

Probably you could patch the gold plugin to support something similar? I
haven't tried it, but I don't see any reason it couldn't work. 
Should look similar to the lld patch I posted.  gold doesn't support
"-mllvm", but I think you can do the same thing with
"-plugin-opt".

ld.lld is generally compatible with commonly used GNU linker flags.

As far as I know, loading pass plugins into lld or the gold plugin has never
been implemented before, and my patch is the first time anyone has looked at it.
I don't think that many people are using pass plugins in the first place; if
you have to build your own LLVM for whatever reason, it's almost as easy to
just hack the pass manager.

-Eli
> -----Original Message-----
> From: Marc <mh at mh-sec.de>
> Sent: Tuesday, March 10, 2020 12:00 AM
> To: Eli Friedman <efriedma at quicinc.com>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> Subject: [EXT] Re: [llvm-dev] llvm-link has no --allow-multiple-definition
>
> HI Eli,
>
> that sounds like the -mllvm option would be a good a mention in the
> WritingAnLLVMPass document ;-)
>
> Two further questions for this and I am keeping this on the list so
> other people have pointers to this if they struggle with the same thing:
>
> 1. Does this require to use lld? (I am sure it is, just to clarify, for
> the very low chance that it would also work with gnu ld and the llvm
> gold plugin performce this magic)
>
> 1a. Follow up question if this required lld - does lld has feature
> parity (and same parameter names) with e.g. gcc-9 ld?
>
> 2. When you say it does not work with trunk at the moment - did it work
> previously? e.g. with the llvm 9 release? Or did it never work so far
> and is in the progress of being fixed?
>
> Thanks a lot!
>
> Regards,
> Marc
>
>
> On 09.03.20 23:41, Eli Friedman wrote:
> > If you want to load a pass plugin into lld, you need to write
something like " -
> Wl,-mllvm=-load=plugin.so".  Granted, that doesn't actually work
on trunk at the
> moment; posted https://reviews.llvm.org/D75879 .
> >
> > Once you actually have the plugin loaded into the linker,
> EP_FullLinkTimeOptimizationLast should work.
> >
> > -Eli
> >
> >> -----Original Message-----
> >> From: Marc <mh at mh-sec.de>
> >> Sent: Monday, March 9, 2020 2:26 PM
> >> To: Eli Friedman <efriedma at quicinc.com>
> >> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> >> Subject: [EXT] Re: [llvm-dev] llvm-link has no
--allow-multiple-definition
> >>
> >> Hi Eli,
> >>
> >> I know the page, and the only mentioning about LTO or link time
> >> optimization I found is the option
> >> PassManagerBuilder::EP_FullLinkTimeOptimizationLast - and nobody
knows
> >> how this works (and asked about it on this list too before with no
> >> reply), and the author who wrote that part in the llvm code at
Redhat
> >> did not reply to a question how to actually use it. And there is
no
> >> example code or any code using this option anywhere on the
Internet I
> >> could find.
> >>
> >> And loading such a plugin via -Xclang -load -Xclang plugin.so with
> >> EP_FullLinkTimeOptimizationLast doesnt work.
> >>
> >> So if you can shed any light about how to write a plugin that
allows
> >> running a pass within ld - I would be very greatful! (and a large
> >> community of people fuzzing code :)
> >>
> >> Regards,
> >> Marc
> >>
> >>
> >> On 09.03.20 20:24, Eli Friedman wrote:
> >>> Despite the name, llvm-link isn't really a proper linker. 
If you need symbols
> to
> >> get resolved exactly the same way an ELF linker would resolve
them, I would
> >> recommend using lld.
> >>>
> >>> It's possible to write a pass plugin that hooks into the
LTO pipeline.
> >> http://llvm.org/docs/WritingAnLLVMPass.html describes this.
> >>>
> >>> -Eli
> >>>
> >>>> -----Original Message-----
> >>>> From: llvm-dev <llvm-dev-bounces at lists.llvm.org>
On Behalf Of Marc via
> >> llvm-
> >>>> dev
> >>>> Sent: Sunday, March 8, 2020 10:01 AM
> >>>> To: llvm-dev <llvm-dev at lists.llvm.org>
> >>>> Subject: [EXT] [llvm-dev] llvm-link has no
--allow-multiple-definition
> >>>>
> >>>> Hello,
> >>>>
> >>>> I am developing an LTO instrumentation of programs for
fuzzing (for
> >>>> afl++) and I run into the following problem:
> >>>>
> >>>> a lot of open source program have multiple definitons of
the same
> >>>> functions. In normal compile mode this is not a problem,
however with
> >>>> LTO this only works with the linker option:
> >>>>  -Wl,--allow-multiple-definition
> >>>>
> >>>> Now to be able to instrument (it is an "opt"
pass") at link time I need
> >>>> to merge all bitcode files together, and this works great
- unless I run
> >>>> into this error when I link them to llvm-link:
> >>>>
> >>>> error: Linking globals named 'process_arg': symbol
multiply defined!
> >>>>
> >>>> (this is for the bogofilter tools, but happens with other
open source
> >>>> program too).
> >>>>
> >>>> My issue is that neither is --allow-multiple-definition an
llvm-link
> >>>> command line option, nor can I find an equivalent from the
--help output.
> >>>>
> >>>> is there a plan to add this (IMHO) important feature?
Thank you!
> >>>>
> >>>> Regards,
> >>>> Marc
> >>>>
> >>>> --
> >>>> Marc Heuse
> >>>> www.mh-sec.de
> >>>>
> >>>> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
> >>>> _______________________________________________
> >>>> LLVM Developers mailing list
> >>>> llvm-dev at lists.llvm.org
> >>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >> --
> >> Marc Heuse
> >> Mobil: +49 177 9611560
> >> Fax: +49 30 37309726
> >> www.mh-sec.de
> >>
> >> Marc Heuse - IT-Security Consulting
> >> Winsstr. 68
> >> 10405 Berlin
> >>
> >> Ust.-Ident.-Nr.: DE244222388
> >> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
>
> --
> Marc Heuse
> Mobil: +49 177 9611560
> Fax: +49 30 37309726
> www.mh-sec.de
>
> Marc Heuse - IT-Security Consulting
> Winsstr. 68
> 10405 Berlin
>
> Ust.-Ident.-Nr.: DE244222388
> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573

Hi,

On 12/03/2020 01:21, Eli Friedman via llvm-dev wrote:
>  I don't think that many people are using pass plugins in the first
place;
> 
I think that this group is actually not that small (e.g. downstream 
projects, academia, LLVM beginners), but possibly not as active on 
llvm-dev as others.

-Andrzej
> 
>> -----Original Message-----
>> From: Marc <mh at mh-sec.de>
>> Sent: Tuesday, March 10, 2020 12:00 AM
>> To: Eli Friedman <efriedma at quicinc.com>
>> Cc: llvm-dev <llvm-dev at lists.llvm.org>
>> Subject: [EXT] Re: [llvm-dev] llvm-link has no
--allow-multiple-definition
>>
>> HI Eli,
>>
>> that sounds like the -mllvm option would be a good a mention in the
>> WritingAnLLVMPass document ;-)
>>
>> Two further questions for this and I am keeping this on the list so
>> other people have pointers to this if they struggle with the same
thing:
>>
>> 1. Does this require to use lld? (I am sure it is, just to clarify, for
>> the very low chance that it would also work with gnu ld and the llvm
>> gold plugin performce this magic)
>>
>> 1a. Follow up question if this required lld - does lld has feature
>> parity (and same parameter names) with e.g. gcc-9 ld?
>>
>> 2. When you say it does not work with trunk at the moment - did it work
>> previously? e.g. with the llvm 9 release? Or did it never work so far
>> and is in the progress of being fixed?
>>
>> Thanks a lot!
>>
>> Regards,
>> Marc
>>
>>
>> On 09.03.20 23:41, Eli Friedman wrote:
>>> If you want to load a pass plugin into lld, you need to write
something like " -
>> Wl,-mllvm=-load=plugin.so".  Granted, that doesn't actually
work on trunk at the
>> moment; posted https://reviews.llvm.org/D75879 .
>>>
>>> Once you actually have the plugin loaded into the linker,
>> EP_FullLinkTimeOptimizationLast should work.
>>>
>>> -Eli
>>>
>>>> -----Original Message-----
>>>> From: Marc <mh at mh-sec.de>
>>>> Sent: Monday, March 9, 2020 2:26 PM
>>>> To: Eli Friedman <efriedma at quicinc.com>
>>>> Cc: llvm-dev <llvm-dev at lists.llvm.org>
>>>> Subject: [EXT] Re: [llvm-dev] llvm-link has no
--allow-multiple-definition
>>>>
>>>> Hi Eli,
>>>>
>>>> I know the page, and the only mentioning about LTO or link time
>>>> optimization I found is the option
>>>> PassManagerBuilder::EP_FullLinkTimeOptimizationLast - and
nobody knows
>>>> how this works (and asked about it on this list too before with
no
>>>> reply), and the author who wrote that part in the llvm code at
Redhat
>>>> did not reply to a question how to actually use it. And there
is no
>>>> example code or any code using this option anywhere on the
Internet I
>>>> could find.
>>>>
>>>> And loading such a plugin via -Xclang -load -Xclang plugin.so
with
>>>> EP_FullLinkTimeOptimizationLast doesnt work.
>>>>
>>>> So if you can shed any light about how to write a plugin that
allows
>>>> running a pass within ld - I would be very greatful! (and a
large
>>>> community of people fuzzing code :)
>>>>
>>>> Regards,
>>>> Marc
>>>>
>>>>
>>>> On 09.03.20 20:24, Eli Friedman wrote:
>>>>> Despite the name, llvm-link isn't really a proper
linker.  If you need symbols
>> to
>>>> get resolved exactly the same way an ELF linker would resolve
them, I would
>>>> recommend using lld.
>>>>>
>>>>> It's possible to write a pass plugin that hooks into
the LTO pipeline.
>>>> http://llvm.org/docs/WritingAnLLVMPass.html describes this.
>>>>>
>>>>> -Eli
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: llvm-dev <llvm-dev-bounces at
lists.llvm.org> On Behalf Of Marc via
>>>> llvm-
>>>>>> dev
>>>>>> Sent: Sunday, March 8, 2020 10:01 AM
>>>>>> To: llvm-dev <llvm-dev at lists.llvm.org>
>>>>>> Subject: [EXT] [llvm-dev] llvm-link has no
--allow-multiple-definition
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am developing an LTO instrumentation of programs for
fuzzing (for
>>>>>> afl++) and I run into the following problem:
>>>>>>
>>>>>> a lot of open source program have multiple definitons
of the same
>>>>>> functions. In normal compile mode this is not a
problem, however with
>>>>>> LTO this only works with the linker option:
>>>>>>   -Wl,--allow-multiple-definition
>>>>>>
>>>>>> Now to be able to instrument (it is an "opt"
pass") at link time I need
>>>>>> to merge all bitcode files together, and this works
great - unless I run
>>>>>> into this error when I link them to llvm-link:
>>>>>>
>>>>>> error: Linking globals named 'process_arg':
symbol multiply defined!
>>>>>>
>>>>>> (this is for the bogofilter tools, but happens with
other open source
>>>>>> program too).
>>>>>>
>>>>>> My issue is that neither is --allow-multiple-definition
an llvm-link
>>>>>> command line option, nor can I find an equivalent from
the --help output.
>>>>>>
>>>>>> is there a plan to add this (IMHO) important feature?
Thank you!
>>>>>>
>>>>>> Regards,
>>>>>> Marc
>>>>>>
>>>>>> --
>>>>>> Marc Heuse
>>>>>> www.mh-sec.de
>>>>>>
>>>>>> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>> --
>>>> Marc Heuse
>>>> Mobil: +49 177 9611560
>>>> Fax: +49 30 37309726
>>>> www.mh-sec.de
>>>>
>>>> Marc Heuse - IT-Security Consulting
>>>> Winsstr. 68
>>>> 10405 Berlin
>>>>
>>>> Ust.-Ident.-Nr.: DE244222388
>>>> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
>>
>> --
>> Marc Heuse
>> Mobil: +49 177 9611560
>> Fax: +49 30 37309726
>> www.mh-sec.de
>>
>> Marc Heuse - IT-Security Consulting
>> Winsstr. 68
>> 10405 Berlin
>>
>> Ust.-Ident.-Nr.: DE244222388
>> PGP: AF3D 1D4C D810 F0BB 977D  3807 C7EE D0A0 6BE9 F573
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

Hi,

sorry for digging up this old thread, but I couldn't find a satisfactory
answer on how to use PassManagerBuilder::EP_FullLinkTimeOptimizationLast
somewhere else, so hopefully this helps anyone who stumbles accross this
thread.
> Probably you could patch the gold plugin to support something 
> similar? I haven't tried it, but I don't see any reason it
couldn't
> work. Should look similar to the lld patch I posted.  gold doesn't
> support "-mllvm", but I think you can do the same thing with
> "-plugin-opt".
Unfortunately, this doesn't work. I've tried to patch the gold-plugin to
support plugins as described (see
https://github.com/nescio007/llvm-project/commit/a1baff03ec60225fb84c2e6b0b4b98cd165b2b9f).
However, running my own passes always fails with `undefined
symbol`-errors. The reason is that the gold-plugin itself is loaded by
gold with `dlopen` but without specifying the `RTLD_GLOBAL` flag.
Therefore, llvm-specific symbols from the gold-plugin won't be visible
to llvm-passes loaded at a later stage.
> When you say it does not work with trunk at the moment - did it work
> previously? e.g. with the llvm 9 release? Or did it never work so far
> and is in the progress of being fixed?
It seems the patched lld will only be part of llvm 11 (e.g. tag
`llvmorg-11.0.0-rc2` contains the patch), but I could easily backport it
to llvm 10.0.1 as well (see
https://github.com/nescio007/llvm-project/tree/llvm-10.0.1_lld_plugins).

Best
Johannes Krupp


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5345 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200908/84f5140b/attachment.bin>