llvm dev - Sep 2019 - Setting llvm::TargetOptions::GuaranteedTailCallOpt in LTO Code Generation

Hi Dwight,

Welcome to LLVM-dev! A few comments below. Cc'ing a few people who
hopefully can add info on some of the specific issues here.

Teresa

On Wed, Sep 18, 2019 at 9:04 AM Dwight Guth via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> I am lead developer of a project that is using LLVM to implement an
> ahead-of-time compiled functional language. We use llc -tailcallopt to
> ensure that functions that end in a tail call are compiled to a tail call
> at the machine level, because we have a number of cases in our interpreter
> where functions with different function signatures call one another in
> deeply nested recursive calls.
>
Maybe a naive question - would that be fixable?

> We can't use `musttail` because the callee and caller often have
different
> signatures.
>
> We would like to support link time optimization in our programming
> language, because performance is important to us. However, there is no
> clang flag to enable the GuaranteedTailCallOpt flag, and the only way to
> pass target options to the lto plugin currently is via an unsupported API
> that parses those flags to static variables.
>
I assume you mean passing internal options via -mllvm through the linker?

> This works on Linux, but the Mac OS linker does not actually initialize
> the TargetOptions that it passes as an llvm::lto::Config based on the
> parsed static variables, and Apple is uninterested in spending time
> supporting an unsupported LLVM API like -mllvm (understandably).
>
lto::Config is part of the new LTO API. For the most part ld64 uses the old
legacy LTO API, and therefore does not even use llvm::lto::Config (the one
exception is to share the code for computing a cache key). But it doesn't
use this when invoking the code generation passes. I'm surprised that ld64
would not have a way to pass through internal llvm options - presumably
that is necessary for debugging and tuning. +Steven Wu <stevenwu at
apple.com> to
give more info here (I work on Linux code and therefore have only directly
used gold and lld, which both use the new LTO API).

> Is there a change to the LLVM project that you might be willing to accept
> that we might be able to create a patch for that would allow us to get link
> time optimization enabled in our programming language on Mac OS, at least
> in the future? And if so, is it possible that someone could give me
> pointers on how to proceed? I'm a relative novice studying this code
and
> I'm not really sure how all the components fit together at a high level
and
> thus what the correct design for something like this would be.
>
I guess the question is what interface would work for you. Would passing an
internal option like what works on lld or what you are doing with llc be
acceptable?

If you need a more officially supported mechanism, IMO the best way is
probably to create a new function attribute (e.g. 'forcetailcall' or
something equivalent to what GuaranteedTailCallOpt implies). That would be
completely linker agnostic and also not rely on internal options.

> Or am I going to have to resign myself to waiting until lld is well
> supported at linking mach-o files?
>
+Rui Ueyama <ruiu at google.com> and
+Eric Christopher <echristo at gmail.com> to comment on lld Mach-O
support.

Teresa

>
> Thanks,
>
> --
>
> Dwight Guth
>
> Director of Engineering
>
> Email: dwight.guth at runtimeverification.com
>
> <https://www.runtimeverification.com>
>
> <https://github.com/dwightguth>
> <https://www.linkedin.com/company/3142238/>  
<https://twitter.com/rv_inc>
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190918/8bec7c04/attachment.html>

On Wed, Sep 18, 2019, 11:57 AM Teresa Johnson <tejohnson at google.com>
wrote:
> Hi Dwight,
>
> Welcome to LLVM-dev! A few comments below. Cc'ing a few people who
> hopefully can add info on some of the specific issues here.
>
> Teresa
>
> On Wed, Sep 18, 2019 at 9:04 AM Dwight Guth via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I am lead developer of a project that is using LLVM to implement an
>> ahead-of-time compiled functional language. We use llc -tailcallopt to
>> ensure that functions that end in a tail call are compiled to a tail
call
>> at the machine level, because we have a number of cases in our
interpreter
>> where functions with different function signatures call one another in
>> deeply nested recursive calls.
>>
>
> Maybe a naive question - would that be fixable?
>
I doubt that we can get around this easily. It's a programming language
compiler, so the guarantee that if the user writes a tail call in their
code, they will get a tail call at the machine level is pretty important.
Restricting that guarantee to only functions that call themselves would
probably cause a lot of problems, including stack overflows, for code
written in our programming language. Recursion is basically the only way to
loop in most functional languages; that's why the tailcallopt flag was
created.

>
>> We can't use `musttail` because the callee and caller often have
>> different signatures.
>>
>> We would like to support link time optimization in our programming
>> language, because performance is important to us. However, there is no
>> clang flag to enable the GuaranteedTailCallOpt flag, and the only way
to
>> pass target options to the lto plugin currently is via an unsupported
API
>> that parses those flags to static variables.
>>
>
> I assume you mean passing internal options via -mllvm through the linker?
>
Yes that's correct, we are passing -mllvm -tailcallopt to lld on Linux.

>
>> This works on Linux, but the Mac OS linker does not actually initialize
>> the TargetOptions that it passes as an llvm::lto::Config based on the
>> parsed static variables, and Apple is uninterested in spending time
>> supporting an unsupported LLVM API like -mllvm (understandably).
>>
>
> lto::Config is part of the new LTO API. For the most part ld64 uses the
> old legacy LTO API, and therefore does not even use llvm::lto::Config (the
> one exception is to share the code for computing a cache key). But it
> doesn't use this when invoking the code generation passes. I'm
surprised
> that ld64 would not have a way to pass through internal llvm options -
> presumably that is necessary for debugging and tuning. +Steven Wu
> <stevenwu at apple.com> to give more info here (I work on Linux code
and
> therefore have only directly used gold and lld, which both use the new LTO
> API).
>
ld64 does have an -mllvm flag but when you pass -mllvm -tailcallopt, it
will happily parse this flag, but it ignores the resulting value when
initializing the code generator, and when I reported the issue to Apple,
they said they would not fix it because -mllvm is not an officially
supported API.

>
>> Is there a change to the LLVM project that you might be willing to
accept
>> that we might be able to create a patch for that would allow us to get
link
>> time optimization enabled in our programming language on Mac OS, at
least
>> in the future? And if so, is it possible that someone could give me
>> pointers on how to proceed? I'm a relative novice studying this
code and
>> I'm not really sure how all the components fit together at a high
level and
>> thus what the correct design for something like this would be.
>>
>
> I guess the question is what interface would work for you. Would passing
> an internal option like what works on lld or what you are doing with llc be
> acceptable?
>
Yes, this would be fine with us, if it's possible. How would I go about
making this happen?

> If you need a more officially supported mechanism, IMO the best way is
> probably to create a new function attribute (e.g. 'forcetailcall'
or
> something equivalent to what GuaranteedTailCallOpt implies). That would be
> completely linker agnostic and also not rely on internal options.
>
>
>> Or am I going to have to resign myself to waiting until lld is well
>> supported at linking mach-o files?
>>
>
> +Rui Ueyama <ruiu at google.com> and
> +Eric Christopher <echristo at gmail.com> to comment on lld Mach-O
support.
>
> Teresa
>
>
>>
>> Thanks,
>>
>> --
>>
>> Dwight Guth
>>
>> Director of Engineering
>>
>> Email: dwight.guth at runtimeverification.com
>>
>> <https://www.runtimeverification.com>
>>
>> <https://github.com/dwightguth>
>> <https://www.linkedin.com/company/3142238/>
>> <https://twitter.com/rv_inc>
>>
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> --
> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190918/8ca89a11/attachment.html>

Hi Dwight

Thanks for the feedback. For the issue you reported, there has been few reviews
trying to tweak the -mllvm option when using legacy LTO interfaces (myself
included) but it never got enough traction to moving forward. Note how
-tailcallopt is implemented as a -mllvm flag means that it is a debug option and
probably not well tested. The option is also not stable which means it can be
renamed without notification.

I also feel like passing -tailcallopt in the linker stage is kind of fragile. It
is better to create an attribute (on function or callInst) to force tailcallopt
and some compiler flag to generated that during IRGen.

Steven
> On Sep 18, 2019, at 10:09 AM, Dwight Guth <dwight.guth at
runtimeverification.com> wrote:
> 
> On Wed, Sep 18, 2019, 11:57 AM Teresa Johnson <tejohnson at google.com
<mailto:tejohnson at google.com>> wrote:
> Hi Dwight,
> 
> Welcome to LLVM-dev! A few comments below. Cc'ing a few people who
hopefully can add info on some of the specific issues here.
> 
> Teresa
> 
> On Wed, Sep 18, 2019 at 9:04 AM Dwight Guth via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi,
> 
> I am lead developer of a project that is using LLVM to implement an
ahead-of-time compiled functional language. We use llc -tailcallopt to ensure
that functions that end in a tail call are compiled to a tail call at the
machine level, because we have a number of cases in our interpreter where
functions with different function signatures call one another in deeply nested
recursive calls.
> 
> Maybe a naive question - would that be fixable?
> 
> I doubt that we can get around this easily. It's a programming language
compiler, so the guarantee that if the user writes a tail call in their code,
they will get a tail call at the machine level is pretty important. Restricting
that guarantee to only functions that call themselves would probably cause a lot
of problems, including stack overflows, for code written in our programming
language. Recursion is basically the only way to loop in most functional
languages; that's why the tailcallopt flag was created.
> 
>  
> We can't use `musttail` because the callee and caller often have
different signatures.
> 
> We would like to support link time optimization in our programming
language, because performance is important to us. However, there is no clang
flag to enable the GuaranteedTailCallOpt flag, and the only way to pass target
options to the lto plugin currently is via an unsupported API that parses those
flags to static variables.
> 
> I assume you mean passing internal options via -mllvm through the linker?
> 
> Yes that's correct, we are passing -mllvm -tailcallopt to lld on Linux.
> 
>  
> This works on Linux, but the Mac OS linker does not actually initialize the
TargetOptions that it passes as an llvm::lto::Config based on the parsed static
variables, and Apple is uninterested in spending time supporting an unsupported
LLVM API like -mllvm (understandably).
> 
> lto::Config is part of the new LTO API. For the most part ld64 uses the old
legacy LTO API, and therefore does not even use llvm::lto::Config (the one
exception is to share the code for computing a cache key). But it doesn't
use this when invoking the code generation passes. I'm surprised that ld64
would not have a way to pass through internal llvm options - presumably that is
necessary for debugging and tuning. +Steven Wu <mailto:stevenwu at
apple.com> to give more info here (I work on Linux code and therefore have
only directly used gold and lld, which both use the new LTO API).
> 
> ld64 does have an -mllvm flag but when you pass -mllvm -tailcallopt, it
will happily parse this flag, but it ignores the resulting value when
initializing the code generator, and when I reported the issue to Apple, they
said they would not fix it because -mllvm is not an officially supported API.
> 
> 
> 
> Is there a change to the LLVM project that you might be willing to accept
that we might be able to create a patch for that would allow us to get link time
optimization enabled in our programming language on Mac OS, at least in the
future? And if so, is it possible that someone could give me pointers on how to
proceed? I'm a relative novice studying this code and I'm not really
sure how all the components fit together at a high level and thus what the
correct design for something like this would be.
> 
> I guess the question is what interface would work for you. Would passing an
internal option like what works on lld or what you are doing with llc be
acceptable?
> 
> Yes, this would be fine with us, if it's possible. How would I go about
making this happen?
> 
> 
> If you need a more officially supported mechanism, IMO the best way is
probably to create a new function attribute (e.g. 'forcetailcall' or
something equivalent to what GuaranteedTailCallOpt implies). That would be
completely linker agnostic and also not rely on internal options.
> 
> 
> Or am I going to have to resign myself to waiting until lld is well
supported at linking mach-o files?
> 
> +Rui Ueyama <mailto:ruiu at google.com> and 
> +Eric Christopher <mailto:echristo at gmail.com> to comment on lld
Mach-O support.
> 
> Teresa
>  
> 
> Thanks,
> 
> -- 
> Dwight Guth
> Director of Engineering
> 
> Email: dwight.guth at runtimeverification.com <mailto:dwight.guth at
runtimeverification.com>
>  <https://www.runtimeverification.com/>
>  <https://github.com/dwightguth> 
<https://www.linkedin.com/company/3142238/>  
<https://twitter.com/rv_inc>
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> -- 
> Teresa Johnson |	 Software Engineer |	 tejohnson at google.com
<mailto:tejohnson at google.com> |
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190918/5a6abd82/attachment.html>