llvm dev - Mar 2017 - Distributing llc and opt with own package?

Hi Matthias,

what I’m observing right now is that replacing opt+llc with an clang invocation,
and
subsequently fewer intermediate files, increases the consumed time with -O0 by
200%.
We used to always run opt with -mem2reg -globalopt, and I believe those are not
part
of -O0 (is there an easy way to list all passes that -OX flags to clang imply
for the
optimizer and code gen?).

Could the IR imply certain optimization passes to be required? Could llvm be
taught
that say a certain calling convention needs a pass to run?

Cheers,
 Moritz
> On Mar 12, 2017, at 3:41 AM, Matthias Braun <matze at braunis.de>
wrote:
> 
> -mllvm xxx will pass options directly to the llvm libraries though for that
the same caveats apply in that they are more of a developer/debug tool than a
stable interface.
> The stable interface to clang is the usual flags like -O0/-O3/-fXXX/-mXXX
...
> 
> If you find yourself in a position that you want to control the pass
pipeline down to single passes (why would you want to enable/disable mem2reg
really?) then you are relying on LLVM internals anyway. If you need/want that
level of control then your best bet would be to link against the LLVM libraries
or I guess shipping and relying on llc/opt is similarily good/bad. It will work
but you will have to do more work to keep track with llvm releases, development
of new passes, new behaviors, ...
> 
> - Matthias
> 
>> Am 10.03.2017 um 23:50 schrieb Moritz Angermann <moritz.angermann at
gmail.com>:
>> 
>> Hi Matthias,
>> 
>> I’m trying to see if I can make do with the clang interface only.
>> How do I pass flags to opt or llc? Say I want to enable/disable tbaa,
>> which could be done with `--enable-tbaa=true/false` or `-mem2reg` or
>> `-globalopt`? Would the `-llvm` switch be, what I’m looking for or the
>> `-Xclang`, or something else entirely?
>> 
>> Cheers,
>> Moritz
>> 
>>> On Mar 11, 2017, at 2:26 AM, Matthias Braun <mbraun at
apple.com> wrote:
>>> 
>>> Just a word of warning:
>>> 
>>> llc and opt are primarily developed as debugging tools to support
LLVM testing. They are not intended to be shipped or used as code generators for
frontends. They do often work in these settings but you may hit rough edges that
nobody will care to fix and the behavior or commandline interface may change
every release... A more stable alternative is usually to feed .ir/.bc files to
clang or better link to the llvm libraries and generate the code from there.
>>> 
>>> - Matthias
>>> 
>>>> On Mar 10, 2017, at 6:31 AM, Moritz Angermann via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I was wondering, if someone could point me into the right
direction,
>>>> towards building just llc and opt (potentially 4.0 + patches)
to distribute
>>>> them in the interim with compiler?  The idea is that we’d like
to pin our
>>>> llvm ir code generator to a specific set of llc and opt tools,
with
>>>> potential patches applied, until we manage to get all those
patches
>>>> upstreamed.
>>>> 
>>>> I’d like to find answers to the following question:
>>>> - How trivially can I build just llc and opt into (maybe
static?)
>>>> distributable binaries?
>>>> - Would generating them for multiple architectures pose any
specific
>>>> complications?
>>>> 
>>>> Cheers,
>>>> Moritz
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> 
>>

> On Mar 11, 2017, at 4:30 PM, Moritz Angermann via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Matthias,
> 
> what I’m observing right now is that replacing opt+llc with an clang
invocation, and
> subsequently fewer intermediate files, increases the consumed time with -O0
by 200%.
> We used to always run opt with -mem2reg -globalopt, and I believe those are
not part
> of -O0 (is there an easy way to list all passes that -OX flags to clang
imply for the
> optimizer and code gen?).
I believe most of the OptimizationLevel handling happens in
lib/Passes/PassBuilder.cpp.
> 
> Could the IR imply certain optimization passes to be required? Could llvm
be taught
> that say a certain calling convention needs a pass to run?
Todays pass manager computes a static schedule before processing any functions
so that wouldn't be possible. I don't know whether different pipelines
per function would be possible with the new pass manager. Switching the
optimization passes in the pipeline based on the calling convention seems odd to
me; I would at least use a separate function attribute if I were to implement
this.

- Matthias
> 
> Cheers,
> Moritz
> 
>> On Mar 12, 2017, at 3:41 AM, Matthias Braun <matze at braunis.de>
wrote:
>> 
>> -mllvm xxx will pass options directly to the llvm libraries though for
that the same caveats apply in that they are more of a developer/debug tool than
a stable interface.
>> The stable interface to clang is the usual flags like
-O0/-O3/-fXXX/-mXXX ...
>> 
>> If you find yourself in a position that you want to control the pass
pipeline down to single passes (why would you want to enable/disable mem2reg
really?) then you are relying on LLVM internals anyway. If you need/want that
level of control then your best bet would be to link against the LLVM libraries
or I guess shipping and relying on llc/opt is similarily good/bad. It will work
but you will have to do more work to keep track with llvm releases, development
of new passes, new behaviors, ...
>> 
>> - Matthias
>> 
>>> Am 10.03.2017 um 23:50 schrieb Moritz Angermann
<moritz.angermann at gmail.com>:
>>> 
>>> Hi Matthias,
>>> 
>>> I’m trying to see if I can make do with the clang interface only.
>>> How do I pass flags to opt or llc? Say I want to enable/disable
tbaa,
>>> which could be done with `--enable-tbaa=true/false` or `-mem2reg`
or
>>> `-globalopt`? Would the `-llvm` switch be, what I’m looking for or
the
>>> `-Xclang`, or something else entirely?
>>> 
>>> Cheers,
>>> Moritz
>>> 
>>>> On Mar 11, 2017, at 2:26 AM, Matthias Braun <mbraun at
apple.com> wrote:
>>>> 
>>>> Just a word of warning:
>>>> 
>>>> llc and opt are primarily developed as debugging tools to
support LLVM testing. They are not intended to be shipped or used as code
generators for frontends. They do often work in these settings but you may hit
rough edges that nobody will care to fix and the behavior or commandline
interface may change every release... A more stable alternative is usually to
feed .ir/.bc files to clang or better link to the llvm libraries and generate
the code from there.
>>>> 
>>>> - Matthias
>>>> 
>>>>> On Mar 10, 2017, at 6:31 AM, Moritz Angermann via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I was wondering, if someone could point me into the right
direction,
>>>>> towards building just llc and opt (potentially 4.0 +
patches) to distribute
>>>>> them in the interim with compiler?  The idea is that we’d
like to pin our
>>>>> llvm ir code generator to a specific set of llc and opt
tools, with
>>>>> potential patches applied, until we manage to get all those
patches
>>>>> upstreamed.
>>>>> 
>>>>> I’d like to find answers to the following question:
>>>>> - How trivially can I build just llc and opt into (maybe
static?)
>>>>> distributable binaries?
>>>>> - Would generating them for multiple architectures pose any
specific
>>>>> complications?
>>>>> 
>>>>> Cheers,
>>>>> Moritz
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> 
>>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On Mar 11, 2017, at 4:30 PM, Moritz Angermann via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Matthias,
> 
> what I’m observing right now is that replacing opt+llc with an clang
invocation, and
> subsequently fewer intermediate files, increases the consumed time with -O0
by 200%.
> We used to always run opt with -mem2reg -globalopt, and I believe those are
not part
> of -O0 (is there an easy way to list all passes that -OX flags to clang
imply for the
> optimizer and code gen?).
No there isn’t.

> Could the IR imply certain optimization passes to be required? Could llvm
be taught
> that say a certain calling convention needs a pass to run?
This could be done, but we’d need to here more about the motivation. I suspect
the solution may end up different if we understand the exact use-case.

— 
Mehdi


> 
> Cheers,
> Moritz
> 
>> On Mar 12, 2017, at 3:41 AM, Matthias Braun <matze at braunis.de>
wrote:
>> 
>> -mllvm xxx will pass options directly to the llvm libraries though for
that the same caveats apply in that they are more of a developer/debug tool than
a stable interface.
>> The stable interface to clang is the usual flags like
-O0/-O3/-fXXX/-mXXX ...
>> 
>> If you find yourself in a position that you want to control the pass
pipeline down to single passes (why would you want to enable/disable mem2reg
really?) then you are relying on LLVM internals anyway. If you need/want that
level of control then your best bet would be to link against the LLVM libraries
or I guess shipping and relying on llc/opt is similarily good/bad. It will work
but you will have to do more work to keep track with llvm releases, development
of new passes, new behaviors, ...
>> 
>> - Matthias
>> 
>>> Am 10.03.2017 um 23:50 schrieb Moritz Angermann
<moritz.angermann at gmail.com>:
>>> 
>>> Hi Matthias,
>>> 
>>> I’m trying to see if I can make do with the clang interface only.
>>> How do I pass flags to opt or llc? Say I want to enable/disable
tbaa,
>>> which could be done with `--enable-tbaa=true/false` or `-mem2reg`
or
>>> `-globalopt`? Would the `-llvm` switch be, what I’m looking for or
the
>>> `-Xclang`, or something else entirely?
>>> 
>>> Cheers,
>>> Moritz
>>> 
>>>> On Mar 11, 2017, at 2:26 AM, Matthias Braun <mbraun at
apple.com> wrote:
>>>> 
>>>> Just a word of warning:
>>>> 
>>>> llc and opt are primarily developed as debugging tools to
support LLVM testing. They are not intended to be shipped or used as code
generators for frontends. They do often work in these settings but you may hit
rough edges that nobody will care to fix and the behavior or commandline
interface may change every release... A more stable alternative is usually to
feed .ir/.bc files to clang or better link to the llvm libraries and generate
the code from there.
>>>> 
>>>> - Matthias
>>>> 
>>>>> On Mar 10, 2017, at 6:31 AM, Moritz Angermann via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I was wondering, if someone could point me into the right
direction,
>>>>> towards building just llc and opt (potentially 4.0 +
patches) to distribute
>>>>> them in the interim with compiler?  The idea is that we’d
like to pin our
>>>>> llvm ir code generator to a specific set of llc and opt
tools, with
>>>>> potential patches applied, until we manage to get all those
patches
>>>>> upstreamed.
>>>>> 
>>>>> I’d like to find answers to the following question:
>>>>> - How trivially can I build just llc and opt into (maybe
static?)
>>>>> distributable binaries?
>>>>> - Would generating them for multiple architectures pose any
specific
>>>>> complications?
>>>>> 
>>>>> Cheers,
>>>>> Moritz
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> 
>>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On Mar 14, 2017, at 3:02 AM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
>> 
>> On Mar 11, 2017, at 4:30 PM, Moritz Angermann via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>> 
>> Hi Matthias,
>> 
>> what I’m observing right now is that replacing opt+llc with an clang
invocation, and
>> subsequently fewer intermediate files, increases the consumed time with
-O0 by 200%.
>> We used to always run opt with -mem2reg -globalopt, and I believe those
are not part
>> of -O0 (is there an easy way to list all passes that -OX flags to clang
imply for the
>> optimizer and code gen?).
> 
> No there isn’t.
> 
> 
>> Could the IR imply certain optimization passes to be required? Could
llvm be taught
>> that say a certain calling convention needs a pass to run?
> 
> This could be done, but we’d need to here more about the motivation. I
suspect the solution may end up different if we understand the exact use-case.
Sure. This is all about the ghccc. Here the original discussion between Chris
Lattner
and David Terei who initially implemented the llvm backend for ghc:
http://www.nondot.org/sabre/LLVMNotes/GlobalRegisterVariables.txt

Cheers,
 moritz

> — 
> Mehdi
> 
> 
> 
>> 
>> Cheers,
>> Moritz
>> 
>>> On Mar 12, 2017, at 3:41 AM, Matthias Braun <matze at
braunis.de> wrote:
>>> 
>>> -mllvm xxx will pass options directly to the llvm libraries though
for that the same caveats apply in that they are more of a developer/debug tool
than a stable interface.
>>> The stable interface to clang is the usual flags like
-O0/-O3/-fXXX/-mXXX ...
>>> 
>>> If you find yourself in a position that you want to control the
pass pipeline down to single passes (why would you want to enable/disable
mem2reg really?) then you are relying on LLVM internals anyway. If you need/want
that level of control then your best bet would be to link against the LLVM
libraries or I guess shipping and relying on llc/opt is similarily good/bad. It
will work but you will have to do more work to keep track with llvm releases,
development of new passes, new behaviors, ...
>>> 
>>> - Matthias
>>> 
>>>> Am 10.03.2017 um 23:50 schrieb Moritz Angermann
<moritz.angermann at gmail.com>:
>>>> 
>>>> Hi Matthias,
>>>> 
>>>> I’m trying to see if I can make do with the clang interface
only.
>>>> How do I pass flags to opt or llc? Say I want to enable/disable
tbaa,
>>>> which could be done with `--enable-tbaa=true/false` or
`-mem2reg` or
>>>> `-globalopt`? Would the `-llvm` switch be, what I’m looking for
or the
>>>> `-Xclang`, or something else entirely?
>>>> 
>>>> Cheers,
>>>> Moritz
>>>> 
>>>>> On Mar 11, 2017, at 2:26 AM, Matthias Braun <mbraun at
apple.com> wrote:
>>>>> 
>>>>> Just a word of warning:
>>>>> 
>>>>> llc and opt are primarily developed as debugging tools to
support LLVM testing. They are not intended to be shipped or used as code
generators for frontends. They do often work in these settings but you may hit
rough edges that nobody will care to fix and the behavior or commandline
interface may change every release... A more stable alternative is usually to
feed .ir/.bc files to clang or better link to the llvm libraries and generate
the code from there.
>>>>> 
>>>>> - Matthias
>>>>> 
>>>>>> On Mar 10, 2017, at 6:31 AM, Moritz Angermann via
llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I was wondering, if someone could point me into the
right direction,
>>>>>> towards building just llc and opt (potentially 4.0 +
patches) to distribute
>>>>>> them in the interim with compiler?  The idea is that
we’d like to pin our
>>>>>> llvm ir code generator to a specific set of llc and opt
tools, with
>>>>>> potential patches applied, until we manage to get all
those patches
>>>>>> upstreamed.
>>>>>> 
>>>>>> I’d like to find answers to the following question:
>>>>>> - How trivially can I build just llc and opt into
(maybe static?)
>>>>>> distributable binaries?
>>>>>> - Would generating them for multiple architectures pose
any specific
>>>>>> complications?
>>>>>> 
>>>>>> Cheers,
>>>>>> Moritz
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>> 
>>>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev