llvm dev - Sep 2013 - [LLVMdev] [RFC] Internal command line options should not be statically initialized.

On Tue, Sep 17, 2013 at 11:29 AM, Reid Kleckner <rnk at google.com> wrote:
> Wait, I have a terrible idea.  Why don't we roll our own .init_array
style
> appending section?  I think we can make this work for all toolchains we
> support.
>
Andy and I talked about this, but I don't think its worth it. My opinion is:
1. For tool options (the top-level llc, opt, llvm-as etc. opts) it doesn't
matter.
2. For experimental options (options that we would be happy if they were
compiled out of a production compiler/JIT client/whatever), it doesn't
matter.
3. For backend options that need to always be available, lots of them
probably already need to get promoted to real API.
4. For the remaining options (ones that don't need to become API, but also
aren't purely experimental), many of them can probably easily be
initialized by some existing initialization hook (pass initialization,
target initialization).
5. There aren't enough options left not in those categories to motivate
some kind of clever solution.

Another way of looking at it is: the implicitly initialized option syntax
is really convenient for experimental options, but those are exactly the
ones that don't cause problems because we could be happy just compiling
them out. For almost everything else, the implicitly initialized
"feature"
of llvm::cl isn't all that useful, and is in some cases actively harmful.

 - Daniel

We'd have something like:
>
> struct PODOptData {
>   const char *FlagName;
> ...  // Common POD stuff, can be initialized at ParseCommandLine time.
> };
>
> LLVM_SECTION(".llvmopt")
> PODOptData OptionRegisterer = { "foo_flag", ... };
>
> I know the COFF magic to get the section bounds to form an array, and I
> know it exists for ELF, but I don't know how to do it on Darwin.
>
>
>
> On Tue, Sep 17, 2013 at 10:10 AM, Andrew Trick <atrick at apple.com>
wrote:
>
>> LLVM's internal command line library needs to evolve. We have an
>> immediate need to build LLVM as a library free of static initializers,
but
>> before brute-force fixing this problem, I'd like outline the
incremental
>> steps that will lead to a desirable long term solution. We want
>> infrastructure in place to provide an evolutionary path.
>>
>> In the near term, clients who need llvm-the-library with no static
>> initializers will build with LLVM_NO_STATICINIT. In this mode, existing
>> users of cl::opt will default to being optimized away as constant
>> initializers, and those options will not be available for command line
>> parsing.
>>
>> A new option class will need be defined, cl::toolopt. Initially, this
>> will share the implementation and API with cl::opt. The only difference
>> will be that cl::toolopt is immune to LLVM_NO_STATICINIT. Options that
are
>> required for tool support can simply be type-renamed to toolopt. Since
>> these are not defined in a library, their static initializers are
>> irrelevant.
>>
>> Eventually, we would like to eliminate the special LLVM_NO_STATICINIT
>> build mode. This can be done by making the -Asserts build just as
strict.
>> In the meantime, we would like to run as many unit tests as possible
with
>> LLVM_NO_STATICINIT builds. This will be solved by gradually moving
cl::opt
>> definitions buried within LLVM libraries to to a new pattern that
avoids
>> static initialization.
>>
>> One easy pattern to follow is to register the option during pass
>> initialization with all the convenient flags and parameters, but refer
to a
>> globally defined option storage that enforces the singleton and
provides
>> visibility. As long as pass initialization happens before
parseCommandLine,
>> usage should be consistent.
>>
>> Strawman:
>>
>> cl::optval<bool> MyOption; // Just the storage, no
initialization.
>>
>> MyPass() {
>>   // Only registers an option with the same optval once.
>>   Option cl::registerOpt(MyOption, cl::init(false), cl::Hidden,
>>                          cl::desc("Descriptive string..."),
);
>> }
>>
>> -Andy
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130917/af818c1f/attachment.html>

On Sep 17, 2013, at 10:10 AM, Andrew Trick <atrick at apple.com>
wrote:
> LLVM's internal command line library needs to evolve. We have an
immediate need to build LLVM as a library free of static initializers, but
before brute-force fixing this problem, I'd like outline the incremental
steps that will lead to a desirable long term solution. We want infrastructure
in place to provide an evolutionary path.
Thank you for tackling this, we should have fixed this years ago.

Please do a pass over the cl::opts we have, and remove ones that are long dead
or unused.   Do we still need -join-liveintervals? :-)


On Sep 17, 2013, at 12:03 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
> On Tue, Sep 17, 2013 at 11:29 AM, Reid Kleckner <rnk at google.com>
wrote:
> Wait, I have a terrible idea.  Why don't we roll our own .init_array
style appending section?  I think we can make this work for all toolchains we
support.
> 
> Andy and I talked about this, but I don't think its worth it. My
opinion is:
> 1. For tool options (the top-level llc, opt, llvm-as etc. opts) it
doesn't matter.
> 2. For experimental options (options that we would be happy if they were
compiled out of a production compiler/JIT client/whatever), it doesn't
matter.
> 3. For backend options that need to always be available, lots of them
probably already need to get promoted to real API.
> 4. For the remaining options (ones that don't need to become API, but
also aren't purely experimental), many of them can probably easily be
initialized by some existing initialization hook (pass initialization, target
initialization).
> 5. There aren't enough options left not in those categories to motivate
some kind of clever solution.
I think that this is a great summary of the problem.  Having cl::opt's
compiled *out* of non-assert build by default makes a lot of sense to me, and
having tool options use toolopt<> (or something) also makes perfect sense.

If you're going to go and tackle pass-specific options, I think that we
should consider changing the syntax and overall design of the command line
options.   We already have some manual name mangling/namespacification of
options (e.g. -tail-dup-limit=). Perhaps we should formalize this somehow?

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130918/9f083950/attachment.html>

On Wed, Sep 18, 2013 at 11:58 AM, Chris Lattner <clattner at apple.com>
wrote:
> On Sep 17, 2013, at 10:10 AM, Andrew Trick <atrick at apple.com>
wrote:
>
> LLVM's internal command line library needs to evolve. We have an
immediate
> need to build LLVM as a library free of static initializers, but before
> brute-force fixing this problem, I'd like outline the incremental steps
> that will lead to a desirable long term solution. We want infrastructure in
> place to provide an evolutionary path.
>
>
> Thank you for tackling this, we should have fixed this years ago.
>
> Please do a pass over the cl::opts we have, and remove ones that are long
> dead or unused.   Do we still need -join-liveintervals? :-)
>
>
>
> On Sep 17, 2013, at 12:03 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
>
> On Tue, Sep 17, 2013 at 11:29 AM, Reid Kleckner <rnk at google.com>
wrote:
>
>> Wait, I have a terrible idea.  Why don't we roll our own
.init_array
>> style appending section?  I think we can make this work for all
toolchains
>> we support.
>>
>
> Andy and I talked about this, but I don't think its worth it. My
opinion
> is:
> 1. For tool options (the top-level llc, opt, llvm-as etc. opts) it
doesn't
> matter.
> 2. For experimental options (options that we would be happy if they were
> compiled out of a production compiler/JIT client/whatever), it doesn't
> matter.
> 3. For backend options that need to always be available, lots of them
> probably already need to get promoted to real API.
> 4. For the remaining options (ones that don't need to become API, but
also
> aren't purely experimental), many of them can probably easily be
> initialized by some existing initialization hook (pass initialization,
> target initialization).
> 5. There aren't enough options left not in those categories to motivate
> some kind of clever solution.
>
>
> I think that this is a great summary of the problem.  Having cl::opt's
> compiled *out* of non-assert build by default makes a lot of sense to me,
> and having tool options use toolopt<> (or something) also makes
perfect
> sense.
>
> If you're going to go and tackle pass-specific options, I think that we
> should consider changing the syntax and overall design of the command line
> options.   We already have some manual name mangling/namespacification of
> options (e.g. -tail-dup-limit=). Perhaps we should formalize this somehow?
>
>
In my work on LTO this summer, I kept getting a desire to be able to
"parameterize" passes to see how their behavior changes. One thing I
wanted
to do related to some mailing list discussions was to try running a
"light"
inlining pass at various stages, but AFAIK LLVM doesn't have a way to do
something like `opt ... -simple-inliner(40) ... -inline`. In one of
Shuxin's preliminary LTO experiment patches, in order to get a
SimpleInliner with threshold 40, he had to add C++ code (admittedly little,
but still requiring a recompile, and the threshold was hardcoded).

Another example where "pass parametrization" was a huge win that I ran
into
this summer was with the ASan instrumentation pass that uses cl::opt's for
this (it has a LOT of cl::opt's btw). Being able to dynamically configure
the shadow offset and scale with the existing SDK compiler was a crucial
productivity win, so losing that would be a shame. Maybe there could be
something like a clang option to allow
`-configure-pass=asan(mapping-scale=...,offset-log=...)` that actually
associates that configuration with ASan? Idk (that's just off the top of my
head).

-- Sean Silva

> -Chris
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130918/5ea6adf8/attachment.html>

On Sep 18, 2013, at 8:58 AM, Chris Lattner <clattner at apple.com> wrote:
> On Sep 17, 2013, at 10:10 AM, Andrew Trick <atrick at apple.com>
wrote:
>> LLVM's internal command line library needs to evolve. We have an
immediate need to build LLVM as a library free of static initializers, but
before brute-force fixing this problem, I'd like outline the incremental
steps that will lead to a desirable long term solution. We want infrastructure
in place to provide an evolutionary path.
> 
> Thank you for tackling this, we should have fixed this years ago.
> 
> Please do a pass over the cl::opts we have, and remove ones that are long
dead or unused.   Do we still need -join-liveintervals? :-)
> 
> 
> On Sep 17, 2013, at 12:03 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
>> On Tue, Sep 17, 2013 at 11:29 AM, Reid Kleckner <rnk at
google.com> wrote:
>> Wait, I have a terrible idea.  Why don't we roll our own
.init_array style appending section?  I think we can make this work for all
toolchains we support.
>> 
>> Andy and I talked about this, but I don't think its worth it. My
opinion is:
>> 1. For tool options (the top-level llc, opt, llvm-as etc. opts) it
doesn't matter.
>> 2. For experimental options (options that we would be happy if they
were compiled out of a production compiler/JIT client/whatever), it doesn't
matter.
>> 3. For backend options that need to always be available, lots of them
probably already need to get promoted to real API.
>> 4. For the remaining options (ones that don't need to become API,
but also aren't purely experimental), many of them can probably easily be
initialized by some existing initialization hook (pass initialization, target
initialization).
>> 5. There aren't enough options left not in those categories to
motivate some kind of clever solution.
> 
> I think that this is a great summary of the problem.  Having cl::opt's
compiled *out* of non-assert build by default makes a lot of sense to me, and
having tool options use toolopt<> (or something) also makes perfect sense.
> 
> If you're going to go and tackle pass-specific options, I think that we
should consider changing the syntax and overall design of the command line
options.   We already have some manual name mangling/namespacification of
options (e.g. -tail-dup-limit=). Perhaps we should formalize this somehow?
Obviously, based on the 18 responses I've gotten, the tone of my first email
was misleading.

I don’t want to stifle discussion, but to be clear, the only thing I propose to
tackle immediately is the removal of static initializers from libraries. There
are several isolated issues that Filip has found good workarounds for. cl::opt
is the one pervasive problem that can't be weeded out one case at a time.

The purpose of posting an RFC and opening up discussion was to find out from
people who have already thought about this, how the ideal cl::opt framework
should work. I won't be making that happen, rather I'll make sure that
the changes we make don't get in the way of future progress.

I would certainly love to see LLVM internal options be reorganized and help
however I can, but I'll be very sad if that holds up removing static
initializers.

-Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130918/d01cbc32/attachment.html>