llvm dev - May 2016 - [cfe-dev] Fwd: Raising CMake minimum version to 3.4.3

> On Apr 27, 2016, at 11:17 AM, Renato Golin <renato.golin at
linaro.org> wrote:
> 
> On 27 April 2016 at 18:41, Chris Bieneman <beanz at apple.com> wrote:
>> Renato, in your most recent email you comment about differentiating
*stable* vs random versions of CMake built from source. I believe as a community
our recommendation should be that people download CMake sources from
https://CMake.org/download/ instead of git. Then they get the source of a known
stable release. If people choose to pull random git hashes or tags, that is
their decision, but since the minimum version I'm proposing is widely
available I think it will be uncommon.
> 
> I'm guilty of cloning the git repo on my local tree, because it's
> easier to do that than to find the link, download, untar, clean up,
> etc. Then there's a bug, and you need to upgrade, "git pull"
is much
> simpler than repeating the whole process again. :)
> 
> Also, recent "stable", as Chuck has shown, is less stable than
> previous stable. I'd like to err in the side of caution.
> 
> 
>> I've been working to try and un-do the horrible Darwin approach we
have because I think it has some serious problems, but an alternative solution
would be to extend the Darwin approach to every other platform. If you do this
you don't need ExternalProject, but you do need to do some really dirty
hacks. If you want to understand those hacks, just look at
CompilerRTDarwinUtils.cmake in compiler-rt. That is a bit of evil that I’m
responsible for, and I’ll be paying down that debt for a long time to come.
> 
> Point taken, it is horrible. (sorry)
> 
> 
>> Apart from this we also have another problem. The build system makes
configuration decisions based on the compiler *at configuration time*. The
problem is you really want to configure compiler-rt (and the other runtimes)
*after* you've already built clang so the runtimes are configured and built
with the clang you just built. The only way to do that is with some mechanism
similar to ExternalProject.
> 
> <devil's advocate>We could come up with a set of rules that makes
it
> possible (not preferable) to do so at config time.</da>
> 
> Is that where the current complication comes from? What makes it so
> horrible and ends up as a home brew version of ExternalProject?
Recent CMakes have a few interesting things for us in ExternalProject.

CMake 3.1 added BUILD_ALWAYS (2a93843) which allows the build target of a
sub-project to always be called. This is really important because when you use
external projects the top-level build file doesn’t have the dependency
information for the sub-projects. So if you change sources in compiler-rt it
won’t know to rebuild compiler-rt. You need to force it to always invoke ninja
in the sub-project so that it will pick up changes.

Unfortunately even with CMake 3.1 BUILD_ALWAYS doesn’t actually work for Ninja
because of a bug that was fixed in CMake 3.4.1 (b6f4717).

I’m suggesting the version 3.4.3 because it is the last 3.4.x build, and the
CMake community is really strict about what gets put into patch releases.

> 
> 
>> We could roll our own. At this point I'm going to put on my
"non-constructive reality" hat. I'm not going to do that.
> 
> Absolutely not. Ten shots to the head before we start rolling thing
> our own that are perfectly covered by a tool we already use.
> 
> 
>> Replicating ExternalProject would be a lot of work...
> 
> Just to be clear, I didn't mean that. If you're saying that, to get
> the desired functionality we'll either need ExternalProject or
> something identical to it, then the choice is clear. I was asking if
> there was another (good, but less good) solution to the problem.
> 
> Rolling our own is not even a solution, much less a good one.
It isn’t necessarily something identical to it. ExternalProject isn’t actually
designed for this use, and it has loads of functionality we don’t need.
> 
> 
>> As a data point, the one place in LLVM where we don't use
ExternalProject (and we should be) is the cross-compiling goop, and it is a
gigantic mess (see: CrossCompile.cmake and every mention of LLVM_NATIVE_BUILD in
TableGen.cmake). It is filled with bugs and in desperate need of disentangling.
This is another one of my sins that I need to repent.
> 
> Oh, so ExternalProject would also fix cross compilation!? I'm sold! :)
And hopefully we can use it to make the optimized tablegen option more robust
too.
> 
> 
>> I feel I should also point out that ExternalProject isn't a silver
bullet. It isn't going to solve all our problems, and since it is one of the
more actively changing parts of CMake we may find ourselves re-visiting this
conversation in a year or two talking about a newer version of CMake.
> 
> Right, this is an interesting point. I don't want to do this every
> year, so let's be as reasonable as we can.
> 
> Is there anything in 3.5 that can really save us a lot of effort in
> the current work?
I don’t believe so. I’ve read the release notes
(https://cmake.org/cmake/help/v3.5/release/3.5.html
<https://cmake.org/cmake/help/v3.5/release/3.5.html>), and I don’t think
there is anything really important to us in them. I am interested in a few
things being discussed on the cmake-developers list though
(http://public.kitware.com/pipermail/cmake-developers/2016-January/027370.html
<http://public.kitware.com/pipermail/cmake-developers/2016-January/027370.html>).
It looks like that might make CMake 3.6, and it could be interesting for us.
That said, this is likely a case where we could easily support the old and new
versions without much problem and it doesn’t impact build correctness.
> 
> I'm now using CMake 3.2.2 on all buildbots. If I have to move it up to
> 3.4.3, I'm fairly confident that there will be absolutely no problem
> with it.
> 
> But as Chuck said, there were some crashes on 3.5.2 that needed 3.5.3,
> and by using it on so many different platforms, we may uncover bugs,
> and may have to roll back to 3.4 temporarily, and then forwards to
> 3.5, etc.
This is a big part of why I’m suggesting 3.4.3 instead of 3.5.0 or 3.5.1 which
are also widely available.
> 
> I really want to avoid that. But I also want to avoid doing this over
> again next year.
While I don’t want to advocate for us updating frequently “just because”, I do
think that if there are compelling reasons to update every year or two, we
shouldn’t be afraid to.

-Chris
> 
> Finally, Linaro cares more about Linux on ARM (v7+) than bare-metal,
> but we can't just break the rest. I know there are people out there
> that have infrastructure to test that kind of restricted environment,
> not only on ARM, but also MIPS and old PPC32 stuff. I think they
> should voice their concerns before we take any harsh decision.
> 
> cheers,
> --renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160427/45b39cc4/attachment.html>

On 27 April 2016 at 19:44, Chris Bieneman <beanz at apple.com>
wrote:
> I’m suggesting the version 3.4.3 because it is the last 3.4.x build, and
the
> CMake community is really strict about what gets put into patch releases.
That certainly seems to be the case. I'm happy with 3.4.x.

> I don’t believe so. I’ve read the release notes
> (https://cmake.org/cmake/help/v3.5/release/3.5.html), and I don’t think
> there is anything really important to us in them. I am interested in a few
> things being discussed on the cmake-developers list though
>
(http://public.kitware.com/pipermail/cmake-developers/2016-January/027370.html).
> It looks like that might make CMake 3.6, and it could be interesting for
us.
> That said, this is likely a case where we could easily support the old and
> new versions without much problem and it doesn’t impact build correctness.
Is CMake like Linux that the even releases are stable and odd are
experimental (or vice versa)?

If not, I think we can safely update to CMake M.N as soon as M.N+1 is
declared "stable". This worked before with 2.8, and should work with
3.4.

> While I don’t want to advocate for us updating frequently “just because”, I
> do think that if there are compelling reasons to update every year or two,
> we shouldn’t be afraid to.
This is a big issue now because we're still relying on what's on the
distros.

As soon as we officially move to built-in, and as long as we keep our
documentation in order, and make sure to mention strict versions and
build methods (no git, only tarballs), then moving CMake versions
won't be a big issue.

cheers,
--renato

>
> Is CMake like Linux that the even releases are stable and odd are
> experimental (or vice versa)?
>
Nope.  All releases are considered stable and we generalyl have a pretty
long RC cyle, frequently going through 4, 5, or sometimes 6 RC candidates
to find regressions before a minor version release.  And AFAIK the kernel
dropped the whole even=stable odd=testing scheme long ago.

Of course we're always happy to have more testers in an RC cycle :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160427/650aecfe/attachment.html>

Since there seems to be no strong objections remaining I’d like to propose the
following timeline and process:

4/23 - I will send another follow-up email reminding everyone of this change and
timeline
4/30 - I will send a final notice an hour before making the change to the LLVM,
Clang, Compiler-RT, Clang-Tools-Extra, LibCXX, LibCXXABI and Test-Suite repos

During the week of 4/30 I will revert as necessary if bots fail. Hopefully
having the change permanently landed by the middle of the week.

Does this sound agreeable to everyone?

Thanks,
-Chris

> On Apr 28, 2016, at 4:23 PM, Rowan, Jim via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Late to the party, but we’re ok with upgrading to any suitably stable
version.   I don’t have enough background to know which ones are “suitably
stable”.. we have bots that run on windows, some old versions of SuSe, and
Ubuntu (both x86), and native hexagon bots will come to life at some point.  
Sounds to me like 3.4.3 is the right choice.
> 
> It would be nice to have at least a month of lead time. (Which the OP did
provide..)   Upgrading every year or so isn’t a hardship.
> 
> Jim Rowan
> jmr at codeaurora.org
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
by the Linux Foundation
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On May 3, 2016, at 9:41 AM, Smith, Kevin B <kevin.b.smith at
intel.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org
<mailto:cfe-dev-bounces at lists.llvm.org>] On Behalf Of Chris
>> Bieneman via cfe-dev
>> Sent: Tuesday, May 03, 2016 9:07 AM
>> To: LLVM Dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>>; Clang Dev <cfe-
>> dev at lists.llvm.org <mailto:dev at lists.llvm.org>>
>> Cc: Chris Matthews <cmatthews5 at apple.com <mailto:cmatthews5 at
apple.com>>; Galina Kistanova
>> <gkistanova at gmail.com <mailto:gkistanova at gmail.com>>
>> Subject: Re: [cfe-dev] [llvm-dev] Fwd: Raising CMake minimum version to
>> 3.4.3
>> 
>> Since there seems to be no strong objections remaining I’d like to
propose
>> the following timeline and process:
>> 
>> 4/23 - I will send another follow-up email reminding everyone of this
change
>> and timeline
>> 4/30 - I will send a final notice an hour before making the change to
the
>> LLVM, Clang, Compiler-RT, Clang-Tools-Extra, LibCXX, LibCXXABI and
Test-
>> Suite repos
>> 
>> During the week of 4/30 I will revert as necessary if bots fail.
Hopefully
>> having the change permanently landed by the middle of the week.
> 
> Do you mean 5/23 and 5/30?
Yep… Apparently my brain decided it wasn’t ready for April to end yet.

-Chris
> 
>> 
>> Does this sound agreeable to everyone?
>> 
>> Thanks,
>> -Chris
>> 
>> 
>>> On Apr 28, 2016, at 4:23 PM, Rowan, Jim via llvm-dev <llvm-
>> dev at lists.llvm.org> wrote:
>>> 
>>> Late to the party, but we’re ok with upgrading to any suitably
stable
>> version.   I don’t have enough background to know which ones are
“suitably
>> stable”.. we have bots that run on windows, some old versions of SuSe,
and
>> Ubuntu (both x86), and native hexagon bots will come to life at some
point.
>> Sounds to me like 3.4.3 is the right choice.
>>> 
>>> It would be nice to have at least a month of lead time. (Which the
OP did
>> provide..)   Upgrading every year or so isn’t a hardship.
>>> 
>>> Jim Rowan
>>> jmr at codeaurora.org
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>> hosted by the Linux Foundation
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160503/b15604d0/attachment.html>

Renato,

This approach sounds great to me.

Chris M let me know separately that GreenDragon is on CMake 3.5.

I belive Galina and Takumi maintain the largest chunk of remaining public bots.
Do either of you have any thoughts on this?

Thanks,
-Chris
> On May 3, 2016, at 11:19 AM, Renato Golin <renato.golin at
linaro.org> wrote:
> 
> On 3 May 2016 at 17:07, Chris Bieneman via cfe-dev
> <cfe-dev at lists.llvm.org> wrote:
>> Since there seems to be no strong objections remaining I’d like to
propose the following timeline and process:
>> 
>> 4/23 - I will send another follow-up email reminding everyone of this
change and timeline
>> 4/30 - I will send a final notice an hour before making the change to
the LLVM, Clang, Compiler-RT, Clang-Tools-Extra, LibCXX, LibCXXABI and
Test-Suite repos
>> 
>> During the week of 4/30 I will revert as necessary if bots fail.
Hopefully having the change permanently landed by the middle of the week.
>> 
>> Does this sound agreeable to everyone?
> 
> I'd like to propose a different approach. How about we do this the
> other way around? Maybe we should try the "move first, fix
later",
> than the "break first, despair later".
> 
> This week we (Linaro) have finished our buildbot migration, all of
> them running on CMake 3.4.3. I'd like to see if other bot maintainers
> could take the same effort before a certain date.
> 
> So, we can still have the same dates (in May, of course), but with
> different "labels":
> 
> Soon, you send an announcement:
> * Saying the consensus is to move to CMake 3.4.3 as well as a
> compiled CMake for all platforms,
> * That the technical reason is strong enough to do so, even if it
> incurs more work to some people,
> * That *ALL* bot owners should migrate as soon as possible (no later
> than 30th).
> 
> On 23rd, you send a reminder to the list informing all bot owners that
> time is running out.
> * If they can't do it, who could help them
> * If they need push in the CMake community, or a new stable release,
> it's better to have it now than then
> 
> On 30th, you do the migration as you proposed.
> 
> It's all the same, but with the difference that we're involving all
> bot owners, and hopefully not having to revert too many times such a
> troublesome change (CMake changes always mess up the bots anyway).
> 
> Anyway, for ARM/AArch64, we're good to go.
> 
> cheers,
> --renato

+1
On May 3, 2016, at 2:29 PM, Chris Bieneman via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> Renato,
> 
> This approach sounds great to me.
> 
> Chris M let me know separately that GreenDragon is on CMake 3.5.
> 
> I belive Galina and Takumi maintain the largest chunk of remaining public
bots. Do either of you have any thoughts on this?
> 
> Thanks,
> -Chris
> 
>> On May 3, 2016, at 11:19 AM, Renato Golin <renato.golin at
linaro.org> wrote:
>> 
>> On 3 May 2016 at 17:07, Chris Bieneman via cfe-dev
>> <cfe-dev at lists.llvm.org> wrote:
>>> Since there seems to be no strong objections remaining I’d like to
propose the following timeline and process:
>>> 
>>> 4/23 - I will send another follow-up email reminding everyone of
this change and timeline
>>> 4/30 - I will send a final notice an hour before making the change
to the LLVM, Clang, Compiler-RT, Clang-Tools-Extra, LibCXX, LibCXXABI and
Test-Suite repos
>>> 
>>> During the week of 4/30 I will revert as necessary if bots fail.
Hopefully having the change permanently landed by the middle of the week.
>>> 
>>> Does this sound agreeable to everyone?
>> 
>> I'd like to propose a different approach. How about we do this the
>> other way around? Maybe we should try the "move first, fix
later",
>> than the "break first, despair later".
>> 
>> This week we (Linaro) have finished our buildbot migration, all of
>> them running on CMake 3.4.3. I'd like to see if other bot
maintainers
>> could take the same effort before a certain date.
>> 
>> So, we can still have the same dates (in May, of course), but with
>> different "labels":
>> 
>> Soon, you send an announcement:
>> * Saying the consensus is to move to CMake 3.4.3 as well as a
>> compiled CMake for all platforms,
>> * That the technical reason is strong enough to do so, even if it
>> incurs more work to some people,
>> * That *ALL* bot owners should migrate as soon as possible (no later
>> than 30th).
>> 
>> On 23rd, you send a reminder to the list informing all bot owners that
>> time is running out.
>> * If they can't do it, who could help them
>> * If they need push in the CMake community, or a new stable release,
>> it's better to have it now than then
>> 
>> On 30th, you do the migration as you proposed.
>> 
>> It's all the same, but with the difference that we're involving
all
>> bot owners, and hopefully not having to revert too many times such a
>> troublesome change (CMake changes always mess up the bots anyway).
>> 
>> Anyway, for ARM/AArch64, we're good to go.
>> 
>> cheers,
>> --renato
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Jim Rowan
jmr at codeaurora.org
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the
Linux Foundation

Chris,
We have upgrade our bots (both internal and external) to CMake 3.5.2.  So
we are good to go.

-Mike

On Tue, May 3, 2016 at 12:29 PM, Chris Bieneman via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Renato,
>
> This approach sounds great to me.
>
> Chris M let me know separately that GreenDragon is on CMake 3.5.
>
> I belive Galina and Takumi maintain the largest chunk of remaining public
> bots. Do either of you have any thoughts on this?
>
> Thanks,
> -Chris
>
> > On May 3, 2016, at 11:19 AM, Renato Golin <renato.golin at
linaro.org>
> wrote:
> >
> > On 3 May 2016 at 17:07, Chris Bieneman via cfe-dev
> > <cfe-dev at lists.llvm.org> wrote:
> >> Since there seems to be no strong objections remaining I’d like to
> propose the following timeline and process:
> >>
> >> 4/23 - I will send another follow-up email reminding everyone of
this
> change and timeline
> >> 4/30 - I will send a final notice an hour before making the change
to
> the LLVM, Clang, Compiler-RT, Clang-Tools-Extra, LibCXX, LibCXXABI and
> Test-Suite repos
> >>
> >> During the week of 4/30 I will revert as necessary if bots fail.
> Hopefully having the change permanently landed by the middle of the week.
> >>
> >> Does this sound agreeable to everyone?
> >
> > I'd like to propose a different approach. How about we do this the
> > other way around? Maybe we should try the "move first, fix
later",
> > than the "break first, despair later".
> >
> > This week we (Linaro) have finished our buildbot migration, all of
> > them running on CMake 3.4.3. I'd like to see if other bot
maintainers
> > could take the same effort before a certain date.
> >
> > So, we can still have the same dates (in May, of course), but with
> > different "labels":
> >
> > Soon, you send an announcement:
> > * Saying the consensus is to move to CMake 3.4.3 as well as a
> > compiled CMake for all platforms,
> > * That the technical reason is strong enough to do so, even if it
> > incurs more work to some people,
> > * That *ALL* bot owners should migrate as soon as possible (no later
> > than 30th).
> >
> > On 23rd, you send a reminder to the list informing all bot owners that
> > time is running out.
> > * If they can't do it, who could help them
> > * If they need push in the CMake community, or a new stable release,
> > it's better to have it now than then
> >
> > On 30th, you do the migration as you proposed.
> >
> > It's all the same, but with the difference that we're
involving all
> > bot owners, and hopefully not having to revert too many times such a
> > troublesome change (CMake changes always mess up the bots anyway).
> >
> > Anyway, for ARM/AArch64, we're good to go.
> >
> > cheers,
> > --renato
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160504/a4a06b39/attachment.html>