llvm dev - May 2016 - LLVM Releases: Upstream vs. Downstream / Distros

On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org>
wrote:
> This is a long email :-) I've made some comments inline, but I'll
> summarize my thoughts here:
Thanks Hans!

I'll respond them inline, below.

> - I think we should use the bug tracker to capture issues that affect
> releases. It would be cool if a commit hook could update bugzilla
> entries that refer to it.
That seems like a simple hook.

> At least for the major releases, I think we're doing pretty well on
> timing in terms of predictability: since 3.6, we have release every
> six months: first week of March and first week of September (+- a few
> days). Branching has been similarly predictive: mid-January and
> mid-July.
Indeed, we got a lot better more recently (last 2y), and mostly thanks
to you. :)

We used to vary 3 months +-, and now we're down to a few days.
Whatever we decide, I think we should make it official but putting it
out somewhere, so people can rely on that.

Right now, even if you're extra awesome, there's nothing telling the
distros and LLVM-based products that it will be so if someone else
takes over the responsibility, so they can't adapt.

That's what I meant by "quasi-chaotic".

> If there are many downstream releases for which shifting this schedule
> would be useful, I suppose we could do that, but it seems unlikely
> that there would be agreement on this, and changing the schedule is
> disruptive for those who depend on it.
That's the catch. If we want them to participate, the process has to
have some meaning to them. The fact that not many people do, is clear
to me what it means.

We also need to know better how many other releases already depend on
the upstream process (not just Chromium, for obvious reasons), to be
able to do an informed choice of dates and frequency.

The more well positioned and frequent we are, the more people will
help, but there's a point where the curve bends down, and the cost is
just too great. We need to find the inflection point, and that will
require some initial investigations and guesses, and a lot of fine
tuning later. But if we're all on the same page, I think we can do
that, even if it takes time.

I'm particularly concerned with Android, because they not only have
their own tree with heavily modified LLVM components (ex.
Compiler-RT), but they also build differently and so their process are
completely alien to ours. One of the key reasons why these things
happened is because:

 * They couldn't rely on our releases, as fixing bugs and back-porting
wasn't a thing back then
 * They already had their own release schedule, so aligning with ours
brought no extra benefit
 * We always expected people to work off trunk, and everyone had to
create their own process

I don't want to change how people work, just to add one more valid way
of working, which is most stable for upstream releases. :)


> The only reasonable way I see of aligning upstream releases with
> downstream schedules would be to release much more often. This works
> well in Chromium where there's a 6-week staged release schedule. This
> would mean there's always a branch going for the next release, and
> important bug fixes would get merged to that.
Full validation every 6 weeks is just not possible. But a multiple of
that, say every 3~4 months, could be much easier to work around.


> In Chromium we drive
> this from the bug tracker -- it would be very hard to scan each commit
> for things to cherry-pick. This kind of process has a high cost
> though, there has to be good infrastructure for it (buildbots on the
> branch for all targets, for example), developers have to be aware, and
> even then it's a lot of work for those doing the releases. I'm not
> sure we'd want to take this on. I'm also not sure it would be
suitable
> for a compiler, where we want the releases to have long life-time.
This works because you have a closed system. As you say, Chromium is
mostly final product, not a tool to develop other products, and the
validation is a lot simpler.

With Clang, we'd want to involve external releases into it, and it
simply wouldn't scale.


> For the major releases, I've tried to do this. We could certainly
> formalize it by posting it on the web page though.
I think that'd be the first step, yes. But I wanted to start with a
good number. 2 times a year? Would 3 times improve things that much
for the outsiders? Or just moving the dates would be enough for most
people?

That's why I copied so many outsiders, so they can chime in and let us
know what would be good for *them*.

> Most importantly, those folks should get involved :-)
Indeed!

> In practice, we kind of have this for at least some of the targets.
> Maybe we should write this down somewhere instead of me asking for
> (the same) volunteers each time the release process starts?
I give consent to mark me as the ARM/AArch64 release tester for the
foreseeable future. :)

I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
system running on ARM/AArch64 hardware.

cheers,
--renato

On Wed, May 11, 2016 at 11:10 AM, Renato Golin <renato.golin at
linaro.org> wrote:
> We also need to know better how many other releases already depend on
> the upstream process (not just Chromium, for obvious reasons), to be
> able to do an informed choice of dates and frequency.
Just a small note: Chromium doesn't use the releases, but instead
picks a good revision of the trunk every other week or so.

> -----Original Message-----
> From: Renato Golin [mailto:renato.golin at linaro.org]
> Sent: Wednesday, May 11, 2016 11:11 AM
> To: Hans Wennborg
> Cc: LLVM Dev; Clang Dev; Quentin Colombet; Tom Stellard; Robinson, Paul;
> Jim Grosbach; Kristof Beyls; Frédéric Richez; Reid Kleckner; Philip
> Reames; Matthias Braun; Bernhard Rosenkränzer; Sylvestre Ledru; Matthias
> Klose; Stephen Hines; Jeff Law; Ed Maste; Behan Webster
> Subject: Re: LLVM Releases: Upstream vs. Downstream / Distros
> 
> On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org> wrote:
> > This is a long email :-) I've made some comments inline, but
I'll
> > summarize my thoughts here:
> 
> Thanks Hans!
> 
> I'll respond them inline, below.
> 
> 
> > - I think we should use the bug tracker to capture issues that affect
> > releases. It would be cool if a commit hook could update bugzilla
> > entries that refer to it.
> 
> That seems like a simple hook.
> 
> 
> > At least for the major releases, I think we're doing pretty well
on
> > timing in terms of predictability: since 3.6, we have release every
> > six months: first week of March and first week of September (+- a few
> > days). Branching has been similarly predictive: mid-January and
> > mid-July.
> 
> Indeed, we got a lot better more recently (last 2y), and mostly thanks
> to you. :)
Absolutely.  It's enough of a track record to allow reasonable planning.
> 
> We used to vary 3 months +-, and now we're down to a few days.
> Whatever we decide, I think we should make it official but putting it
> out somewhere, so people can rely on that.
A public commitment to the future release schedule would be that much
more justification for planning to participate.
> 
> Right now, even if you're extra awesome, there's nothing telling
the
> distros and LLVM-based products that it will be so if someone else
> takes over the responsibility, so they can't adapt.
> 
> That's what I meant by "quasi-chaotic".
> 
> 
> > If there are many downstream releases for which shifting this schedule
> > would be useful, I suppose we could do that, but it seems unlikely
> > that there would be agreement on this, and changing the schedule is
> > disruptive for those who depend on it.
> 
> That's the catch. If we want them to participate, the process has to
> have some meaning to them. The fact that not many people do, is clear
> to me what it means.
> 
> We also need to know better how many other releases already depend on
> the upstream process (not just Chromium, for obvious reasons), to be
> able to do an informed choice of dates and frequency.
> 
> The more well positioned and frequent we are, the more people will
> help, but there's a point where the curve bends down, and the cost is
> just too great. We need to find the inflection point, and that will
> require some initial investigations and guesses, and a lot of fine
> tuning later. But if we're all on the same page, I think we can do
> that, even if it takes time.
> 
> I'm particularly concerned with Android, because they not only have
> their own tree with heavily modified LLVM components (ex.
> Compiler-RT), but they also build differently and so their process are
> completely alien to ours. One of the key reasons why these things
> happened is because:
> 
>  * They couldn't rely on our releases, as fixing bugs and back-porting
> wasn't a thing back then
>  * They already had their own release schedule, so aligning with ours
> brought no extra benefit
>  * We always expected people to work off trunk, and everyone had to
> create their own process
> 
> I don't want to change how people work, just to add one more valid way
> of working, which is most stable for upstream releases. :)
> 
> 
> 
> > The only reasonable way I see of aligning upstream releases with
> > downstream schedules would be to release much more often. This works
> > well in Chromium where there's a 6-week staged release schedule.
This
> > would mean there's always a branch going for the next release, and
> > important bug fixes would get merged to that.
> 
> Full validation every 6 weeks is just not possible. But a multiple of
> that, say every 3~4 months, could be much easier to work around.
> 
> 
> 
> > In Chromium we drive
> > this from the bug tracker -- it would be very hard to scan each commit
> > for things to cherry-pick. This kind of process has a high cost
> > though, there has to be good infrastructure for it (buildbots on the
> > branch for all targets, for example), developers have to be aware, and
> > even then it's a lot of work for those doing the releases. I'm
not
> > sure we'd want to take this on. I'm also not sure it would be
suitable
> > for a compiler, where we want the releases to have long life-time.
> 
> This works because you have a closed system. As you say, Chromium is
> mostly final product, not a tool to develop other products, and the
> validation is a lot simpler.
> 
> With Clang, we'd want to involve external releases into it, and it
> simply wouldn't scale.
> 
> 
> 
> > For the major releases, I've tried to do this. We could certainly
> > formalize it by posting it on the web page though.
> 
> I think that'd be the first step, yes. But I wanted to start with a
> good number. 2 times a year? Would 3 times improve things that much
> for the outsiders? Or just moving the dates would be enough for most
> people?
> 
> That's why I copied so many outsiders, so they can chime in and let us
> know what would be good for *them*.
Data point: At Sony we ship our stuff every 6 months and that isn't
going to change.  Our policy has been to base our releases on upstream
releases, and given our current lead times, the current upstream release
schedule is actually not bad, especially if it has stopped drifting.
Having four releases pretty consistently follow the current schedule is
extremely positive, thanks!

It would help our internal planning to publish the schedule for future
releases.  You never know, we might even be able to fork an internal
branch in time to help with the release testing, although that is not
in any way a lightweight process (so no promises).

There has been some talk about moving toward releasing from upstream
trunk, or more precisely to streamlining our internal testing process
in order to allow us to seriously contemplate releasing from upstream
trunk.  I can't argue with the streamlining part, for sure.  Whether
the rest of it works out must remain to be seen.
--paulr
> 
> 
> > Most importantly, those folks should get involved :-)
> 
> Indeed!
> 
> 
> > In practice, we kind of have this for at least some of the targets.
> > Maybe we should write this down somewhere instead of me asking for
> > (the same) volunteers each time the release process starts?
> 
> I give consent to mark me as the ARM/AArch64 release tester for the
> foreseeable future. :)
> 
> I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
> system running on ARM/AArch64 hardware.
> 
> cheers,
> --renato

On 12 May 2016 at 01:10, Robinson, Paul <paul.robinson at sony.com>
wrote:
> A public commitment to the future release schedule would be that much
> more justification for planning to participate.
Indeed, I think we're all in agreement there.

Not only having people committed to validate the release, or having a
"known" ballpark time, but a public commitment: web pages updates,
emails sent to the appropriate lists, deadlines exposed, etc.

For all those interested, release-testers at lists.llvm.org is the list
we discuss about the validation and plans. Regardless, the web pages
should also be changed as part of the planning and release processes.

> Data point: At Sony we ship our stuff every 6 months and that isn't
> going to change.  Our policy has been to base our releases on upstream
> releases, and given our current lead times, the current upstream release
> schedule is actually not bad, especially if it has stopped drifting.
> Having four releases pretty consistently follow the current schedule is
> extremely positive, thanks!
Excellent, thanks! It seems that the timing is (so far) less of a
problem than I anticipated. This is good news.

> It would help our internal planning to publish the schedule for future
> releases.  You never know, we might even be able to fork an internal
> branch in time to help with the release testing, although that is not
> in any way a lightweight process (so no promises).
I'm betting on the fact that this will get easier the more often we
do, and the more of us that does it.

Filtering false positives is a local cost that cannot be avoided, and
only automated to a point, but filtering thousands of real bugs and
reporting them all should be, in no way, the responsibility of any
downstream release.

The more downstream releases and OS distributions we have doing the
testing (like Bero said), the less each one of us will have to track.
And the more of them that say they're affected by a bug, the higher
the priority is should have to the upstream release.

We've done something similar with the GCC abi tag, but that was a
separate email thread, started by Jeff, and external to our release
process. The number of people involved is staggering, but yet, the
patches are sitting there waiting for review for a very long time.

I think this expresses my point really well. Downstream releases and
OS distros can help us *a lot* with validation, not necessarily
implementing core functionality or even fixing those bugs themselves.
But if we don't fix those bugs or implement the features they need,
they won't have *any* incentive in spending a lot of time and
resources validating the upstream release, and then, all of them will
spend *more* time validating their own.

Increasing the importance of stable releases might get us more work to
do for other people with no real benefit to us, yes. But it'll also
bring us a massive validation network and transform Clang/LLVM into a
production compiler from start (upstream) and that will benefit
everyone, including all users of all downstream and upstream tools.


> There has been some talk about moving toward releasing from upstream
> trunk, or more precisely to streamlining our internal testing process
> in order to allow us to seriously contemplate releasing from upstream
> trunk.  I can't argue with the streamlining part, for sure.  Whether
> the rest of it works out must remain to be seen.
That's what Chromium does, and it's mainly because of the problems
Bero exposed (trunk is often much better than any release).

But also they are using the tool to compile a very small subset of
programs (mainly Chromium/Chrome), so it's *a lot* easier to validate
that.

When you're actually shipping a toolchain, you have to worry not only
with the programs you have, but also customers programs you don't (and
can't) have access to.

If the release process (including minor releases) ends up as frequent
as possible, wouldn't that be similar to merging to trunk every other
month?

In that case, the validation process will be minimal (almost the same
code), but you'd have to spend a bit more time sifting through patches
(which I want to automate) instead.

Hope that makes some sense.

cheers,
--renato

FWIW, for our ARM Compiler product, we follow top-of-trunk, not the releases.
Next to picking up new functionality quicker, it also allows us to detect
regressions
in LLVM against our in-house testing quickly, not 6 months later. We find that
when
we find a regression within 24 to 48 hours of the commit introducing it,
it's much
cheaper to get it fixed.

In my opinion, it would be better overall for the LLVM project if top-of-trunk
is
tested as much as possible, if testing resources are so scarce that a choice
has to be made between testing top-of-trunk or testing a release branch.

Thanks,

Kristof

On 12 May 2016, at 02:10, Robinson, Paul <paul.robinson at
sony.com<mailto:paul.robinson at sony.com>> wrote:

-----Original Message-----
From: Renato Golin [mailto:renato.golin at linaro.org]
Sent: Wednesday, May 11, 2016 11:11 AM
To: Hans Wennborg
Cc: LLVM Dev; Clang Dev; Quentin Colombet; Tom Stellard; Robinson, Paul;
Jim Grosbach; Kristof Beyls; Frédéric Richez; Reid Kleckner; Philip
Reames; Matthias Braun; Bernhard Rosenkränzer; Sylvestre Ledru; Matthias
Klose; Stephen Hines; Jeff Law; Ed Maste; Behan Webster
Subject: Re: LLVM Releases: Upstream vs. Downstream / Distros

On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org<mailto:hans
at chromium.org>> wrote:
This is a long email :-) I've made some comments inline, but I'll
summarize my thoughts here:

Thanks Hans!

I'll respond them inline, below.


- I think we should use the bug tracker to capture issues that affect
releases. It would be cool if a commit hook could update bugzilla
entries that refer to it.

That seems like a simple hook.


At least for the major releases, I think we're doing pretty well on
timing in terms of predictability: since 3.6, we have release every
six months: first week of March and first week of September (+- a few
days). Branching has been similarly predictive: mid-January and
mid-July.

Indeed, we got a lot better more recently (last 2y), and mostly thanks
to you. :)

Absolutely.  It's enough of a track record to allow reasonable planning.


We used to vary 3 months +-, and now we're down to a few days.
Whatever we decide, I think we should make it official but putting it
out somewhere, so people can rely on that.

A public commitment to the future release schedule would be that much
more justification for planning to participate.


Right now, even if you're extra awesome, there's nothing telling the
distros and LLVM-based products that it will be so if someone else
takes over the responsibility, so they can't adapt.

That's what I meant by "quasi-chaotic".


If there are many downstream releases for which shifting this schedule
would be useful, I suppose we could do that, but it seems unlikely
that there would be agreement on this, and changing the schedule is
disruptive for those who depend on it.

That's the catch. If we want them to participate, the process has to
have some meaning to them. The fact that not many people do, is clear
to me what it means.

We also need to know better how many other releases already depend on
the upstream process (not just Chromium, for obvious reasons), to be
able to do an informed choice of dates and frequency.

The more well positioned and frequent we are, the more people will
help, but there's a point where the curve bends down, and the cost is
just too great. We need to find the inflection point, and that will
require some initial investigations and guesses, and a lot of fine
tuning later. But if we're all on the same page, I think we can do
that, even if it takes time.

I'm particularly concerned with Android, because they not only have
their own tree with heavily modified LLVM components (ex.
Compiler-RT), but they also build differently and so their process are
completely alien to ours. One of the key reasons why these things
happened is because:

* They couldn't rely on our releases, as fixing bugs and back-porting
wasn't a thing back then
* They already had their own release schedule, so aligning with ours
brought no extra benefit
* We always expected people to work off trunk, and everyone had to
create their own process

I don't want to change how people work, just to add one more valid way
of working, which is most stable for upstream releases. :)



The only reasonable way I see of aligning upstream releases with
downstream schedules would be to release much more often. This works
well in Chromium where there's a 6-week staged release schedule. This
would mean there's always a branch going for the next release, and
important bug fixes would get merged to that.

Full validation every 6 weeks is just not possible. But a multiple of
that, say every 3~4 months, could be much easier to work around.



In Chromium we drive
this from the bug tracker -- it would be very hard to scan each commit
for things to cherry-pick. This kind of process has a high cost
though, there has to be good infrastructure for it (buildbots on the
branch for all targets, for example), developers have to be aware, and
even then it's a lot of work for those doing the releases. I'm not
sure we'd want to take this on. I'm also not sure it would be suitable
for a compiler, where we want the releases to have long life-time.

This works because you have a closed system. As you say, Chromium is
mostly final product, not a tool to develop other products, and the
validation is a lot simpler.

With Clang, we'd want to involve external releases into it, and it
simply wouldn't scale.



For the major releases, I've tried to do this. We could certainly
formalize it by posting it on the web page though.

I think that'd be the first step, yes. But I wanted to start with a
good number. 2 times a year? Would 3 times improve things that much
for the outsiders? Or just moving the dates would be enough for most
people?

That's why I copied so many outsiders, so they can chime in and let us
know what would be good for *them*.

Data point: At Sony we ship our stuff every 6 months and that isn't
going to change.  Our policy has been to base our releases on upstream
releases, and given our current lead times, the current upstream release
schedule is actually not bad, especially if it has stopped drifting.
Having four releases pretty consistently follow the current schedule is
extremely positive, thanks!

It would help our internal planning to publish the schedule for future
releases.  You never know, we might even be able to fork an internal
branch in time to help with the release testing, although that is not
in any way a lightweight process (so no promises).

There has been some talk about moving toward releasing from upstream
trunk, or more precisely to streamlining our internal testing process
in order to allow us to seriously contemplate releasing from upstream
trunk.  I can't argue with the streamlining part, for sure.  Whether
the rest of it works out must remain to be seen.
--paulr



Most importantly, those folks should get involved :-)

Indeed!


In practice, we kind of have this for at least some of the targets.
Maybe we should write this down somewhere instead of me asking for
(the same) volunteers each time the release process starts?

I give consent to mark me as the ARM/AArch64 release tester for the
foreseeable future. :)

I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
system running on ARM/AArch64 hardware.

cheers,
--renato

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/fe24d394/attachment.html>

>
>
> I'm particularly concerned with Android, because they not only have
> their own tree with heavily modified LLVM components (ex.
> Compiler-RT), but they also build differently and so their process are
> completely alien to ours. One of the key reasons why these things
> happened is because:
>

Errr, Stephen has spoken up here, but my folks are in contact with android
folks pretty much every week, and I don't think what you are stating is
correct on a lot of fronts.

I'm also a little concerned about you speaking for android here, instead of
android speaking for android here :P.  I'll be frank: I don't think you
know enough details of internal history of android to state, affirmatively,
why these things happened for android, and what you are suggesting  is,
AFAIK, not correct or accurate.

So if android is your particular concern here, i can pretty much state that
android LLVM is on a release process close to the rest of Google, which is
'follow TOT very closely'.

I don't think changing how stable works would change that, for a lot of
reasons (mostly around cost of ToT regressions, etc).

So i think you need to find a new motivating example :)

>  * They couldn't rely on our releases, as fixing bugs and back-porting
> wasn't a thing back then
>
This is, AFAIK,  not accurate. Renderscript has its own history not worth
getting into, but outside of the renderscript version, LLVM in android is
very close to TOT. It has been as long as someone really cared about it.


>  * They already had their own release schedule, so aligning with ours
> brought no extra benefit
>
This was not a concern.

 * We always expected people to work off trunk, and everyone had
to
> create their own process
>
Android mostly shares a process with the rest of Google these days, in the
sense that they rely on  validation we do in other contexts.
>
> I don't want to change how people work, just to add one more valid way
> of working, which is most stable for upstream releases. :)
>

Full validation every 6 weeks is just not possible. But a multiple
of
> that, say every 3~4 months, could be much easier to work around.

FWIW: Full validation is already done on a faster-than-6 week time
schedule, so i'm also going to suggest that your "just not
possible" claim
is false :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/d4a37888/attachment.html>

On 12 May 2016 at 16:57, Daniel Berlin <dberlin at dberlin.org>
wrote:
> Errr, Stephen has spoken up here, but my folks are in contact with android
> folks pretty much every week, and I don't think what you are stating is
> correct on a lot of fronts.
I obviously don't speak for Android and have already apologised to
Steve about my choice of words.

> So if android is your particular concern here, i can pretty much state that
> android LLVM is on a release process close to the rest of Google, which is
> 'follow TOT very closely'.
Isn't this what I said?

Following ToT very closely is only good for groups that have high
involvement in LLVM, like Google and Android.

And for that reason (and others), Android doesn't use the upstream
releases. I was wondering if we could make anything so they would.

The major benefit wouldn't be, as I explained, specifically for
Google/Android, but for Android users, Linux users, Linux distros,
LLVM library users (including Renderscript), etc.

--renato