llvm dev - May 2016 - LLVM Releases: Upstream vs. Downstream / Distros

Folks,

There has been enough discussion about keeping development downstream
and how painful that is. Even more so, I think we all agree, is having
downstream releases while tracking upstream releases, trunk and other
branches (ex. Android).

I have proposed "en passant" a few times already, but now I'm
going to
do it to a wider audience:

Shall we sync our upstream release with the bulk of other downstream
ones as well as OS distributions?


This work involves a *lot* of premises that are not encoded yet, so
we'll need a lot of work from all of us. But from the recent problems
with GCC abi_tag and the arduous job of downstream release managers to
know which patches to pick, I think there has been a lot of wasted
effort by everyone, and that generates stress, conflicts, etc.

I'd also like to reinforce the basic idea of software engineering: to
use the same pattern for the same problem. So, if we have one way to
link sequences of patches and merge them upstream, we ought to use the
same pattern (preferably the same scripts) downstream, too. Of course
there will be differences, but we should treat them as the exception,
not the rule.

So, a list of things will need to be solved to get to a low waste
release process:


  1. Timing

Many downstream release managers, as well as distro maintainers have
complained about the timing of our releases, and how unreliable they
are, and how that makes it hard for them to plan their own branches,
cherry-picks and merges. If we release too early, they miss out
important optimisations, if we do too late, they'll have to branch
"just before" and risk having to back-port late fixes to their own
modified trees.

Products that rely on LLVM already have their own life cycles and we
can't change that. Nor we can make all downstream products align to
our quasi-chaotic release process. However, the important of the
upstream release for upstream developers is *a lot* lower than for the
downstream maintainers, so unless the latter group puts their weight
(and effort) in the upstream process, little is going to happen to
help them.

A few (random) ideas:

 * Do an average on all product cycles, pick the least costly time to
release. This would marginalise those beyond the first sigma and we'd
make their lives much harder than those within one sigma.
 * Do the same average on the projects that are willing to lend a
serious hand to the upstream release process. This has the same
problem, but it's based on actual effort. It does concentrate bias on
the better funded projects, but it's also easier for low key projects
to change their release schedules.
 * Try to release more often. The current cost of a release is high,
but if we managed to lower it (by having more people, more automation,
shared efforts), than it could be feasible and much fairer than
weighted averages.


  2. Process

Our release process is *very* lean, and that's what makes it
quasi-chaotic. In the beginning, not many people / companies wanted to
help or cared about the releases, so the process was what whomever was
doing, did. The major release process is now better defined, but the
same happened to the minor releases.

For example, we have no defined date to start, or to end. We have no
assigned people to do the official releases, or test the supported
targets. We still rely on voluntary work from all parties. That's ok
when the release is just "a point in time", but if downstream releases
and OS distributions start relying on our releases, we really should
get a bit more professional.

A few (random) ideas:

 * We should have predictable release times, both for starting it and
finishing it. There will be complications, but we should treat them as
the exception, not the rule.
 * We should have appointed members of the community that would be
responsible for those releases, in the same way we have code owners
(volunteers, but no less responsible), so that we can guarantee a
consistent validation across all relevant targets. This goes beyond
x86/ARM/MIPS/PPC and includes the other targets like AMD, NVidia, BPF,
etc.
 * The upstream release should be, as much as possible, independent of
which OS they run on. OS specific releases should be done in the
distributions themselves, and people interested should raise the
concern in their own communities.
 * Downstream managers should be an integral part of the upstream
release process. Whenever the release manager sends the email, they
should test on their end and reply with GREEN/RED flags.
 * Downstream managers should also propose back-ports that are
important to them in the upstream release. It's likely that a fix is
important to a number of downstream releases but not many people
upstream (since we're all using trunk). So, unless they tell us, we
won't know.
 * OS distribution managers should test on their builds, too. I know
FreeBSD and Mandriva build by default with Clang. I know that Debian
has an experimental build. I know that RedHat and Ubuntu have LLVM
packages that they do care. All that has to be tested *at least* every
major release, but hopefully on all releases. (those who already do
that, thank you!)
 * A number of upstream groups, or downstream releases that don't
track upstream releases, should *also* test them on their own
workloads. Doing so, will get the upstream release in a much better
quality level, and in turn, allow those projects to use it on their
own internal releases.
 * Every *new* bug found in any of those downstream tests should be
reported in Bugzilla with the appropriate category (critical / major /
minor). All major bugs have to be closed for the release to be out,
etc. (the specific process will have to be agreed and documented).


  3. Automation

As exposed in the timing and process sections, automation is key to
reducing costs for all parties. We should collate the encoded process
we have upstream with the process projects have downstream, and
convert upstream everything that we can / is relevant.

For example, finding which patches revert / fix another one that was
already cherry-picked is a common task that should be identical to
everyone. A script that would sweep the commit logs, looking for
clues, would be useful to everyone.

A few (random) ideas:

 * We should discuss the process, express the consensus on the
official documentation, and encode it in a set of scripts. It's a lot
easier to tell a developer "please do X because it helps our script
back-port your patch" than to say "please do X because it's
nice" or
"do X because it's in the 'guideline'".
 * There's no way to force (via git-hook) developers to add a bugzilla
ID or a review number on the commit message (not all commits are
equal), so the scripts that scan commits will have to be smart enough,
but that'll create false-positives, and they can't commit without
human intervention. Showing why a commit wasn't picked up by the
script, or was erroneously picked up, is a good way to educate people.
 * We could have a somewhat-common interface with downstream releases,
so some scripts that they use could be upstreamed, if many of them
used the same entry point for testing, validating, building,
packaging.
 * We could have the scripts that distros use for building their own
packages in our tree, so they could maintain them locally and we'd
know which changes are happening and would be much easier to warn the
others, common up the interface, etc.


In the end, we're a bunch of people from *very* different communities
doing similar work. In the spirit of open source, I'd like to propose
that we share the work and the responsibility of producing high
quality software with minimal waste.

I don't think anyone receiving this email disagrees with the statement
that we can't just take and not give back, and that being part of this
community means we may have to work harder than our employers would
think brings direct profit, so that they can profit even more
indirectly later, and with that, everyone that uses or depends on our
software.

My personal and very humble opinion is that coalescing the release
process will, in the long term, actually *save* us a lot of work, and
the quality will be increased. Even if we don't reach perfection, and
by no means I think we will, at least we'll have something slightly
better. If anything, at least we tried.

I'd hate to continue doing an inefficient process without even trying
an alternative...

Comments?

cheers,
--renato

This is a long email :-) I've made some comments inline, but I'll
summarize my thoughts here:

- I like to think that the major releases have been shipped on a
pretty reliable six-month schedule lately. So we have that going for
us :-)

- It seems hard to align our upstream schedule to various downstream
preferences. One way would be to release much more often, but I don't
know if that's really desirable.

- I would absolutely like to see more involvement in the upstream
release processes from downstream folks and distros.

- I think we should use the bug tracker to capture issues that affect
releases. It would be cool if a commit hook could update bugzilla
entries that refer to it.

Cheers,
Hans


On Wed, May 11, 2016 at 7:08 AM, Renato Golin <renato.golin at linaro.org>
wrote:
> Folks,
>
> There has been enough discussion about keeping development downstream
> and how painful that is. Even more so, I think we all agree, is having
> downstream releases while tracking upstream releases, trunk and other
> branches (ex. Android).
>
> I have proposed "en passant" a few times already, but now I'm
going to
> do it to a wider audience:
>
> Shall we sync our upstream release with the bulk of other downstream
> ones as well as OS distributions?
>
>
> This work involves a *lot* of premises that are not encoded yet, so
> we'll need a lot of work from all of us. But from the recent problems
> with GCC abi_tag and the arduous job of downstream release managers to
> know which patches to pick, I think there has been a lot of wasted
> effort by everyone, and that generates stress, conflicts, etc.
>
> I'd also like to reinforce the basic idea of software engineering: to
> use the same pattern for the same problem. So, if we have one way to
> link sequences of patches and merge them upstream, we ought to use the
> same pattern (preferably the same scripts) downstream, too. Of course
> there will be differences, but we should treat them as the exception,
> not the rule.
>
> So, a list of things will need to be solved to get to a low waste
> release process:
>
>
>   1. Timing
>
> Many downstream release managers, as well as distro maintainers have
> complained about the timing of our releases, and how unreliable they
> are, and how that makes it hard for them to plan their own branches,
> cherry-picks and merges. If we release too early, they miss out
> important optimisations, if we do too late, they'll have to branch
> "just before" and risk having to back-port late fixes to their
own
> modified trees.
>
> Products that rely on LLVM already have their own life cycles and we
> can't change that. Nor we can make all downstream products align to
> our quasi-chaotic release process. However, the important of the
> upstream release for upstream developers is *a lot* lower than for the
> downstream maintainers, so unless the latter group puts their weight
> (and effort) in the upstream process, little is going to happen to
> help them.
>
> A few (random) ideas:
>
>  * Do an average on all product cycles, pick the least costly time to
> release. This would marginalise those beyond the first sigma and we'd
> make their lives much harder than those within one sigma.
>  * Do the same average on the projects that are willing to lend a
> serious hand to the upstream release process. This has the same
> problem, but it's based on actual effort. It does concentrate bias on
> the better funded projects, but it's also easier for low key projects
> to change their release schedules.
>  * Try to release more often. The current cost of a release is high,
> but if we managed to lower it (by having more people, more automation,
> shared efforts), than it could be feasible and much fairer than
> weighted averages.
My random thoughts:

At least for the major releases, I think we're doing pretty well on
timing in terms of predictability: since 3.6, we have release every
six months: first week of March and first week of September (+- a few
days). Branching has been similarly predictive: mid-January and
mid-July.

If there are many downstream releases for which shifting this schedule
would be useful, I suppose we could do that, but it seems unlikely
that there would be agreement on this, and changing the schedule is
disruptive for those who depend on it.

The only reasonable way I see of aligning upstream releases with
downstream schedules would be to release much more often. This works
well in Chromium where there's a 6-week staged release schedule. This
would mean there's always a branch going for the next release, and
important bug fixes would get merged to that. In Chromium we drive
this from the bug tracker -- it would be very hard to scan each commit
for things to cherry-pick. This kind of process has a high cost
though, there has to be good infrastructure for it (buildbots on the
branch for all targets, for example), developers have to be aware, and
even then it's a lot of work for those doing the releases. I'm not
sure we'd want to take this on. I'm also not sure it would be suitable
for a compiler, where we want the releases to have long life-time.

>   2. Process
>
> Our release process is *very* lean, and that's what makes it
> quasi-chaotic. In the beginning, not many people / companies wanted to
> help or cared about the releases, so the process was what whomever was
> doing, did. The major release process is now better defined, but the
> same happened to the minor releases.
>
> For example, we have no defined date to start, or to end.
For the major releases, I've tried to do this. We could certainly
formalize it by posting it on the web page though.
> We have no
> assigned people to do the official releases, or test the supported
> targets. We still rely on voluntary work from all parties. That's ok
> when the release is just "a point in time", but if downstream
releases
> and OS distributions start relying on our releases, we really should
> get a bit more professional.
Most importantly, those folks should get involved :-)
>
> A few (random) ideas:
>
>  * We should have predictable release times, both for starting it and
> finishing it. There will be complications, but we should treat them as
> the exception, not the rule.
SGTM, we pretty much already have this for major releases.
>  * We should have appointed members of the community that would be
> responsible for those releases, in the same way we have code owners
> (volunteers, but no less responsible), so that we can guarantee a
> consistent validation across all relevant targets. This goes beyond
> x86/ARM/MIPS/PPC and includes the other targets like AMD, NVidia, BPF,
> etc.
In practice, we kind of have this for at least some of the targets.
Maybe we should write this down somewhere instead of me asking for
(the same) volunteers each time the release process starts?
>  * The upstream release should be, as much as possible, independent of
> which OS they run on. OS specific releases should be done in the
> distributions themselves, and people interested should raise the
> concern in their own communities.
>  * Downstream managers should be an integral part of the upstream
> release process. Whenever the release manager sends the email, they
> should test on their end and reply with GREEN/RED flags.
>  * Downstream managers should also propose back-ports that are
> important to them in the upstream release. It's likely that a fix is
> important to a number of downstream releases but not many people
> upstream (since we're all using trunk). So, unless they tell us, we
> won't know.
>  * OS distribution managers should test on their builds, too. I know
> FreeBSD and Mandriva build by default with Clang. I know that Debian
> has an experimental build. I know that RedHat and Ubuntu have LLVM
> packages that they do care. All that has to be tested *at least* every
> major release, but hopefully on all releases. (those who already do
> that, thank you!)
>  * A number of upstream groups, or downstream releases that don't
> track upstream releases, should *also* test them on their own
> workloads. Doing so, will get the upstream release in a much better
> quality level, and in turn, allow those projects to use it on their
> own internal releases.
>  * Every *new* bug found in any of those downstream tests should be
> reported in Bugzilla with the appropriate category (critical / major /
> minor). All major bugs have to be closed for the release to be out,
> etc. (the specific process will have to be agreed and documented).
>
>
>   3. Automation
>
> As exposed in the timing and process sections, automation is key to
> reducing costs for all parties. We should collate the encoded process
> we have upstream with the process projects have downstream, and
> convert upstream everything that we can / is relevant.
>
> For example, finding which patches revert / fix another one that was
> already cherry-picked is a common task that should be identical to
> everyone. A script that would sweep the commit logs, looking for
> clues, would be useful to everyone.
>
> A few (random) ideas:
>
>  * We should discuss the process, express the consensus on the
> official documentation, and encode it in a set of scripts. It's a lot
> easier to tell a developer "please do X because it helps our script
> back-port your patch" than to say "please do X because it's
nice" or
> "do X because it's in the 'guideline'".
>  * There's no way to force (via git-hook) developers to add a bugzilla
> ID or a review number on the commit message (not all commits are
> equal), so the scripts that scan commits will have to be smart enough,
> but that'll create false-positives, and they can't commit without
> human intervention. Showing why a commit wasn't picked up by the
> script, or was erroneously picked up, is a good way to educate people.
>  * We could have a somewhat-common interface with downstream releases,
> so some scripts that they use could be upstreamed, if many of them
> used the same entry point for testing, validating, building,
> packaging.
>  * We could have the scripts that distros use for building their own
> packages in our tree, so they could maintain them locally and we'd
> know which changes are happening and would be much easier to warn the
> others, common up the interface, etc.
>
>
> In the end, we're a bunch of people from *very* different communities
> doing similar work. In the spirit of open source, I'd like to propose
> that we share the work and the responsibility of producing high
> quality software with minimal waste.
>
> I don't think anyone receiving this email disagrees with the statement
> that we can't just take and not give back, and that being part of this
> community means we may have to work harder than our employers would
> think brings direct profit, so that they can profit even more
> indirectly later, and with that, everyone that uses or depends on our
> software.
>
> My personal and very humble opinion is that coalescing the release
> process will, in the long term, actually *save* us a lot of work, and
> the quality will be increased. Even if we don't reach perfection, and
> by no means I think we will, at least we'll have something slightly
> better. If anything, at least we tried.
>
> I'd hate to continue doing an inefficient process without even trying
> an alternative...
>
> Comments?
>
> cheers,
> --renato

On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org>
wrote:
> This is a long email :-) I've made some comments inline, but I'll
> summarize my thoughts here:
Thanks Hans!

I'll respond them inline, below.

> - I think we should use the bug tracker to capture issues that affect
> releases. It would be cool if a commit hook could update bugzilla
> entries that refer to it.
That seems like a simple hook.

> At least for the major releases, I think we're doing pretty well on
> timing in terms of predictability: since 3.6, we have release every
> six months: first week of March and first week of September (+- a few
> days). Branching has been similarly predictive: mid-January and
> mid-July.
Indeed, we got a lot better more recently (last 2y), and mostly thanks
to you. :)

We used to vary 3 months +-, and now we're down to a few days.
Whatever we decide, I think we should make it official but putting it
out somewhere, so people can rely on that.

Right now, even if you're extra awesome, there's nothing telling the
distros and LLVM-based products that it will be so if someone else
takes over the responsibility, so they can't adapt.

That's what I meant by "quasi-chaotic".

> If there are many downstream releases for which shifting this schedule
> would be useful, I suppose we could do that, but it seems unlikely
> that there would be agreement on this, and changing the schedule is
> disruptive for those who depend on it.
That's the catch. If we want them to participate, the process has to
have some meaning to them. The fact that not many people do, is clear
to me what it means.

We also need to know better how many other releases already depend on
the upstream process (not just Chromium, for obvious reasons), to be
able to do an informed choice of dates and frequency.

The more well positioned and frequent we are, the more people will
help, but there's a point where the curve bends down, and the cost is
just too great. We need to find the inflection point, and that will
require some initial investigations and guesses, and a lot of fine
tuning later. But if we're all on the same page, I think we can do
that, even if it takes time.

I'm particularly concerned with Android, because they not only have
their own tree with heavily modified LLVM components (ex.
Compiler-RT), but they also build differently and so their process are
completely alien to ours. One of the key reasons why these things
happened is because:

 * They couldn't rely on our releases, as fixing bugs and back-porting
wasn't a thing back then
 * They already had their own release schedule, so aligning with ours
brought no extra benefit
 * We always expected people to work off trunk, and everyone had to
create their own process

I don't want to change how people work, just to add one more valid way
of working, which is most stable for upstream releases. :)


> The only reasonable way I see of aligning upstream releases with
> downstream schedules would be to release much more often. This works
> well in Chromium where there's a 6-week staged release schedule. This
> would mean there's always a branch going for the next release, and
> important bug fixes would get merged to that.
Full validation every 6 weeks is just not possible. But a multiple of
that, say every 3~4 months, could be much easier to work around.


> In Chromium we drive
> this from the bug tracker -- it would be very hard to scan each commit
> for things to cherry-pick. This kind of process has a high cost
> though, there has to be good infrastructure for it (buildbots on the
> branch for all targets, for example), developers have to be aware, and
> even then it's a lot of work for those doing the releases. I'm not
> sure we'd want to take this on. I'm also not sure it would be
suitable
> for a compiler, where we want the releases to have long life-time.
This works because you have a closed system. As you say, Chromium is
mostly final product, not a tool to develop other products, and the
validation is a lot simpler.

With Clang, we'd want to involve external releases into it, and it
simply wouldn't scale.


> For the major releases, I've tried to do this. We could certainly
> formalize it by posting it on the web page though.
I think that'd be the first step, yes. But I wanted to start with a
good number. 2 times a year? Would 3 times improve things that much
for the outsiders? Or just moving the dates would be enough for most
people?

That's why I copied so many outsiders, so they can chime in and let us
know what would be good for *them*.

> Most importantly, those folks should get involved :-)
Indeed!

> In practice, we kind of have this for at least some of the targets.
> Maybe we should write this down somewhere instead of me asking for
> (the same) volunteers each time the release process starts?
I give consent to mark me as the ARM/AArch64 release tester for the
foreseeable future. :)

I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
system running on ARM/AArch64 hardware.

cheers,
--renato

Hi,
for those who don't know me, I'm an AOSP developer at work and an
OpenMandriva developer (including maintainer of its toolchains) outside of
work, so I'm looking at this from (at least) 2 different perspectives and
may jump around between them.
Replies inline...

On 11 May 2016 at 16:08, Renato Golin <renato.golin at linaro.org> wrote:
> There has been enough discussion about keeping development downstream
> and how painful that is. Even more so, I think we all agree, is having
> downstream releases while tracking upstream releases, trunk and other
> branches (ex. Android).
>
In the OpenMandriva world, we usually try to have clang (our primary
compiler) as close as possible to the latest upstream stable release.
We're currently following the release_38 branch, and expect to jump on
trunk as soon as our distro release has happened (because we expect 3.9 to
be ready before we'll make our subsequent release - better to get on the
branch we'll be using for the next release early than to suddenly face
problems when updating to the next release).

In the AOSP world, we obviously have to (somewhat) follow what Google does,
which is typically pick a trunk snapshot and work from there - but we have
some work underway to extract their patches so we can apply them on top of
a release or snapshot of our choice (current thought is mostly nightly
builds for testing).

This work involves a *lot* of premises that are not encoded yet,
so
> we'll need a lot of work from all of us. But from the recent problems
> with GCC abi_tag and the arduous job of downstream release managers to
> know which patches to pick, I think there has been a lot of wasted
> effort by everyone, and that generates stress, conflicts, etc.
>
>From both perspectives, it would be great to have a common set of
"known
good" and relevant patches like gcc abi_tag, or fixes for bugs commonly
encountered.
Ideally, I'd like to see those patches just backported on the release_38
branch to keep the number of external patches low.

gcc abi_tag is a bit of a headache in the OpenMandriva world, while we
build just about everything with clang these days, of course it would be
good to restore binary compatibility with the bigger distributions (almost
all of which are using current gcc with the new abi enabled).

Many downstream release managers, as well as distro maintainers
have
> complained about the timing of our releases,

The timing has been quite predictable lately -- but of course the website
still says "TBD" for both 3.8.1 and 3.9.0, maybe communicating the
(likely)
plan could use some improvement.

>  * Do the same average on the projects that are willing to lend a
> serious hand to the upstream release process.

What can we (this time being OpenMandriva) do? We don't have any great
compiler engineers, but we're heavy users - would it help to run a mass
build of all packages for all supported architectures (at this time: i586,
x86_64, armv7hnl, aarch64) to detect errors on a prerelease builds? We have
the infrastructure in place, even showing a fairly nice list of failed
builds along with build logs. (But of course we there will be false
positives caused by e.g. a library update that happened around the same
time as the compiler update.)

 * Try to release more often. The current cost of a release is
high,
> but if we managed to lower it (by having more people, more automation,
> shared efforts), than it could be feasible and much fairer than
> weighted averages.
>
That would be a good idea IMO, we've run into "current trunk is much
better
than the last stable release anyway" situations more than once (in both
projects).

> For example, we have no defined date to start, or to end. We have no
> assigned people to do the official releases, or test the supported
> targets. We still rely on voluntary work from all parties. That's ok
> when the release is just "a point in time", but if downstream
releases
> and OS distributions start relying on our releases, we really should
> get a bit more professional.
>
Backporting some more fixes to the stable branches would be great too (but
of course I realize that's a daunting and not very interesting task).

>  * Downstream managers should be an integral part of the upstream
> release process. Whenever the release manager sends the email, they
> should test on their end and reply with GREEN/RED flags.
>  * Downstream managers should also propose back-ports that are
> important to them in the upstream release. It's likely that a fix is
> important to a number of downstream releases but not many people
> upstream (since we're all using trunk). So, unless they tell us, we
> won't know.
>
Sounds good to me, volunteering to participate in both.

 * We could have the scripts that distros use for building their
own
> packages in our tree, so they could maintain them locally and we'd
> know which changes are happening and would be much easier to warn the
> others, common up the interface, etc.
>
While interesting from an upstream perspective, I doubt that will happen
reliably -- there's too many people working on the build scripts who would
not automatically have write access to the tree etc. and most distro build
farms rely on having the build scripts in a common place, so duplication
would be unavoidable.

ttyl
bero
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/4504b957/attachment.html>

Thanks Bero! Comments inline.


On 11 May 2016 at 23:47, Bernhard Rosenkränzer
<bernhard.rosenkranzer at linaro.org> wrote:
> In the OpenMandriva world, we usually try to have clang (our primary
> compiler) as close as possible to the latest upstream stable release.
It would be good to know what's stopping you from using a fully
upstream process (like patch trunk, back-port, pull).

I'm sure you're not the only one having the same problems, and even if
the only take-out of this thread is to streamline our back-port
process, all downstream releases will already benefit.


> We're currently following the release_38 branch, and expect to jump on
trunk
> as soon as our distro release has happened (because we expect 3.9 to be
> ready before we'll make our subsequent release - better to get on the
branch
> we'll be using for the next release early than to suddenly face
problems
> when updating to the next release).
That's a very good strategy, indeed. And somewhat independent on how
good the release is.

Early help can also increase downstream participation pre-release, and
will eventually considerably reduce the need to back-ports.


> In the AOSP world, we obviously have to (somewhat) follow what Google does,
> which is typically pick a trunk snapshot and work from there - but we have
> some work underway to extract their patches so we can apply them on top of
a
> release or snapshot of our choice (current thought is mostly nightly builds
> for testing).
This sounds awful, and is the very thing I'm trying to minimise. I can
certainly understand the need to local patches, but using different
trunks makes the whole thing very messy.

If the releases are good and timely, and if the back-porting process
is efficient, I hope we'll never have the need to do that.

> From both perspectives, it would be great to have a common set of
"known
> good" and relevant patches like gcc abi_tag, or fixes for bugs
commonly
> encountered.
> Ideally, I'd like to see those patches just backported on the
release_38
> branch to keep the number of external patches low.
Indeed, my point exactly. Downstream releases and distros can easily
share sets of known "good" patches for this or that, but I'd very
much
prefer to have as much as possible upstream.

> gcc abi_tag is a bit of a headache in the OpenMandriva world, while we
build
> just about everything with clang these days, of course it would be good to
> restore binary compatibility with the bigger distributions (almost all of
> which are using current gcc with the new abi enabled).
Ubuntu LTS has just released with LLVM 3.8, which doesn't have the
abi_tag fixes.

If we don't back-port them to 3.8.1 or 3.8.2, Ubuntu will have to do
that on their own, and that's exactly what I'm trying to solve here.

I also feel like they're not the only one with that problem...


> The timing has been quite predictable lately -- but of course the website
> still says "TBD" for both 3.8.1 and 3.9.0, maybe communicating
the (likely)
> plan could use some improvement.
Right, Hans was saying how we should improve in that area, too.

I think that's a very easy consensus to reach, but we still need all
require people to commit to a more rigorous schedule.

> What can we (this time being OpenMandriva) do? We don't have any great
> compiler engineers, but we're heavy users - would it help to run a mass
> build of all packages for all supported architectures (at this time: i586,
> x86_64, armv7hnl, aarch64) to detect errors on a prerelease builds?
YES PLEASE!! :)

> We have
> the infrastructure in place, even showing a fairly nice list of failed
> builds along with build logs. (But of course we there will be false
> positives caused by e.g. a library update that happened around the same
time
> as the compiler update.)
Can you compare the failures of two different builds?

We don't want to fix *all* the problems during the releases, we just
want to know if the new release breaks more stuff than the previous.

How we deal with the remaining bugs is irrelevant to this discussion...

> That would be a good idea IMO, we've run into "current trunk is
much better
> than the last stable release anyway" situations more than once (in
both
> projects).
I expect that, if distros chime in during the release process, this
will be a lot less of a problem.

It also seems that the timing is not that bad, so maybe the best
course of action now is to streamline the process and only if the
pressure is still great, we change the timings.

> Backporting some more fixes to the stable branches would be great too (but
> of course I realize that's a daunting and not very interesting task).
I think that's crucial to keeping the releases relevant.

> Sounds good to me, volunteering to participate in both.
Thanks Bero! If you haven't yet, please subscribe to the
release-testers at lists.llvm.org mailing list.

> While interesting from an upstream perspective, I doubt that will happen
> reliably -- there's too many people working on the build scripts who
would
> not automatically have write access to the tree etc. and most distro build
> farms rely on having the build scripts in a common place, so duplication
> would be unavoidable.
I was sceptical about the shared scripts, too. It was an idea that
came to my mind in the last minute, but I'm not sure how much that
would help anyway.

cheers,
--renato

> On May 11, 2016, at 9:16 AM, Hans Wennborg via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> This is a long email :-) I've made some comments inline, but I'll
> summarize my thoughts here:
> 
> - I like to think that the major releases have been shipped on a
> pretty reliable six-month schedule lately. So we have that going for
> us :-)
> 
> - It seems hard to align our upstream schedule to various downstream
> preferences. One way would be to release much more often, but I don't
> know if that's really desirable.
I’d like to also point out that it’s not only about the schedule, but also a lot
about the release strategy. For example, we branch a very long time before we
release externally. At the beginning of a new release qualification, we will
cherry-pick most of what’s happening on trunk to the release branch. At that
time it is still good to take new optimizations, or refactorings that will make
working on the branch later easier. And then we slow down, but we still take
miscompile fixes and common crashers for something like 4 months after the
branch point. This is very different from what is happening upstream, and I’m
not sure it would be good for the community releases to try to unify with this.

That being said, it makes things a bit easier for us when we branch close to an
open-source release, and it would be nice to be able to plan on doing this.

The result of our qualification work is of course contributed to trunk, it’s
left to the discretion of individual contributors if they want to nominate fixes
for the current open-source release branch. I’m sure we could be better at this,
and I’ll try to message this internally.

Fred
> - I would absolutely like to see more involvement in the upstream
> release processes from downstream folks and distress.
> - I think we should use the bug tracker to capture issues that affect
> releases. It would be cool if a commit hook could update bugzilla
> entries that refer to it.
> 
> Cheers,
> Hans
> 
> 
> On Wed, May 11, 2016 at 7:08 AM, Renato Golin <renato.golin at
linaro.org <mailto:renato.golin at linaro.org>> wrote:
>> Folks,
>> 
>> There has been enough discussion about keeping development downstream
>> and how painful that is. Even more so, I think we all agree, is having
>> downstream releases while tracking upstream releases, trunk and other
>> branches (ex. Android).
>> 
>> I have proposed "en passant" a few times already, but now
I'm going to
>> do it to a wider audience:
>> 
>> Shall we sync our upstream release with the bulk of other downstream
>> ones as well as OS distributions?
>> 
>> 
>> This work involves a *lot* of premises that are not encoded yet, so
>> we'll need a lot of work from all of us. But from the recent
problems
>> with GCC abi_tag and the arduous job of downstream release managers to
>> know which patches to pick, I think there has been a lot of wasted
>> effort by everyone, and that generates stress, conflicts, etc.
>> 
>> I'd also like to reinforce the basic idea of software engineering:
to
>> use the same pattern for the same problem. So, if we have one way to
>> link sequences of patches and merge them upstream, we ought to use the
>> same pattern (preferably the same scripts) downstream, too. Of course
>> there will be differences, but we should treat them as the exception,
>> not the rule.
>> 
>> So, a list of things will need to be solved to get to a low waste
>> release process:
>> 
>> 
>>  1. Timing
>> 
>> Many downstream release managers, as well as distro maintainers have
>> complained about the timing of our releases, and how unreliable they
>> are, and how that makes it hard for them to plan their own branches,
>> cherry-picks and merges. If we release too early, they miss out
>> important optimisations, if we do too late, they'll have to branch
>> "just before" and risk having to back-port late fixes to
their own
>> modified trees.
>> 
>> Products that rely on LLVM already have their own life cycles and we
>> can't change that. Nor we can make all downstream products align to
>> our quasi-chaotic release process. However, the important of the
>> upstream release for upstream developers is *a lot* lower than for the
>> downstream maintainers, so unless the latter group puts their weight
>> (and effort) in the upstream process, little is going to happen to
>> help them.
>> 
>> A few (random) ideas:
>> 
>> * Do an average on all product cycles, pick the least costly time to
>> release. This would marginalise those beyond the first sigma and
we'd
>> make their lives much harder than those within one sigma.
>> * Do the same average on the projects that are willing to lend a
>> serious hand to the upstream release process. This has the same
>> problem, but it's based on actual effort. It does concentrate bias
on
>> the better funded projects, but it's also easier for low key
projects
>> to change their release schedules.
>> * Try to release more often. The current cost of a release is high,
>> but if we managed to lower it (by having more people, more automation,
>> shared efforts), than it could be feasible and much fairer than
>> weighted averages.
> 
> My random thoughts:
> 
> At least for the major releases, I think we're doing pretty well on
> timing in terms of predictability: since 3.6, we have release every
> six months: first week of March and first week of September (+- a few
> days). Branching has been similarly predictive: mid-January and
> mid-July.
> 
> If there are many downstream releases for which shifting this schedule
> would be useful, I suppose we could do that, but it seems unlikely
> that there would be agreement on this, and changing the schedule is
> disruptive for those who depend on it.
> 
> The only reasonable way I see of aligning upstream releases with
> downstream schedules would be to release much more often. This works
> well in Chromium where there's a 6-week staged release schedule. This
> would mean there's always a branch going for the next release, and
> important bug fixes would get merged to that. In Chromium we drive
> this from the bug tracker -- it would be very hard to scan each commit
> for things to cherry-pick. This kind of process has a high cost
> though, there has to be good infrastructure for it (buildbots on the
> branch for all targets, for example), developers have to be aware, and
> even then it's a lot of work for those doing the releases. I'm not
> sure we'd want to take this on. I'm also not sure it would be
suitable
> for a compiler, where we want the releases to have long life-time.
> 
> 
>>  2. Process
>> 
>> Our release process is *very* lean, and that's what makes it
>> quasi-chaotic. In the beginning, not many people / companies wanted to
>> help or cared about the releases, so the process was what whomever was
>> doing, did. The major release process is now better defined, but the
>> same happened to the minor releases.
>> 
>> For example, we have no defined date to start, or to end.
> 
> For the major releases, I've tried to do this. We could certainly
> formalize it by posting it on the web page though.
> 
>> We have no
>> assigned people to do the official releases, or test the supported
>> targets. We still rely on voluntary work from all parties. That's
ok
>> when the release is just "a point in time", but if downstream
releases
>> and OS distributions start relying on our releases, we really should
>> get a bit more professional.
> 
> Most importantly, those folks should get involved :-)
> 
>> 
>> A few (random) ideas:
>> 
>> * We should have predictable release times, both for starting it and
>> finishing it. There will be complications, but we should treat them as
>> the exception, not the rule.
> 
> SGTM, we pretty much already have this for major releases.
> 
>> * We should have appointed members of the community that would be
>> responsible for those releases, in the same way we have code owners
>> (volunteers, but no less responsible), so that we can guarantee a
>> consistent validation across all relevant targets. This goes beyond
>> x86/ARM/MIPS/PPC and includes the other targets like AMD, NVidia, BPF,
>> etc.
> 
> In practice, we kind of have this for at least some of the targets.
> Maybe we should write this down somewhere instead of me asking for
> (the same) volunteers each time the release process starts?
> 
>> * The upstream release should be, as much as possible, independent of
>> which OS they run on. OS specific releases should be done in the
>> distributions themselves, and people interested should raise the
>> concern in their own communities.
>> * Downstream managers should be an integral part of the upstream
>> release process. Whenever the release manager sends the email, they
>> should test on their end and reply with GREEN/RED flags.
>> * Downstream managers should also propose back-ports that are
>> important to them in the upstream release. It's likely that a fix
is
>> important to a number of downstream releases but not many people
>> upstream (since we're all using trunk). So, unless they tell us, we
>> won't know.
>> * OS distribution managers should test on their builds, too. I know
>> FreeBSD and Mandriva build by default with Clang. I know that Debian
>> has an experimental build. I know that RedHat and Ubuntu have LLVM
>> packages that they do care. All that has to be tested *at least* every
>> major release, but hopefully on all releases. (those who already do
>> that, thank you!)
>> * A number of upstream groups, or downstream releases that don't
>> track upstream releases, should *also* test them on their own
>> workloads. Doing so, will get the upstream release in a much better
>> quality level, and in turn, allow those projects to use it on their
>> own internal releases.
>> * Every *new* bug found in any of those downstream tests should be
>> reported in Bugzilla with the appropriate category (critical / major /
>> minor). All major bugs have to be closed for the release to be out,
>> etc. (the specific process will have to be agreed and documented).
>> 
>> 
>>  3. Automation
>> 
>> As exposed in the timing and process sections, automation is key to
>> reducing costs for all parties. We should collate the encoded process
>> we have upstream with the process projects have downstream, and
>> convert upstream everything that we can / is relevant.
>> 
>> For example, finding which patches revert / fix another one that was
>> already cherry-picked is a common task that should be identical to
>> everyone. A script that would sweep the commit logs, looking for
>> clues, would be useful to everyone.
>> 
>> A few (random) ideas:
>> 
>> * We should discuss the process, express the consensus on the
>> official documentation, and encode it in a set of scripts. It's a
lot
>> easier to tell a developer "please do X because it helps our
script
>> back-port your patch" than to say "please do X because
it's nice" or
>> "do X because it's in the 'guideline'".
>> * There's no way to force (via git-hook) developers to add a
bugzilla
>> ID or a review number on the commit message (not all commits are
>> equal), so the scripts that scan commits will have to be smart enough,
>> but that'll create false-positives, and they can't commit
without
>> human intervention. Showing why a commit wasn't picked up by the
>> script, or was erroneously picked up, is a good way to educate people.
>> * We could have a somewhat-common interface with downstream releases,
>> so some scripts that they use could be upstreamed, if many of them
>> used the same entry point for testing, validating, building,
>> packaging.
>> * We could have the scripts that distros use for building their own
>> packages in our tree, so they could maintain them locally and we'd
>> know which changes are happening and would be much easier to warn the
>> others, common up the interface, etc.
>> 
>> 
>> In the end, we're a bunch of people from *very* different
communities
>> doing similar work. In the spirit of open source, I'd like to
propose
>> that we share the work and the responsibility of producing high
>> quality software with minimal waste.
>> 
>> I don't think anyone receiving this email disagrees with the
statement
>> that we can't just take and not give back, and that being part of
this
>> community means we may have to work harder than our employers would
>> think brings direct profit, so that they can profit even more
>> indirectly later, and with that, everyone that uses or depends on our
>> software.
>> 
>> My personal and very humble opinion is that coalescing the release
>> process will, in the long term, actually *save* us a lot of work, and
>> the quality will be increased. Even if we don't reach perfection,
and
>> by no means I think we will, at least we'll have something slightly
>> better. If anything, at least we tried.
>> 
>> I'd hate to continue doing an inefficient process without even
trying
>> an alternative...
>> 
>> Comments?
>> 
>> cheers,
>> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160513/d5573231/attachment.html>

On 05/11/2016 08:08 AM, Renato Golin wrote:
>
> Shall we sync our upstream release with the bulk of other downstream
> ones as well as OS distributions?
I suspect that'll, in general, be hard.

The downstream consumers are all going to have their own needs & 
schedules and as a result I doubt you'll find any upstream release 
schedule & cadence that is acceptable to enough downstream consumers to 
make this viable.

With that in mind, I'm a proponent of timing based schedules with as 
much predictability as can be made.  That allows the downstream 
consumers to make informed decisions based on likely release dates.

>
>
> This work involves a *lot* of premises that are not encoded yet, so
> we'll need a lot of work from all of us. But from the recent problems
> with GCC abi_tag and the arduous job of downstream release managers to
> know which patches to pick, I think there has been a lot of wasted
> effort by everyone, and that generates stress, conflicts, etc.
Ideally as we continue to open more lines of communication we won't run 
into anything as problematical as the abi_tag stuff.  While it's 
important, I wouldn't make it the primary driver for where you're trying
to go.

WRT downstream consumers.  The more the downstream consumer is wired 
into the development community, the more risk (via patches) the 
downstream consumer can reasonably take.  A downstream consumer without 
intimate knowledge of the issues probably shouldn't be taking 
incomplete/unapproved patches and applying them to their tree.

>
>   1. Timing
>
> Many downstream release managers, as well as distro maintainers have
> complained about the timing of our releases, and how unreliable they
> are, and how that makes it hard for them to plan their own branches,
> cherry-picks and merges. If we release too early, they miss out
> important optimisations, if we do too late, they'll have to branch
> "just before" and risk having to back-port late fixes to their
own
> modified trees.
And this just gets bigger as the project gets more downstream consumers. 
    Thus I think you pick a time based release schedule, whatever it may 
be and the downstream consumers can then adjust.

Note that this can have the effect of encouraging them to engage more 
upstream to ensure issues of concern to them are addressed in a timely 
manner.
>
>   2. Process
>
> Our release process is *very* lean, and that's what makes it
> quasi-chaotic. In the beginning, not many people / companies wanted to
> help or cared about the releases, so the process was what whomever was
> doing, did. The major release process is now better defined, but the
> same happened to the minor releases.
>
> For example, we have no defined date to start, or to end. We have no
> assigned people to do the official releases, or test the supported
> targets. We still rely on voluntary work from all parties. That's ok
> when the release is just "a point in time", but if downstream
releases
> and OS distributions start relying on our releases, we really should
> get a bit more professional.
Can't argue with getting a bit more structured, but watch out for going 
too far.  I'd really like to squish down the release phase on the GCC 
side, but it's damn hard at this point.

>
> A few (random) ideas:
>
>  * We should have predictable release times, both for starting it and
> finishing it. There will be complications, but we should treat them as
> the exception, not the rule.
Yes.
>  * We should have appointed members of the community that would be
> responsible for those releases, in the same way we have code owners
> (volunteers, but no less responsible), so that we can guarantee a
> consistent validation across all relevant targets. This goes beyond
> x86/ARM/MIPS/PPC and includes the other targets like AMD, NVidia, BPF,
> etc.
Good luck ;-)  Don't take this wrong, but wrangling volunteers into 
release work is hard.  It's sometimes hard to find a way to motivate 
them to focus on issues important for the release when there's new 
development work they want to be doing.

Mark Mitchell found one good tool for that in his years as the GCC 
release manager -- namely tightening what was allowed on the trunk as 
the desired release date got closer.  ie, there's a free-for-all period, 
then just bugfixes, then just regression fixes, then just doc fixes. 
Developers then had a clear vested interest in moving the release 
forward -- they couldn't commit their new development work until the 
release manager opened the trunk for new development.

>  * OS distribution managers should test on their builds, too. I know
> FreeBSD and Mandriva build by default with Clang. I know that Debian
> has an experimental build. I know that RedHat and Ubuntu have LLVM
> packages that they do care. All that has to be tested *at least* every
> major release, but hopefully on all releases. (those who already do
> that, thank you!)
LLVM's usage on the Fedora side is still small, and smaller still within 
Red Hat.  But we do have an interest in this stuff "just working". 
Given current staffing levels I would expect Fedora to follow the 
upstream releases closely with minimal changes.
>  * Every *new* bug found in any of those downstream tests should be
> reported in Bugzilla with the appropriate category (critical / major /
> minor). All major bugs have to be closed for the release to be out,
> etc. (the specific process will have to be agreed and documented).
Yes, GCC has a similar policy and it has worked reasonably well.  In 
fact it's a good lever for the release manager if you've got a locked 
trunk.  If the developers don't address the issues, then the release 
doesn't branch and the trunk doesn't open for development.  It aligns 
the release manager and a good chunk of the development team's goals.


jeff

Folks,

First, thanks everyone that replied, it made many things much more
clear to many people (especially me!).

But would be good to do a re-cap, since some discussions ended up
list-only, and too many people said too many things to keep track. I
read the whole thread again, and this is my summary of what people
do/want the most.

TL;DR version:

 * Upstream needs to be more effective at communicating and
formalising the process.
 * Downstream / Distros need to be respectively more vocal / involved
about testing ToT and releases.
 * Upstream may need to change some process (bugzilla meta, feature
freeze, more back-ports, git branches) to facilitate downstream help.
 * Downstream needs to be more pushy with their back-port suggestions,
so we do it upstream more often.
 * We all need to come up with a better way of tracking patches for
back-port (separate thread ongoing)

Now, the (slightly) longer version:

By far, the most important thing is cadence and transparency. We
already do a good job at the former, not so much at the latter.

The proposals were:
 - Formalise the dates/duration on a webpage, clarify volunteered
roles, channels (release list, etc).
 - Formalise how we deal with proposals (llvm-commits to release list
/ owner), some suggested using git branches (I like this, but svn).
 - Formalise how we deal with bugs (bugzilla meta, make it green),
this could also be used as back-port proposal.
 - Formalise how we deal with back-ports, how long we do, overlapping
minor releases, etc.

The other important factor is top-of-tree validation as a way to get
more stable releases. We have lots of buildbots and the downstream
release folks are already validating ToT enough. We need the distros
in as well.
Same goes for releases, but that process isn't clear, it needs to be.

The proposals were:
 - Have distros build all packages with ToT as often as possible, report bugs.
 - Do the same process on stable branches, at least once (RC1).
 - Downstream releases also need to acknowledge when a validation came
back green (instead of just report bugs).
 - All parties need to coordinate the process in the same place, for
example, a meta bug in bugzilla, and give their *ack* when things are
good for them.

The third big point was following changes on long running downstream
stable branches. Many releases/distros keep stable local branches for
many years, and have to back-port on their own or keep local patches
for that long.

The proposals were:
 - Better tracking of upstream patch streams, fixes for old stable
release bugs. Making a bug depend on an old release meta may help.
 - Distros and releases report keeping lots of local patches (some
already on trunk). Back-porting them to minor releases would help
everyone, keeping old releases for longer, too.
 - Create patch bundles and publish them somewhere (changelog?
bugzilla?) so that downstream knows all patches to back-port without
duplicating efforts.

Other interesting points were:
 - Moving versions for small projects / API users is not an easy task.
Having release numbers that make sense and multiple releases at a time
may help not introduce API changes to old releases.
 - Volume of changes is too great. Having a slow down (staged freeze)
may help developers invest more in stability just before the branch.
We sort of have that, needs formalisation.
 - If we have major bugs after the branch, we stop the whole process,
which makes it slower and more unpredictable. Feature freeze may help
with that, but can affect the branch date.
 - Volunteering is still required, but we could keep documented who
does what, and change the docs as that changes.
 - Upstreaming package maintenance scripts had mixed views, but I
still encourage those that want, to try.

cheers,
--renato