[llvm-dev] LLVM Releases: Upstream vs. Downstream / Distros

Jeff Law via llvm-dev
Fri, 13 May 2016 21:27:38 -0000

On 05/11/2016 08:08 AM, Renato Golin wrote:
>
> Shall we sync our upstream release with the bulk of other downstream
> ones as well as OS distributions?
I suspect that'll, in general, be hard.

The downstream consumers are all going to have their own needs &
schedules and as a result I doubt you'll find any upstream release
schedule & cadence that is acceptable to enough downstream consumers to
make this viable.

With that in mind, I'm a proponent of timing based schedules with as
much predictability as can be made.  That allows the downstream
consumers to make informed decisions based on likely release dates.


>
>
> This work involves a *lot* of premises that are not encoded yet, so
> we'll need a lot of work from all of us. But from the recent problems
> with GCC abi_tag and the arduous job of downstream release managers to
> know which patches to pick, I think there has been a lot of wasted
> effort by everyone, and that generates stress, conflicts, etc.
Ideally as we continue to open more lines of communication we won't run
into anything as problematical as the abi_tag stuff.  While it's
important, I wouldn't make it the primary driver for where you're trying
to go.

WRT downstream consumers.  The more the downstream consumer is wired
into the development community, the more risk (via patches) the
downstream consumer can reasonably take.  A downstream consumer without
intimate knowledge of the issues probably shouldn't be taking
incomplete/unapproved patches and applying them to their tree.


>
>   1. Timing
>
> Many downstream release managers, as well as distro maintainers have
> complained about the timing of our releases, and how unreliable they
> are, and how that makes it hard for them to plan their own branches,
> cherry-picks and merges. If we release too early, they miss out
> important optimisations, if we do too late, they'll have to branch
> "just before" and risk having to back-port late fixes to their
own
> modified trees.
And this just gets bigger as the project gets more downstream consumers.
    Thus I think you pick a time based release schedule, whatever it may
be and the downstream consumers can then adjust.

Note that this can have the effect of encouraging them to engage more
upstream to ensure issues of concern to them are addressed in a timely
manner.

>
>   2. Process
>
> Our release process is *very* lean, and that's what makes it
> quasi-chaotic. In the beginning, not many people / companies wanted to
> help or cared about the releases, so the process was what whomever was
> doing, did. The major release process is now better defined, but the
> same happened to the minor releases.
>
> For example, we have no defined date to start, or to end. We have no
> assigned people to do the official releases, or test the supported
> targets. We still rely on voluntary work from all parties. That's ok
> when the release is just "a point in time", but if downstream
releases
> and OS distributions start relying on our releases, we really should
> get a bit more professional.
Can't argue with getting a bit more structured, but watch out for going
too far.  I'd really like to squish down the release phase on the GCC
side, but it's damn hard at this point.


>
> A few (random) ideas:
>
>  * We should have predictable release times, both for starting it and
> finishing it. There will be complications, but we should treat them as
> the exception, not the rule.
Yes.

>  * We should have appointed members of the community that would be
> responsible for those releases, in the same way we have code owners
> (volunteers, but no less responsible), so that we can guarantee a
> consistent validation across all relevant targets. This goes beyond
> x86/ARM/MIPS/PPC and includes the other targets like AMD, NVidia, BPF,
> etc.
Good luck ;-)  Don't take this wrong, but wrangling volunteers into
release work is hard.  It's sometimes hard to find a way to motivate
them to focus on issues important for the release when there's new
development work they want to be doing.

Mark Mitchell found one good tool for that in his years as the GCC
release manager -- namely tightening what was allowed on the trunk as
the desired release date got closer.  ie, there's a free-for-all period,
then just bugfixes, then just regression fixes, then just doc fixes.
Developers then had a clear vested interest in moving the release
forward -- they couldn't commit their new development work until the
release manager opened the trunk for new development.


>  * OS distribution managers should test on their builds, too. I know
> FreeBSD and Mandriva build by default with Clang. I know that Debian
> has an experimental build. I know that RedHat and Ubuntu have LLVM
> packages that they do care. All that has to be tested *at least* every
> major release, but hopefully on all releases. (those who already do
> that, thank you!)
LLVM's usage on the Fedora side is still small, and smaller still within
Red Hat.  But we do have an interest in this stuff "just working".
Given current staffing levels I would expect Fedora to follow the
upstream releases closely with minimal changes.

>  * Every *new* bug found in any of those downstream tests should be
> reported in Bugzilla with the appropriate category (critical / major /
> minor). All major bugs have to be closed for the release to be out,
> etc. (the specific process will have to be agreed and documented).
Yes, GCC has a similar policy and it has worked reasonably well.  In
fact it's a good lever for the release manager if you've got a locked
trunk.  If the developers don't address the issues, then the release
doesn't branch and the trunk doesn't open for development.  It aligns
the release manager and a good chunk of the development team's goals.


jeff