llvm dev - May 2016 - LLVM Releases: Upstream vs. Downstream / Distros

FWIW, for our ARM Compiler product, we follow top-of-trunk, not the releases.
Next to picking up new functionality quicker, it also allows us to detect
regressions
in LLVM against our in-house testing quickly, not 6 months later. We find that
when
we find a regression within 24 to 48 hours of the commit introducing it,
it's much
cheaper to get it fixed.

In my opinion, it would be better overall for the LLVM project if top-of-trunk
is
tested as much as possible, if testing resources are so scarce that a choice
has to be made between testing top-of-trunk or testing a release branch.

Thanks,

Kristof

On 12 May 2016, at 02:10, Robinson, Paul <paul.robinson at
sony.com<mailto:paul.robinson at sony.com>> wrote:

-----Original Message-----
From: Renato Golin [mailto:renato.golin at linaro.org]
Sent: Wednesday, May 11, 2016 11:11 AM
To: Hans Wennborg
Cc: LLVM Dev; Clang Dev; Quentin Colombet; Tom Stellard; Robinson, Paul;
Jim Grosbach; Kristof Beyls; Frédéric Richez; Reid Kleckner; Philip
Reames; Matthias Braun; Bernhard Rosenkränzer; Sylvestre Ledru; Matthias
Klose; Stephen Hines; Jeff Law; Ed Maste; Behan Webster
Subject: Re: LLVM Releases: Upstream vs. Downstream / Distros

On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org<mailto:hans
at chromium.org>> wrote:
This is a long email :-) I've made some comments inline, but I'll
summarize my thoughts here:

Thanks Hans!

I'll respond them inline, below.


- I think we should use the bug tracker to capture issues that affect
releases. It would be cool if a commit hook could update bugzilla
entries that refer to it.

That seems like a simple hook.


At least for the major releases, I think we're doing pretty well on
timing in terms of predictability: since 3.6, we have release every
six months: first week of March and first week of September (+- a few
days). Branching has been similarly predictive: mid-January and
mid-July.

Indeed, we got a lot better more recently (last 2y), and mostly thanks
to you. :)

Absolutely.  It's enough of a track record to allow reasonable planning.


We used to vary 3 months +-, and now we're down to a few days.
Whatever we decide, I think we should make it official but putting it
out somewhere, so people can rely on that.

A public commitment to the future release schedule would be that much
more justification for planning to participate.


Right now, even if you're extra awesome, there's nothing telling the
distros and LLVM-based products that it will be so if someone else
takes over the responsibility, so they can't adapt.

That's what I meant by "quasi-chaotic".


If there are many downstream releases for which shifting this schedule
would be useful, I suppose we could do that, but it seems unlikely
that there would be agreement on this, and changing the schedule is
disruptive for those who depend on it.

That's the catch. If we want them to participate, the process has to
have some meaning to them. The fact that not many people do, is clear
to me what it means.

We also need to know better how many other releases already depend on
the upstream process (not just Chromium, for obvious reasons), to be
able to do an informed choice of dates and frequency.

The more well positioned and frequent we are, the more people will
help, but there's a point where the curve bends down, and the cost is
just too great. We need to find the inflection point, and that will
require some initial investigations and guesses, and a lot of fine
tuning later. But if we're all on the same page, I think we can do
that, even if it takes time.

I'm particularly concerned with Android, because they not only have
their own tree with heavily modified LLVM components (ex.
Compiler-RT), but they also build differently and so their process are
completely alien to ours. One of the key reasons why these things
happened is because:

* They couldn't rely on our releases, as fixing bugs and back-porting
wasn't a thing back then
* They already had their own release schedule, so aligning with ours
brought no extra benefit
* We always expected people to work off trunk, and everyone had to
create their own process

I don't want to change how people work, just to add one more valid way
of working, which is most stable for upstream releases. :)



The only reasonable way I see of aligning upstream releases with
downstream schedules would be to release much more often. This works
well in Chromium where there's a 6-week staged release schedule. This
would mean there's always a branch going for the next release, and
important bug fixes would get merged to that.

Full validation every 6 weeks is just not possible. But a multiple of
that, say every 3~4 months, could be much easier to work around.



In Chromium we drive
this from the bug tracker -- it would be very hard to scan each commit
for things to cherry-pick. This kind of process has a high cost
though, there has to be good infrastructure for it (buildbots on the
branch for all targets, for example), developers have to be aware, and
even then it's a lot of work for those doing the releases. I'm not
sure we'd want to take this on. I'm also not sure it would be suitable
for a compiler, where we want the releases to have long life-time.

This works because you have a closed system. As you say, Chromium is
mostly final product, not a tool to develop other products, and the
validation is a lot simpler.

With Clang, we'd want to involve external releases into it, and it
simply wouldn't scale.



For the major releases, I've tried to do this. We could certainly
formalize it by posting it on the web page though.

I think that'd be the first step, yes. But I wanted to start with a
good number. 2 times a year? Would 3 times improve things that much
for the outsiders? Or just moving the dates would be enough for most
people?

That's why I copied so many outsiders, so they can chime in and let us
know what would be good for *them*.

Data point: At Sony we ship our stuff every 6 months and that isn't
going to change.  Our policy has been to base our releases on upstream
releases, and given our current lead times, the current upstream release
schedule is actually not bad, especially if it has stopped drifting.
Having four releases pretty consistently follow the current schedule is
extremely positive, thanks!

It would help our internal planning to publish the schedule for future
releases.  You never know, we might even be able to fork an internal
branch in time to help with the release testing, although that is not
in any way a lightweight process (so no promises).

There has been some talk about moving toward releasing from upstream
trunk, or more precisely to streamlining our internal testing process
in order to allow us to seriously contemplate releasing from upstream
trunk.  I can't argue with the streamlining part, for sure.  Whether
the rest of it works out must remain to be seen.
--paulr



Most importantly, those folks should get involved :-)

Indeed!


In practice, we kind of have this for at least some of the targets.
Maybe we should write this down somewhere instead of me asking for
(the same) volunteers each time the release process starts?

I give consent to mark me as the ARM/AArch64 release tester for the
foreseeable future. :)

I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
system running on ARM/AArch64 hardware.

cheers,
--renato

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/fe24d394/attachment.html>

On 12 May 2016 at 15:56, Kristof Beyls <Kristof.Beyls at arm.com>
wrote:
> We find that when we find a regression within 24 to 48 hours of the commit
introducing it,
> it's much cheaper to get it fixed.
Hi Kristof,

Indeed, that is true.

But there are so much additional (internal) buildbots can do. If you
have any additional validation steps that you only do when you pick a
candidate for release, and there are failures in that set, you won't
see them during the trunk monitoring.

Also, distributions don't have as many compiler developers as most
groups releasing toolchains, and they have to rely on the public
buildbots (which are by no means comprehensive).

This is not about downstream *development*, but downstream *release*
process, which in most situations, require additional validation.

Following trunk for your development process is cheaper, as you said,
and most people agree. But if you release your product a few weeks
after the release is out, it may be better to get the release itself,
since it has been validated but many other groups (assuming everyone
join in), than trunk.

> In my opinion, it would be better overall for the LLVM project if
top-of-trunk is
> tested as much as possible, if testing resources are so scarce that a
choice
> has to be made between testing top-of-trunk or testing a release branch.
That's the balance I'm trying to get right. :)

There's also the other topic about the community.

LLVM is mostly a tool kit for compilers, yes, and most LLVM developers
are using it as such. But more and more projects are depending on the
actual compiler solution (Clang+LLVM+RT), and by the reaction of most
distros I've spoken to, we could improve our OSS community
interactions quite a lot.

I can certainly understand why most companies are worried about their
own processes, but that's undermining the ability of the OSS release
to achieve it's goal, which is to be used to build all kinds of
software in the wild.

We have FreeBSD, Mandriva and Android using it by default, which is a
*big* win. We have Debian, RedHat and Canonical worried about the
integration of LLVM in their packages, that's awesome.

I just want us to look at that picture, and see if we can do anything
better for them, that will in turn, make our processes slightly
cheaper because of the synergy it will create.

cheers,
--renato

Just some relatively quick thoughts, since I am still on vacation. +Luis
from ChromeOS toolchain.

On Thu, May 12, 2016 at 7:56 AM, Kristof Beyls <Kristof.Beyls at arm.com>
wrote:
> FWIW, for our ARM Compiler product, we follow top-of-trunk, not the
> releases.
> Next to picking up new functionality quicker, it also allows us to detect
> regressions
> in LLVM against our in-house testing quickly, not 6 months later. We find
> that when
> we find a regression within 24 to 48 hours of the commit introducing it,
> it's much
> cheaper to get it fixed.
>
> In my opinion, it would be better overall for the LLVM project if
> top-of-trunk is
> tested as much as possible, if testing resources are so scarce that a
> choice
> has to be made between testing top-of-trunk or testing a release branch.
>
I am 100% in agreement here. TOT regressions cost more than people think. I
have seen firsthand how this hurts the non-LLVM parts of Android. In
Android, more testing resources are devoted to release branches, and thus
the development branch picks up bizarre regressions that are harder to
track down and fix than they should be. Since I don't update the compiler
in the release branch (at least not without good reason), this means that
compiler regressions are harder for me to track down, because nobody pays
attention to dev results until it is hurting them in the next release. :( I
have some ideas to make this better in the future, but nothing to share
just yet.

>
> Thanks,
>
> Kristof
>
> On 12 May 2016, at 02:10, Robinson, Paul <paul.robinson at sony.com>
wrote:
>
> -----Original Message-----
> From: Renato Golin [mailto:renato.golin at linaro.org
> <renato.golin at linaro.org>]
> Sent: Wednesday, May 11, 2016 11:11 AM
> To: Hans Wennborg
> Cc: LLVM Dev; Clang Dev; Quentin Colombet; Tom Stellard; Robinson, Paul;
> Jim Grosbach; Kristof Beyls; Frédéric Richez; Reid Kleckner; Philip
> Reames; Matthias Braun; Bernhard Rosenkränzer; Sylvestre Ledru; Matthias
> Klose; Stephen Hines; Jeff Law; Ed Maste; Behan Webster
> Subject: Re: LLVM Releases: Upstream vs. Downstream / Distros
>
> On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org> wrote:
>
> This is a long email :-) I've made some comments inline, but I'll
> summarize my thoughts here:
>
>
> Thanks Hans!
>
> I'll respond them inline, below.
>
>
> - I think we should use the bug tracker to capture issues that affect
> releases. It would be cool if a commit hook could update bugzilla
> entries that refer to it.
>
>
> That seems like a simple hook.
>
>
> At least for the major releases, I think we're doing pretty well on
> timing in terms of predictability: since 3.6, we have release every
> six months: first week of March and first week of September (+- a few
> days). Branching has been similarly predictive: mid-January and
> mid-July.
>
>
> Indeed, we got a lot better more recently (last 2y), and mostly thanks
> to you. :)
>
>
> Absolutely.  It's enough of a track record to allow reasonable
planning.
>
>
> We used to vary 3 months +-, and now we're down to a few days.
> Whatever we decide, I think we should make it official but putting it
> out somewhere, so people can rely on that.
>
>
> A public commitment to the future release schedule would be that much
> more justification for planning to participate.
>
>
> Right now, even if you're extra awesome, there's nothing telling
the
> distros and LLVM-based products that it will be so if someone else
> takes over the responsibility, so they can't adapt.
>
> That's what I meant by "quasi-chaotic".
>
>
> If there are many downstream releases for which shifting this schedule
> would be useful, I suppose we could do that, but it seems unlikely
> that there would be agreement on this, and changing the schedule is
> disruptive for those who depend on it.
>
>
> That's the catch. If we want them to participate, the process has to
> have some meaning to them. The fact that not many people do, is clear
> to me what it means.
>
> We also need to know better how many other releases already depend on
> the upstream process (not just Chromium, for obvious reasons), to be
> able to do an informed choice of dates and frequency.
>
> The more well positioned and frequent we are, the more people will
> help, but there's a point where the curve bends down, and the cost is
> just too great. We need to find the inflection point, and that will
> require some initial investigations and guesses, and a lot of fine
> tuning later. But if we're all on the same page, I think we can do
> that, even if it takes time.
>
> I'm particularly concerned with Android, because they not only have
> their own tree with heavily modified LLVM components (ex.
> Compiler-RT), but they also build differently and so their process are
> completely alien to ours. One of the key reasons why these things
> happened is because:
>
>
I am not sure why you think Android's compiler-rt is an example of a
"heavily modified" component. As I see it, our compiler-rt matches
upstream
almost exactly (with one minor mistake from a duplicate merge that results
in extra copies of some static functions that we don't even build). We do
have 3 cherry-picks for some MIPS ASan patches, but all of those come
directly from TOT master.

There are some Android-only patches in LLVM and Clang (thanks to
RenderScript, and not being able to easily upstream these - i.e. the RS
frontend doesn't live as an LLVM upstream project, so it is hard to write
tests for these modifications). None of those patches should impact things
greatly. Assuming you sync to the same CL (plus <10 cherry-picks to all
projects), any C/C++ regression should be reproducible with a normal TOT
build. This has been true since we did upstream our most divergent patch
(re: calling conventions for ARM vectors) a few months ago.

>
> * They couldn't rely on our releases, as fixing bugs and back-porting
> wasn't a thing back then
>
> The real problem is retaining history. Release branches don't make this
very nice, and necessitate that we swap out an entire chunk of history for
a different chunk of history every time we change releases. That is pretty
obnoxious for keeping a good idea of what has happened, and thus following
master makes this easier for us (and I assume others too). We did try using
a release branch back for LLVM 3.5, but the maintenance cost for then
updating things when we wanted to move past 3.5 showed me why I don't want
to really look

> * They already had their own release schedule, so aligning with ours
> brought no extra benefit
>
> It is not 100% clear that Android will want to be dependent on LLVM's
release schedule. I think that there are definitely benefits to having
everyone do extra validation, but I am unconvinced that it is the
"same"
validation for everyone that is valuable, hence this might not make things
go much smoother/faster for those groups.

Thanks,
Steve
> * We always expected people to work off trunk, and everyone had to
> create their own process
>
> I don't want to change how people work, just to add one more valid way
> of working, which is most stable for upstream releases. :)
>
>
>
> The only reasonable way I see of aligning upstream releases with
> downstream schedules would be to release much more often. This works
> well in Chromium where there's a 6-week staged release schedule. This
> would mean there's always a branch going for the next release, and
> important bug fixes would get merged to that.
>
>
> Full validation every 6 weeks is just not possible. But a multiple of
> that, say every 3~4 months, could be much easier to work around.
>
>
>
> In Chromium we drive
> this from the bug tracker -- it would be very hard to scan each commit
> for things to cherry-pick. This kind of process has a high cost
> though, there has to be good infrastructure for it (buildbots on the
> branch for all targets, for example), developers have to be aware, and
> even then it's a lot of work for those doing the releases. I'm not
> sure we'd want to take this on. I'm also not sure it would be
suitable
> for a compiler, where we want the releases to have long life-time.
>
>
> This works because you have a closed system. As you say, Chromium is
> mostly final product, not a tool to develop other products, and the
> validation is a lot simpler.
>
> With Clang, we'd want to involve external releases into it, and it
> simply wouldn't scale.
>
>
>
> For the major releases, I've tried to do this. We could certainly
> formalize it by posting it on the web page though.
>
>
> I think that'd be the first step, yes. But I wanted to start with a
> good number. 2 times a year? Would 3 times improve things that much
> for the outsiders? Or just moving the dates would be enough for most
> people?
>
> That's why I copied so many outsiders, so they can chime in and let us
> know what would be good for *them*.
>
>
> Data point: At Sony we ship our stuff every 6 months and that isn't
> going to change.  Our policy has been to base our releases on upstream
> releases, and given our current lead times, the current upstream release
> schedule is actually not bad, especially if it has stopped drifting.
> Having four releases pretty consistently follow the current schedule is
> extremely positive, thanks!
>
> It would help our internal planning to publish the schedule for future
> releases.  You never know, we might even be able to fork an internal
> branch in time to help with the release testing, although that is not
> in any way a lightweight process (so no promises).
>
> There has been some talk about moving toward releasing from upstream
> trunk, or more precisely to streamlining our internal testing process
> in order to allow us to seriously contemplate releasing from upstream
> trunk.  I can't argue with the streamlining part, for sure.  Whether
> the rest of it works out must remain to be seen.
> --paulr
>
>
>
> Most importantly, those folks should get involved :-)
>
>
> Indeed!
>
>
> In practice, we kind of have this for at least some of the targets.
> Maybe we should write this down somewhere instead of me asking for
> (the same) volunteers each time the release process starts?
>
>
> I give consent to mark me as the ARM/AArch64 release tester for the
> foreseeable future. :)
>
> I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
> system running on ARM/AArch64 hardware.
>
> cheers,
> --renato
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/d068e881/attachment.html>

On 12 May 2016 at 16:56, Kristof Beyls <Kristof.Beyls at arm.com> wrote:
> In my opinion, it would be better overall for the LLVM project if
> top-of-trunk is
> tested as much as possible, if testing resources are so scarce that a
> choice
> has to be made between testing top-of-trunk or testing a release branch.
>
I agree that trunk is more important, with both of my hats on.

But releases are not completely irrelevant - one thing making them
important is the fact that there's other projects out there using the LLVM
libraries - and as a distro, we have to make sure they all work (so they
agree on the same API), preferably without having to ship multiple versions
of LLVM and preferably without having to patch external code too much to
adjust to API changes.

In OpenMandriva, we have to keep Mesa, creduce, emscripten and the
LLVMified Qt moc working (list expected to grow -- ultimately we'd also
like to use the system LLVM libraries for the swift compiler).
In AOSP, RenderScript relies on the LLVM API, but there's nothing else
using it, so there's currently no need to force a common version of the API
between different projects there.

ttyl
bero
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/05590a18/attachment.html>

On 12 May 2016, at 16:27, Bernhard Rosenkränzer via cfe-dev <cfe-dev at
lists.llvm.org> wrote:
> 
> On 12 May 2016 at 16:56, Kristof Beyls <Kristof.Beyls at arm.com>
wrote:
>> In my opinion, it would be better overall for the LLVM project if
top-of-trunk is
>> tested as much as possible, if testing resources are so scarce that a
choice
>> has to be made between testing top-of-trunk or testing a release
branch.
>> 
> I agree that trunk is more important, with both of my hats on.
> 
> But releases are not completely irrelevant - one thing making them
important is the fact that there's other projects out there using the LLVM
libraries - and as a distro, we have to make sure they all work (so they agree
on the same API), preferably without having to ship multiple versions of LLVM
and preferably without having to patch external code too much to adjust to API
changes.
> 
> In OpenMandriva, we have to keep Mesa, creduce, emscripten and the
LLVMified Qt moc working (list expected to grow -- ultimately we'd also like
to use the system LLVM libraries for the swift compiler).
> In AOSP, RenderScript relies on the LLVM API, but there's nothing else
using it, so there's currently no need to force a common version of the API
between different projects there.
I think that our API stability policy is really hurting us here.  Downstream
users of clang have few problems - clang is basically backwards compatible and
so you only need to test for regressions.  We periodically build the entire
FreeBSD ports collection (around 25,000 open source packages) with the
clang-devel port (which is a periodically updated trunk snapshot) and report
regressions.  That’s very easy to do.  It’s easy for me to test for clang
regressions by just setting CC=clang-devel CXX=clang++-devel, rebuilding my own
projects and running their test suites.

In contrast, for projects that I maintain that use LLVM as a library, I
generally hop from release to release.  We’ve had several instances of APIs
changing multiple times between releases, so updating the code to each one is
more effort than just updating to the newest one.  APIs often go in long before
documentation (which is often nonexistent or rushed in for the release), so
you’re more likely to see useful documentation if you pick a release.  Finally,
most distributions package releases so if you depend on LLVM X.Y then your code
is easy to package, whereas if you depend on LLVM svn rxzy then it’s basically
impossible.  Even between releases, expecting everyone who might use / test your
project’s trunk to build the specific LLVM revision that you depend on is too
much of a barrier to entry for many potential contributors and so following
trunk reduces the number of contributors and the amount of testing.

The end result is that shortly after a release (sometimes every alternate
release) is branched a load of downstream projects update to the new APIs, test
things, and find a bunch of regressions that have been sitting in the tree for
months.  We then have to scrabble to bisect and try to track them down.

TL;DR version: If we want downstream people to test ToT, then we need to make
updating LLVM library consumers to ToT far less painful than it is now.

David


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/4138a442/attachment.bin>

On 12 May 2016 at 16:22, Stephen Hines <srhines at google.com>
wrote:
> I am 100% in agreement here. TOT regressions cost more than people think. I
> have seen firsthand how this hurts the non-LLVM parts of Android.
Hi Steve,

I think we're all in agreement of ToT development and testing. This is
more about releases and upstream users, including OS distributions.

> I am not sure why you think Android's compiler-rt is an example of a
> "heavily modified" component. As I see it, our compiler-rt
matches upstream
> almost exactly (with one minor mistake from a duplicate merge that results
> in extra copies of some static functions that we don't even build). We
do
> have 3 cherry-picks for some MIPS ASan patches, but all of those come
> directly from TOT master.
Sorry, that was a bit heavy-handed... I meant that it's hard to change
the Android's copy of RT because of how it's built.

This patch is an example: https://android-review.googlesource.com/#/c/125910/1

It introduces a set of nice changes that cannot go as in into LLVM's
compiler-rt because of how RT is built in LLVM.

This is not Android's fault per se, but it's an example of how
proliferation of patches can happen if one downstream repo depends on
another, as is the case for AOSP+Android+(anyone else that develops
Android).

That change should have been developed on upstream RT to begin with,
but the merge would be hard to control on the third-generation copy.

That's the reason why I want to bring all downstream repos to only
depend on upstream LLVM.

> The real problem is retaining history. Release branches don't make this
very
> nice, and necessitate that we swap out an entire chunk of history for a
> different chunk of history every time we change releases.
That's interesting... I haven't heard that before, so I don't know
exactly what you mean. :)

Can you give an example?

> It is not 100% clear that Android will want to be dependent on LLVM's
> release schedule. I think that there are definitely benefits to having
> everyone do extra validation, but I am unconvinced that it is the
"same"
> validation for everyone that is valuable, hence this might not make things
> go much smoother/faster for those groups.
That's a very good point, and one that I was considering while
thinking about this.

Will Sony's validation be any relevant to Chromium builds? Probably
not. But ARM's extra validation will be relevant to anyone using ARM
targets, and that in turn will bring more adoption to ARM. Sony's
validation, will be interesting to the CPUs and GPUs they validate on,
so that's a bonus to anyone who uses the same hardware.

I can't put a concrete value on any of this, but having people chiming
in is the best way to know what kind of things people do, and how
valuable they are to each other.

cheers,
--renato

I'll bite:

Do you really believe you could ever get these folks to choose consistent
llvm versions anyway?

Or will you always have to do the work yourself anyway?

In my talks with a number of these projects, they pretty much don't care
what anyone else does, and plan to stick to their own import/etc schedules
no matter what LLVM does with stable releases :)

(For reference, Google *ships* the equivalent of about 13-16 linux
distributions in products, uses about 5-6x that internally, and we have a
single monolithic source repository for the most part.  I have the joy of
owning the third party software policies/etc for it, and so  end up
responsible for trying to deal with maintaining single versions of llvm for
tens to hundreds of packages).


On Thu, May 12, 2016 at 8:27 AM, Bernhard Rosenkränzer <
llvm-dev at lists.llvm.org> wrote:
> On 12 May 2016 at 16:56, Kristof Beyls <Kristof.Beyls at arm.com>
wrote:
>
>> In my opinion, it would be better overall for the LLVM project if
>> top-of-trunk is
>> tested as much as possible, if testing resources are so scarce that a
>> choice
>> has to be made between testing top-of-trunk or testing a release
branch.
>>
>
> I agree that trunk is more important, with both of my hats on.
>
> But releases are not completely irrelevant - one thing making them
> important is the fact that there's other projects out there using the
LLVM
> libraries - and as a distro, we have to make sure they all work (so they
> agree on the same API), preferably without having to ship multiple versions
> of LLVM and preferably without having to patch external code too much to
> adjust to API changes.
>
> In OpenMandriva, we have to keep Mesa, creduce, emscripten and the
> LLVMified Qt moc working (list expected to grow -- ultimately we'd also
> like to use the system LLVM libraries for the swift compiler).
> In AOSP, RenderScript relies on the LLVM API, but there's nothing else
> using it, so there's currently no need to force a common version of the
API
> between different projects there.
>
> ttyl
> bero
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160512/6d7287db/attachment.html>

> On May 12, 2016, at 7:56 AM, Kristof Beyls <kristof.beyls at arm.com>
wrote:
> 
> FWIW, for our ARM Compiler product, we follow top-of-trunk, not the
releases.
> Next to picking up new functionality quicker, it also allows us to detect
regressions
> in LLVM against our in-house testing quickly, not 6 months later. We find
that when
> we find a regression within 24 to 48 hours of the commit introducing it,
it's much
> cheaper to get it fixed.
> 
> In my opinion, it would be better overall for the LLVM project if
top-of-trunk is
> tested as much as possible, if testing resources are so scarce that a
choice
> has to be made between testing top-of-trunk or testing a release branch.
+1 to everything Kristof said.

I will also add to the discussion that I believe it does not make sense to try
to align releases with open source in general because different people have
different goals. For instance, when we (the open source community) branch for a
release, we try to make the branch as stable as possible and only pull-in fixes.
Other people may have different goal, like it is possible a new feature was
expected for that release and that we need to pull that in. And we are back to
the problem we were discussing in the other thread: helping release manager for
general commits.

Cheers,
-Quentin
> 
> Thanks,
> 
> Kristof
> 
>> On 12 May 2016, at 02:10, Robinson, Paul <paul.robinson at sony.com
<mailto:paul.robinson at sony.com>> wrote:
>> 
>>> -----Original Message-----
>>> From: Renato Golin [mailto:renato.golin at linaro.org
<mailto:renato.golin at linaro.org>]
>>> Sent: Wednesday, May 11, 2016 11:11 AM
>>> To: Hans Wennborg
>>> Cc: LLVM Dev; Clang Dev; Quentin Colombet; Tom Stellard; Robinson,
Paul;
>>> Jim Grosbach; Kristof Beyls; Frédéric Richez; Reid Kleckner; Philip
>>> Reames; Matthias Braun; Bernhard Rosenkränzer; Sylvestre Ledru;
Matthias
>>> Klose; Stephen Hines; Jeff Law; Ed Maste; Behan Webster
>>> Subject: Re: LLVM Releases: Upstream vs. Downstream / Distros
>>> 
>>> On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org
<mailto:hans at chromium.org>> wrote:
>>>> This is a long email :-) I've made some comments inline,
but I'll
>>>> summarize my thoughts here:
>>> 
>>> Thanks Hans!
>>> 
>>> I'll respond them inline, below.
>>> 
>>> 
>>>> - I think we should use the bug tracker to capture issues that
affect
>>>> releases. It would be cool if a commit hook could update
bugzilla
>>>> entries that refer to it.
>>> 
>>> That seems like a simple hook.
>>> 
>>> 
>>>> At least for the major releases, I think we're doing pretty
well on
>>>> timing in terms of predictability: since 3.6, we have release
every
>>>> six months: first week of March and first week of September (+-
a few
>>>> days). Branching has been similarly predictive: mid-January and
>>>> mid-July.
>>> 
>>> Indeed, we got a lot better more recently (last 2y), and mostly
thanks
>>> to you. :)
>> 
>> Absolutely.  It's enough of a track record to allow reasonable
planning.
>> 
>>> 
>>> We used to vary 3 months +-, and now we're down to a few days.
>>> Whatever we decide, I think we should make it official but putting
it
>>> out somewhere, so people can rely on that.
>> 
>> A public commitment to the future release schedule would be that much
>> more justification for planning to participate.
>> 
>>> 
>>> Right now, even if you're extra awesome, there's nothing
telling the
>>> distros and LLVM-based products that it will be so if someone else
>>> takes over the responsibility, so they can't adapt.
>>> 
>>> That's what I meant by "quasi-chaotic".
>>> 
>>> 
>>>> If there are many downstream releases for which shifting this
schedule
>>>> would be useful, I suppose we could do that, but it seems
unlikely
>>>> that there would be agreement on this, and changing the
schedule is
>>>> disruptive for those who depend on it.
>>> 
>>> That's the catch. If we want them to participate, the process
has to
>>> have some meaning to them. The fact that not many people do, is
clear
>>> to me what it means.
>>> 
>>> We also need to know better how many other releases already depend
on
>>> the upstream process (not just Chromium, for obvious reasons), to
be
>>> able to do an informed choice of dates and frequency.
>>> 
>>> The more well positioned and frequent we are, the more people will
>>> help, but there's a point where the curve bends down, and the
cost is
>>> just too great. We need to find the inflection point, and that will
>>> require some initial investigations and guesses, and a lot of fine
>>> tuning later. But if we're all on the same page, I think we can
do
>>> that, even if it takes time.
>>> 
>>> I'm particularly concerned with Android, because they not only
have
>>> their own tree with heavily modified LLVM components (ex.
>>> Compiler-RT), but they also build differently and so their process
are
>>> completely alien to ours. One of the key reasons why these things
>>> happened is because:
>>> 
>>> * They couldn't rely on our releases, as fixing bugs and
back-porting
>>> wasn't a thing back then
>>> * They already had their own release schedule, so aligning with
ours
>>> brought no extra benefit
>>> * We always expected people to work off trunk, and everyone had to
>>> create their own process
>>> 
>>> I don't want to change how people work, just to add one more
valid way
>>> of working, which is most stable for upstream releases. :)
>>> 
>>> 
>>> 
>>>> The only reasonable way I see of aligning upstream releases
with
>>>> downstream schedules would be to release much more often. This
works
>>>> well in Chromium where there's a 6-week staged release
schedule. This
>>>> would mean there's always a branch going for the next
release, and
>>>> important bug fixes would get merged to that.
>>> 
>>> Full validation every 6 weeks is just not possible. But a multiple
of
>>> that, say every 3~4 months, could be much easier to work around.
>>> 
>>> 
>>> 
>>>> In Chromium we drive
>>>> this from the bug tracker -- it would be very hard to scan each
commit
>>>> for things to cherry-pick. This kind of process has a high cost
>>>> though, there has to be good infrastructure for it (buildbots
on the
>>>> branch for all targets, for example), developers have to be
aware, and
>>>> even then it's a lot of work for those doing the releases.
I'm not
>>>> sure we'd want to take this on. I'm also not sure it
would be suitable
>>>> for a compiler, where we want the releases to have long
life-time.
>>> 
>>> This works because you have a closed system. As you say, Chromium
is
>>> mostly final product, not a tool to develop other products, and the
>>> validation is a lot simpler.
>>> 
>>> With Clang, we'd want to involve external releases into it, and
it
>>> simply wouldn't scale.
>>> 
>>> 
>>> 
>>>> For the major releases, I've tried to do this. We could
certainly
>>>> formalize it by posting it on the web page though.
>>> 
>>> I think that'd be the first step, yes. But I wanted to start
with a
>>> good number. 2 times a year? Would 3 times improve things that much
>>> for the outsiders? Or just moving the dates would be enough for
most
>>> people?
>>> 
>>> That's why I copied so many outsiders, so they can chime in and
let us
>>> know what would be good for *them*.
>> 
>> Data point: At Sony we ship our stuff every 6 months and that isn't
>> going to change.  Our policy has been to base our releases on upstream
>> releases, and given our current lead times, the current upstream
release
>> schedule is actually not bad, especially if it has stopped drifting.
>> Having four releases pretty consistently follow the current schedule is
>> extremely positive, thanks!
>> 
>> It would help our internal planning to publish the schedule for future
>> releases.  You never know, we might even be able to fork an internal
>> branch in time to help with the release testing, although that is not
>> in any way a lightweight process (so no promises).
>> 
>> There has been some talk about moving toward releasing from upstream
>> trunk, or more precisely to streamlining our internal testing process
>> in order to allow us to seriously contemplate releasing from upstream
>> trunk.  I can't argue with the streamlining part, for sure. 
Whether
>> the rest of it works out must remain to be seen.
>> --paulr
>> 
>>> 
>>> 
>>>> Most importantly, those folks should get involved :-)
>>> 
>>> Indeed!
>>> 
>>> 
>>>> In practice, we kind of have this for at least some of the
targets.
>>>> Maybe we should write this down somewhere instead of me asking
for
>>>> (the same) volunteers each time the release process starts?
>>> 
>>> I give consent to mark me as the ARM/AArch64 release tester for the
>>> foreseeable future. :)
>>> 
>>> I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on
their
>>> system running on ARM/AArch64 hardware.
>>> 
>>> cheers,
>>> --renato
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160513/fe58e44e/attachment.html>