llvm dev - Apr 2013 - [LLVMdev] RFC: Bug fix releases for 3.3 and beyond

On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling
wrote:
> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at stellard.net> wrote:
> 
> > Hi,
> > 
> > I would really like to see the LLVM project start to make official bug
fix
> > releases (e.g. 3.3.1, 3.3.2, etc.).  I think that this would be useful
for a
> > lot of the users of LLVM, especially projects that use LLVM as a
library.
> > I am willing to help maintain bug fix releases, and I'm wondering
if
> > this is something that the LLVM project would officially support with
> > a stable SVN branch and by hosting the official stable tarball
releases.
> > 
> > I realize that maintaining stable branches is a lot of work, so I
would
> > like to come up with a procedure that makes maintaining these branches
> > as easy as possible.  Here is a rough idea of what I had in
> > mind, but please suggest alternatives if you know of a better way:
> > 
> > 1. Developer fixes a bug or makes a change that he/she thinks would
make
> > a good candidate for the stable branch.  Commits would require
approval
> > from the Code Owner in order to be backported to stable.
> > 
> > 2a. When the developer commits that change, he/she adds to the end of
the
> > commit message something like:
> > 
> > Note: This is a candidate for the stable branch
> > 
> > 2b. Alternatively, if a user discovers a bug in a stable release that
has
> > been fixed in ToT, he/she could request to have the fix backported.
> > 
> > 3. The developer would be encouraged, but not required to cherry-pick
the
> > commit to the stable branch.  The stable maintainer would periodically
> > search the commit logs and cherry-pick any commits that had been
missed,
> > consulting with the author of the commit in the case of a difficult
> > merge conflict.
> > 
> > 4. After some interval of time, the stable maintainer would announce
> > plans for a stable release and testing would begin.
> > 
> > What does everyone think?  Would something like this be doable?
> > 
> As Chris said, the only thing preventing this is manpower. But if there are
people ready and willing to do this, then I don't see it as a Bad Thing.
> 
> My first comment is that these should be strictly bug fix releases. So your
(1) above wouldn't include changes other than bug fixes. There would need to
be a hierarchy for bugs. The reason is that *any* change to the stable branch
inherently carries risk. So for instance, a bug that's a crasher, but would
affect only a small number of people may not be worthy of inclusion into a
dot-release. Etc.
> 
I was thinking for shared code it would be strictly bug fix only, but
maybe things like additional C API implementations or standalone passes
might be OK too assuming there is interest.

However, for target specific code, I think backend code owners should
have a little more leeway as far as what gets accepted.  For the
R600 backend I can see a situation where I may want to backport a new
feature or something else, so it would be nice to have a little more
flexibility and maybe other backend owners would want to have this
option as well.
> My second comment is that top-of-tree moves very fast. This can make
changes hard to back-port to a stable branch that may be months old. You are
stuck having to either do it yourself, or begging the original author to
back-port the fix. :-)
> 
I understand.  I think if there is a patch that is too difficult for the
release maintainer to backport he/she should be able to go to the author
and request help.  If the author or some other party doesn't care enough
about the fix to help backport it, then it probably isn't important
enough to go to stable.
> How frequently do you expect to do a dot-release? Our major release
schedule is roughly every 6 months. Do you think bimonthly would be too many? or
would you do it only when enough changes require it?
> 
This is hard to say, but probably as required.
> Finally, there is testing to consider. Obviously, full testing would be too
rough on the community; people simply don't have enough time to spend a full
month testing a dot-release. What are you thoughts on how testing would proceed?
> 
What is it about testing for releases that takes a month to complete?
Is it just coordinating with all the interested parties and making sure
they are happy with the state of the code?

-Tom

On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote:
> On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote:
>> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at stellard.net>
wrote:
>> 
>> As Chris said, the only thing preventing this is manpower. But if there
are people ready and willing to do this, then I don't see it as a Bad Thing.
>> 
>> My first comment is that these should be strictly bug fix releases. So
your (1) above wouldn't include changes other than bug fixes. There would
need to be a hierarchy for bugs. The reason is that *any* change to the stable
branch inherently carries risk. So for instance, a bug that's a crasher, but
would affect only a small number of people may not be worthy of inclusion into a
dot-release. Etc.
>> 
> 
> I was thinking for shared code it would be strictly bug fix only, but
> maybe things like additional C API implementations or standalone passes
> might be OK too assuming there is interest.
> 
> However, for target specific code, I think backend code owners should
> have a little more leeway as far as what gets accepted.  For the
> R600 backend I can see a situation where I may want to backport a new
> feature or something else, so it would be nice to have a little more
> flexibility and maybe other backend owners would want to have this
> option as well.
If we are going to go this route, I would be *very* selective about taking
non-bug fixes. Target specific enhancements to our non-"tier 1"
platforms ("tier 1" being x86 and ARM) would depend upon that back-end
owner's judgement. However for the "tier 1" platforms, I will
continue to be very skeptical. Adding new C APIs is challenging, because once we
release with a C API it's there forever. Of course, there will be a suitable
amount of bake time, etc.

Basically, I will be very skeptical about any changes that will go into a
dot-release. The two questions, which I would want answered, are "why is
this change important enough to warrant adding it to a new release?" and
"how safe is this change?".
>> My second comment is that top-of-tree moves very fast. This can make
changes hard to back-port to a stable branch that may be months old. You are
stuck having to either do it yourself, or begging the original author to
back-port the fix. :-)
> 
> I understand.  I think if there is a patch that is too difficult for the
> release maintainer to backport he/she should be able to go to the author
> and request help.  If the author or some other party doesn't care
enough
> about the fix to help backport it, then it probably isn't important
> enough to go to stable.
> 
Fair enough. (And that's exactly what should happen. :-) )
>> How frequently do you expect to do a dot-release? Our major release
schedule is roughly every 6 months. Do you think bimonthly would be too many? or
would you do it only when enough changes require it?
> 
> This is hard to say, but probably as required.
> 
>> Finally, there is testing to consider. Obviously, full testing would be
too rough on the community; people simply don't have enough time to spend a
full month testing a dot-release. What are you thoughts on how testing would
proceed?
> 
> What is it about testing for releases that takes a month to complete?
> Is it just coordinating with all the interested parties and making sure
> they are happy with the state of the code?
> 
Pretty much. Basically, we allow for a two week test and fix sequence. The
testers are given one week to test the newest release candidate. We could give
them a shorter amount of time, but that may put too much pressure on our
volunteers. (In actuality, most testers are finished quickly.) We post the
binaries they build so that the community can test on their software. We then
give developers a week to address any issues that came up during the testing
phase. This cycle is repeated once (twice if something disastrous comes up in
the second cycle).

Two weeks is more or less an arbitrary choice, but one which allows us ample
time to make sure that the release is as good as can be.

That said, there are a couple of options that we could consider:

1) Shorten the release schedule for a dot-release.

  - Considering that we are working with a known good base, we could reasonably
expect fewer, if any, regressions. So we wouldn't have to worry as much
about allocating so much time for fixing bugs. Plus, those who created the bad
patch should be able to fix it fairly quickly.

2) Don't test quite as much, and tag the dot-release as "beta" or
some other weasel wordy thing.

  - I'm not a big fan of this (even though I proposed it). I want external
developers to be confident with the releases we send out.

I'm sure there are other choices here. But I think that the greatest hurdle
we will come across is finding the resources to do the testing.

-bw

For the record, I'm not opposed to this. I'm mostly playing devil's
advocate. But I do feel that these are issues that need to be thought out before
we continue.

-bw

On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote:
> On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote:
>> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at stellard.net>
wrote:
>> 
>>> Hi,
>>> 
>>> I would really like to see the LLVM project start to make official
bug fix
>>> releases (e.g. 3.3.1, 3.3.2, etc.).  I think that this would be
useful for a
>>> lot of the users of LLVM, especially projects that use LLVM as a
library.
>>> I am willing to help maintain bug fix releases, and I'm
wondering if
>>> this is something that the LLVM project would officially support
with
>>> a stable SVN branch and by hosting the official stable tarball
releases.
>>> 
>>> I realize that maintaining stable branches is a lot of work, so I
would
>>> like to come up with a procedure that makes maintaining these
branches
>>> as easy as possible.  Here is a rough idea of what I had in
>>> mind, but please suggest alternatives if you know of a better way:
>>> 
>>> 1. Developer fixes a bug or makes a change that he/she thinks would
make
>>> a good candidate for the stable branch.  Commits would require
approval
>>> from the Code Owner in order to be backported to stable.
>>> 
>>> 2a. When the developer commits that change, he/she adds to the end
of the
>>> commit message something like:
>>> 
>>> Note: This is a candidate for the stable branch
>>> 
>>> 2b. Alternatively, if a user discovers a bug in a stable release
that has
>>> been fixed in ToT, he/she could request to have the fix backported.
>>> 
>>> 3. The developer would be encouraged, but not required to
cherry-pick the
>>> commit to the stable branch.  The stable maintainer would
periodically
>>> search the commit logs and cherry-pick any commits that had been
missed,
>>> consulting with the author of the commit in the case of a difficult
>>> merge conflict.
>>> 
>>> 4. After some interval of time, the stable maintainer would
announce
>>> plans for a stable release and testing would begin.
>>> 
>>> What does everyone think?  Would something like this be doable?
>>> 
>> As Chris said, the only thing preventing this is manpower. But if there
are people ready and willing to do this, then I don't see it as a Bad Thing.
>> 
>> My first comment is that these should be strictly bug fix releases. So
your (1) above wouldn't include changes other than bug fixes. There would
need to be a hierarchy for bugs. The reason is that *any* change to the stable
branch inherently carries risk. So for instance, a bug that's a crasher, but
would affect only a small number of people may not be worthy of inclusion into a
dot-release. Etc.
>> 
> 
> I was thinking for shared code it would be strictly bug fix only, but
> maybe things like additional C API implementations or standalone passes
> might be OK too assuming there is interest.
> 
> However, for target specific code, I think backend code owners should
> have a little more leeway as far as what gets accepted.  For the
> R600 backend I can see a situation where I may want to backport a new
> feature or something else, so it would be nice to have a little more
> flexibility and maybe other backend owners would want to have this
> option as well.
>> My second comment is that top-of-tree moves very fast. This can make
changes hard to back-port to a stable branch that may be months old. You are
stuck having to either do it yourself, or begging the original author to
back-port the fix. :-)
>> 
> 
> I understand.  I think if there is a patch that is too difficult for the
> release maintainer to backport he/she should be able to go to the author
> and request help.  If the author or some other party doesn't care
enough
> about the fix to help backport it, then it probably isn't important
> enough to go to stable.
> 
>> How frequently do you expect to do a dot-release? Our major release
schedule is roughly every 6 months. Do you think bimonthly would be too many? or
would you do it only when enough changes require it?
>> 
> 
> This is hard to say, but probably as required.
> 
>> Finally, there is testing to consider. Obviously, full testing would be
too rough on the community; people simply don't have enough time to spend a
full month testing a dot-release. What are you thoughts on how testing would
proceed?
>> 
> 
> What is it about testing for releases that takes a month to complete?
> Is it just coordinating with all the interested parties and making sure
> they are happy with the state of the code?
> 
> -Tom

Like Bill said, would prefer if we take only bug fixes excluding the
enhancement/feature change,

Developer who fixes the bug are most likely the best person to back port it
to a release branch.

How do we decide when to release bug fix release? is it by the number of
bug fixes or user request or fixed time?



On Wed, Apr 3, 2013 at 6:21 AM, Bill Wendling <wendling at apple.com>
wrote:
> For the record, I'm not opposed to this. I'm mostly playing
devil's
> advocate. But I do feel that these are issues that need to be thought out
> before we continue.
>
> -bw
>
> On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote:
>
> > On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote:
> >> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at
stellard.net> wrote:
> >>
> >>> Hi,
> >>>
> >>> I would really like to see the LLVM project start to make
official bug
> fix
> >>> releases (e.g. 3.3.1, 3.3.2, etc.).  I think that this would
be useful
> for a
> >>> lot of the users of LLVM, especially projects that use LLVM as
a
> library.
> >>> I am willing to help maintain bug fix releases, and I'm
wondering if
> >>> this is something that the LLVM project would officially
support with
> >>> a stable SVN branch and by hosting the official stable tarball
> releases.
> >>>
> >>> I realize that maintaining stable branches is a lot of work,
so I would
> >>> like to come up with a procedure that makes maintaining these
branches
> >>> as easy as possible.  Here is a rough idea of what I had in
> >>> mind, but please suggest alternatives if you know of a better
way:
> >>>
> >>> 1. Developer fixes a bug or makes a change that he/she thinks
would
> make
> >>> a good candidate for the stable branch.  Commits would require
approval
> >>> from the Code Owner in order to be backported to stable.
> >>>
> >>> 2a. When the developer commits that change, he/she adds to the
end of
> the
> >>> commit message something like:
> >>>
> >>> Note: This is a candidate for the stable branch
> >>>
> >>> 2b. Alternatively, if a user discovers a bug in a stable
release that
> has
> >>> been fixed in ToT, he/she could request to have the fix
backported.
> >>>
> >>> 3. The developer would be encouraged, but not required to
cherry-pick
> the
> >>> commit to the stable branch.  The stable maintainer would
periodically
> >>> search the commit logs and cherry-pick any commits that had
been
> missed,
> >>> consulting with the author of the commit in the case of a
difficult
> >>> merge conflict.
> >>>
> >>> 4. After some interval of time, the stable maintainer would
announce
> >>> plans for a stable release and testing would begin.
> >>>
> >>> What does everyone think?  Would something like this be
doable?
> >>>
> >> As Chris said, the only thing preventing this is manpower. But if
there
> are people ready and willing to do this, then I don't see it as a Bad
Thing.
> >>
> >> My first comment is that these should be strictly bug fix
releases. So
> your (1) above wouldn't include changes other than bug fixes. There
would
> need to be a hierarchy for bugs. The reason is that *any* change to the
> stable branch inherently carries risk. So for instance, a bug that's a
> crasher, but would affect only a small number of people may not be worthy
> of inclusion into a dot-release. Etc.
> >>
> >
> > I was thinking for shared code it would be strictly bug fix only, but
> > maybe things like additional C API implementations or standalone
passes
> > might be OK too assuming there is interest.
> >
> > However, for target specific code, I think backend code owners should
> > have a little more leeway as far as what gets accepted.  For the
> > R600 backend I can see a situation where I may want to backport a new
> > feature or something else, so it would be nice to have a little more
> > flexibility and maybe other backend owners would want to have this
> > option as well.
> >> My second comment is that top-of-tree moves very fast. This can
make
> changes hard to back-port to a stable branch that may be months old. You
> are stuck having to either do it yourself, or begging the original author
> to back-port the fix. :-)
> >>
> >
> > I understand.  I think if there is a patch that is too difficult for
the
> > release maintainer to backport he/she should be able to go to the
author
> > and request help.  If the author or some other party doesn't care
enough
> > about the fix to help backport it, then it probably isn't
important
> > enough to go to stable.
> >
> >> How frequently do you expect to do a dot-release? Our major
release
> schedule is roughly every 6 months. Do you think bimonthly would be too
> many? or would you do it only when enough changes require it?
> >>
> >
> > This is hard to say, but probably as required.
> >
> >> Finally, there is testing to consider. Obviously, full testing
would be
> too rough on the community; people simply don't have enough time to
spend a
> full month testing a dot-release. What are you thoughts on how testing
> would proceed?
> >>
> >
> > What is it about testing for releases that takes a month to complete?
> > Is it just coordinating with all the interested parties and making
sure
> > they are happy with the state of the code?
> >
> > -Tom
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>


-- 
Warm Regards
--Dev
OpenPegasus Developer
"It's Always better to try and fail instead of not doing/trying
anything"
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130403/bfd7f28a/attachment.html>

On Tue, Apr 02, 2013 at 05:47:33PM -0700, Bill Wendling
wrote:
> On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote:
> 
> > On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote:
> >> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at
stellard.net> wrote:
> >> 
> >> As Chris said, the only thing preventing this is manpower. But if
there are people ready and willing to do this, then I don't see it as a Bad
Thing.
> >> 
> >> My first comment is that these should be strictly bug fix
releases. So your (1) above wouldn't include changes other than bug fixes.
There would need to be a hierarchy for bugs. The reason is that *any* change to
the stable branch inherently carries risk. So for instance, a bug that's a
crasher, but would affect only a small number of people may not be worthy of
inclusion into a dot-release. Etc.
> >> 
> > 
> > I was thinking for shared code it would be strictly bug fix only, but
> > maybe things like additional C API implementations or standalone
passes
> > might be OK too assuming there is interest.
> > 
> > However, for target specific code, I think backend code owners should
> > have a little more leeway as far as what gets accepted.  For the
> > R600 backend I can see a situation where I may want to backport a new
> > feature or something else, so it would be nice to have a little more
> > flexibility and maybe other backend owners would want to have this
> > option as well.
> 
> If we are going to go this route, I would be *very* selective about taking
non-bug fixes. Target specific enhancements to our non-"tier 1"
platforms ("tier 1" being x86 and ARM) would depend upon that back-end
owner's judgement. However for the "tier 1" platforms, I will
continue to be very skeptical. Adding new C APIs is challenging, because once we
release with a C API it's there forever. Of course, there will be a suitable
amount of bake time, etc.
> 
> Basically, I will be very skeptical about any changes that will go into a
dot-release. The two questions, which I would want answered, are "why is
this change important enough to warrant adding it to a new release?" and
"how safe is this change?".
> 
OK, I think having a more strict policy for "tier 1" platforms makes
sense.  For the C API, I was assuming that functions would only be added
to dot-releases that were also added to ToT, however it's fine with
me if anyone wants to exclude these changes from dot-releases, I was just
throwing that out there as an example of a non-bug fix change that might
be wanted in a dot-release.
> >> My second comment is that top-of-tree moves very fast. This can
make changes hard to back-port to a stable branch that may be months old. You
are stuck having to either do it yourself, or begging the original author to
back-port the fix. :-)
> > 
> > I understand.  I think if there is a patch that is too difficult for
the
> > release maintainer to backport he/she should be able to go to the
author
> > and request help.  If the author or some other party doesn't care
enough
> > about the fix to help backport it, then it probably isn't
important
> > enough to go to stable.
> > 
> Fair enough. (And that's exactly what should happen. :-) )
> 
> >> How frequently do you expect to do a dot-release? Our major
release schedule is roughly every 6 months. Do you think bimonthly would be too
many? or would you do it only when enough changes require it?
> > 
> > This is hard to say, but probably as required.
> > 
> >> Finally, there is testing to consider. Obviously, full testing
would be too rough on the community; people simply don't have enough time to
spend a full month testing a dot-release. What are you thoughts on how testing
would proceed?
> > 
> > What is it about testing for releases that takes a month to complete?
> > Is it just coordinating with all the interested parties and making
sure
> > they are happy with the state of the code?
> > 
> Pretty much. Basically, we allow for a two week test and fix sequence. The
testers are given one week to test the newest release candidate. We could give
them a shorter amount of time, but that may put too much pressure on our
volunteers. (In actuality, most testers are finished quickly.) We post the
binaries they build so that the community can test on their software. We then
give developers a week to address any issues that came up during the testing
phase. This cycle is repeated once (twice if something disastrous comes up in
the second cycle).
> 
> Two weeks is more or less an arbitrary choice, but one which allows us
ample time to make sure that the release is as good as can be.
>
> That said, there are a couple of options that we could consider:
> 
> 1) Shorten the release schedule for a dot-release.
> 
>   - Considering that we are working with a known good base, we could
reasonably expect fewer, if any, regressions. So we wouldn't have to worry
as much about allocating so much time for fixing bugs. Plus, those who created
the bad patch should be able to fix it fairly quickly.
>
I think allowing one week for testing for dot-release would be fine,
though we might be able to shorten it if we could keep track of which
groups wanted to test the dot-release and have them report back when
they're done.

The work required to fix regressions should be much less for a
dot-release than a standard release, and I think it might be better
to shorten then time developers have to fix bugs rather than the time
for testing.  This could be done by overlapping the testing and bug fix
periods and also by having a policy of reverting commits that regress a
dot-release if they aren't fixed by the end of the bug fix period.

-Tom
> 2) Don't test quite as much, and tag the dot-release as
"beta" or some other weasel wordy thing.
> 
>   - I'm not a big fan of this (even though I proposed it). I want
external developers to be confident with the releases we send out.
> 
> I'm sure there are other choices here. But I think that the greatest
hurdle we will come across is finding the resources to do the testing.
> 
> -bw
>

On 4/2/2013 4:10 PM, Tom Stellard wrote:
>
> This is hard to say, but probably as required.
As required by what/whom?

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

On Wed, Apr 03, 2013 at 01:58:41PM -0500, Krzysztof Parzyszek
wrote:
> On 4/2/2013 4:10 PM, Tom Stellard wrote:
> >
> >This is hard to say, but probably as required.
> 
> As required by what/whom?
> 
I think the release manager would make this call based on feedback from
users and/or developers.

-Tom
> -Krzysztof
> 
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by The Linux Foundation
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev