On Tue, Apr 02, 2013 at 05:47:33PM -0700, Bill Wendling wrote:> On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote: > > > On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote: > >> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at stellard.net> wrote: > >> > >> As Chris said, the only thing preventing this is manpower. But if there are people ready and willing to do this, then I don't see it as a Bad Thing. > >> > >> My first comment is that these should be strictly bug fix releases. So your (1) above wouldn't include changes other than bug fixes. There would need to be a hierarchy for bugs. The reason is that *any* change to the stable branch inherently carries risk. So for instance, a bug that's a crasher, but would affect only a small number of people may not be worthy of inclusion into a dot-release. Etc. > >> > > > > I was thinking for shared code it would be strictly bug fix only, but > > maybe things like additional C API implementations or standalone passes > > might be OK too assuming there is interest. > > > > However, for target specific code, I think backend code owners should > > have a little more leeway as far as what gets accepted. For the > > R600 backend I can see a situation where I may want to backport a new > > feature or something else, so it would be nice to have a little more > > flexibility and maybe other backend owners would want to have this > > option as well. > > If we are going to go this route, I would be *very* selective about taking non-bug fixes. Target specific enhancements to our non-"tier 1" platforms ("tier 1" being x86 and ARM) would depend upon that back-end owner's judgement. However for the "tier 1" platforms, I will continue to be very skeptical. Adding new C APIs is challenging, because once we release with a C API it's there forever. Of course, there will be a suitable amount of bake time, etc. > > Basically, I will be very skeptical about any changes that will go into a dot-release. The two questions, which I would want answered, are "why is this change important enough to warrant adding it to a new release?" and "how safe is this change?". >OK, I think having a more strict policy for "tier 1" platforms makes sense. For the C API, I was assuming that functions would only be added to dot-releases that were also added to ToT, however it's fine with me if anyone wants to exclude these changes from dot-releases, I was just throwing that out there as an example of a non-bug fix change that might be wanted in a dot-release.> >> My second comment is that top-of-tree moves very fast. This can make changes hard to back-port to a stable branch that may be months old. You are stuck having to either do it yourself, or begging the original author to back-port the fix. :-) > > > > I understand. I think if there is a patch that is too difficult for the > > release maintainer to backport he/she should be able to go to the author > > and request help. If the author or some other party doesn't care enough > > about the fix to help backport it, then it probably isn't important > > enough to go to stable. > > > Fair enough. (And that's exactly what should happen. :-) ) > > >> How frequently do you expect to do a dot-release? Our major release schedule is roughly every 6 months. Do you think bimonthly would be too many? or would you do it only when enough changes require it? > > > > This is hard to say, but probably as required. > > > >> Finally, there is testing to consider. Obviously, full testing would be too rough on the community; people simply don't have enough time to spend a full month testing a dot-release. What are you thoughts on how testing would proceed? > > > > What is it about testing for releases that takes a month to complete? > > Is it just coordinating with all the interested parties and making sure > > they are happy with the state of the code? > > > Pretty much. Basically, we allow for a two week test and fix sequence. The testers are given one week to test the newest release candidate. We could give them a shorter amount of time, but that may put too much pressure on our volunteers. (In actuality, most testers are finished quickly.) We post the binaries they build so that the community can test on their software. We then give developers a week to address any issues that came up during the testing phase. This cycle is repeated once (twice if something disastrous comes up in the second cycle). > > Two weeks is more or less an arbitrary choice, but one which allows us ample time to make sure that the release is as good as can be. > > That said, there are a couple of options that we could consider: > > 1) Shorten the release schedule for a dot-release. > > - Considering that we are working with a known good base, we could reasonably expect fewer, if any, regressions. So we wouldn't have to worry as much about allocating so much time for fixing bugs. Plus, those who created the bad patch should be able to fix it fairly quickly. >I think allowing one week for testing for dot-release would be fine, though we might be able to shorten it if we could keep track of which groups wanted to test the dot-release and have them report back when they're done. The work required to fix regressions should be much less for a dot-release than a standard release, and I think it might be better to shorten then time developers have to fix bugs rather than the time for testing. This could be done by overlapping the testing and bug fix periods and also by having a policy of reverting commits that regress a dot-release if they aren't fixed by the end of the bug fix period. -Tom> 2) Don't test quite as much, and tag the dot-release as "beta" or some other weasel wordy thing. > > - I'm not a big fan of this (even though I proposed it). I want external developers to be confident with the releases we send out. > > I'm sure there are other choices here. But I think that the greatest hurdle we will come across is finding the resources to do the testing. > > -bw >
On Apr 3, 2013, at 11:30 AM, Tom Stellard <tom at stellard.net> wrote:> On Tue, Apr 02, 2013 at 05:47:33PM -0700, Bill Wendling wrote: >> On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote: >> >>> On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote: >>>> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at stellard.net> wrote: >>>> >>>> As Chris said, the only thing preventing this is manpower. But if there are people ready and willing to do this, then I don't see it as a Bad Thing. >>>> >>>> My first comment is that these should be strictly bug fix releases. So your (1) above wouldn't include changes other than bug fixes. There would need to be a hierarchy for bugs. The reason is that *any* change to the stable branch inherently carries risk. So for instance, a bug that's a crasher, but would affect only a small number of people may not be worthy of inclusion into a dot-release. Etc. >>>> >>> >>> I was thinking for shared code it would be strictly bug fix only, but >>> maybe things like additional C API implementations or standalone passes >>> might be OK too assuming there is interest. >>> >>> However, for target specific code, I think backend code owners should >>> have a little more leeway as far as what gets accepted. For the >>> R600 backend I can see a situation where I may want to backport a new >>> feature or something else, so it would be nice to have a little more >>> flexibility and maybe other backend owners would want to have this >>> option as well. >> >> If we are going to go this route, I would be *very* selective about taking non-bug fixes. Target specific enhancements to our non-"tier 1" platforms ("tier 1" being x86 and ARM) would depend upon that back-end owner's judgement. However for the "tier 1" platforms, I will continue to be very skeptical. Adding new C APIs is challenging, because once we release with a C API it's there forever. Of course, there will be a suitable amount of bake time, etc. >> >> Basically, I will be very skeptical about any changes that will go into a dot-release. The two questions, which I would want answered, are "why is this change important enough to warrant adding it to a new release?" and "how safe is this change?". >> > > OK, I think having a more strict policy for "tier 1" platforms makes > sense. For the C API, I was assuming that functions would only be added > to dot-releases that were also added to ToT, however it's fine with > me if anyone wants to exclude these changes from dot-releases, I was just > throwing that out there as an example of a non-bug fix change that might > be wanted in a dot-release. > >>>> My second comment is that top-of-tree moves very fast. This can make changes hard to back-port to a stable branch that may be months old. You are stuck having to either do it yourself, or begging the original author to back-port the fix. :-) >>> >>> I understand. I think if there is a patch that is too difficult for the >>> release maintainer to backport he/she should be able to go to the author >>> and request help. If the author or some other party doesn't care enough >>> about the fix to help backport it, then it probably isn't important >>> enough to go to stable. >>> >> Fair enough. (And that's exactly what should happen. :-) ) >> >>>> How frequently do you expect to do a dot-release? Our major release schedule is roughly every 6 months. Do you think bimonthly would be too many? or would you do it only when enough changes require it? >>> >>> This is hard to say, but probably as required. >>> >>>> Finally, there is testing to consider. Obviously, full testing would be too rough on the community; people simply don't have enough time to spend a full month testing a dot-release. What are you thoughts on how testing would proceed? >>> >>> What is it about testing for releases that takes a month to complete? >>> Is it just coordinating with all the interested parties and making sure >>> they are happy with the state of the code? >>> >> Pretty much. Basically, we allow for a two week test and fix sequence. The testers are given one week to test the newest release candidate. We could give them a shorter amount of time, but that may put too much pressure on our volunteers. (In actuality, most testers are finished quickly.) We post the binaries they build so that the community can test on their software. We then give developers a week to address any issues that came up during the testing phase. This cycle is repeated once (twice if something disastrous comes up in the second cycle). >> >> Two weeks is more or less an arbitrary choice, but one which allows us ample time to make sure that the release is as good as can be. >> >> That said, there are a couple of options that we could consider: >> >> 1) Shorten the release schedule for a dot-release. >> >> - Considering that we are working with a known good base, we could reasonably expect fewer, if any, regressions. So we wouldn't have to worry as much about allocating so much time for fixing bugs. Plus, those who created the bad patch should be able to fix it fairly quickly. >> > > I think allowing one week for testing for dot-release would be fine, > though we might be able to shorten it if we could keep track of which > groups wanted to test the dot-release and have them report back when > they're done. > > The work required to fix regressions should be much less for a > dot-release than a standard release, and I think it might be better > to shorten then time developers have to fix bugs rather than the time > for testing. This could be done by overlapping the testing and bug fix > periods and also by having a policy of reverting commits that regress a > dot-release if they aren't fixed by the end of the bug fix period. >As a previous LLVM release manager, the hardest part about releases is getting people to fix release blocker regressions. So, I would suggest that if a patch introduces new regressions, that it gets reverted if no one is actually fixing it. Otherwise, you get stuck waiting forever for a fix and then dot releases become a lot less useful. My advice is to start small with very concrete goals. Start with only severe and critical bug fixes and do not add features. Keep the number of changes small and figure out a system to nominate and approve bug fixes for inclusion in the dot release. Also there should be a timeframe where dot releases for a specific version are dropped. Since we release every 6 months, when do you plan to stop "maintaing" a release branch? Will there only be a 3.2.1-3.3.3 (if you release every 2 months) and then you move onto the new release? The one thing I want to make sure happens, is that people have the same confidence in dot release as the a new major release. So, dot releases would need to have the same qualification criteria. Thanks for volunteering Tom. I think this will be great. -Tanya> -Tom > >> 2) Don't test quite as much, and tag the dot-release as "beta" or some other weasel wordy thing. >> >> - I'm not a big fan of this (even though I proposed it). I want external developers to be confident with the releases we send out. >> >> I'm sure there are other choices here. But I think that the greatest hurdle we will come across is finding the resources to do the testing. >> >> -bw >> > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On Mon, Apr 08, 2013 at 11:32:48AM -0700, Tanya Lattner wrote:> > On Apr 3, 2013, at 11:30 AM, Tom Stellard <tom at stellard.net> wrote: > > > On Tue, Apr 02, 2013 at 05:47:33PM -0700, Bill Wendling wrote: > >> On Apr 2, 2013, at 2:10 PM, Tom Stellard <tom at stellard.net> wrote: > >> > >>> On Tue, Apr 02, 2013 at 11:52:09AM -0700, Bill Wendling wrote: > >>>> On Apr 2, 2013, at 9:51 AM, Tom Stellard <tom at stellard.net> wrote: > >>>> > >>>> As Chris said, the only thing preventing this is manpower. But if there are people ready and willing to do this, then I don't see it as a Bad Thing. > >>>> > >>>> My first comment is that these should be strictly bug fix releases. So your (1) above wouldn't include changes other than bug fixes. There would need to be a hierarchy for bugs. The reason is that *any* change to the stable branch inherently carries risk. So for instance, a bug that's a crasher, but would affect only a small number of people may not be worthy of inclusion into a dot-release. Etc. > >>>> > >>> > >>> I was thinking for shared code it would be strictly bug fix only, but > >>> maybe things like additional C API implementations or standalone passes > >>> might be OK too assuming there is interest. > >>> > >>> However, for target specific code, I think backend code owners should > >>> have a little more leeway as far as what gets accepted. For the > >>> R600 backend I can see a situation where I may want to backport a new > >>> feature or something else, so it would be nice to have a little more > >>> flexibility and maybe other backend owners would want to have this > >>> option as well. > >> > >> If we are going to go this route, I would be *very* selective about taking non-bug fixes. Target specific enhancements to our non-"tier 1" platforms ("tier 1" being x86 and ARM) would depend upon that back-end owner's judgement. However for the "tier 1" platforms, I will continue to be very skeptical. Adding new C APIs is challenging, because once we release with a C API it's there forever. Of course, there will be a suitable amount of bake time, etc. > >> > >> Basically, I will be very skeptical about any changes that will go into a dot-release. The two questions, which I would want answered, are "why is this change important enough to warrant adding it to a new release?" and "how safe is this change?". > >> > > > > OK, I think having a more strict policy for "tier 1" platforms makes > > sense. For the C API, I was assuming that functions would only be added > > to dot-releases that were also added to ToT, however it's fine with > > me if anyone wants to exclude these changes from dot-releases, I was just > > throwing that out there as an example of a non-bug fix change that might > > be wanted in a dot-release. > > > >>>> My second comment is that top-of-tree moves very fast. This can make changes hard to back-port to a stable branch that may be months old. You are stuck having to either do it yourself, or begging the original author to back-port the fix. :-) > >>> > >>> I understand. I think if there is a patch that is too difficult for the > >>> release maintainer to backport he/she should be able to go to the author > >>> and request help. If the author or some other party doesn't care enough > >>> about the fix to help backport it, then it probably isn't important > >>> enough to go to stable. > >>> > >> Fair enough. (And that's exactly what should happen. :-) ) > >> > >>>> How frequently do you expect to do a dot-release? Our major release schedule is roughly every 6 months. Do you think bimonthly would be too many? or would you do it only when enough changes require it? > >>> > >>> This is hard to say, but probably as required. > >>> > >>>> Finally, there is testing to consider. Obviously, full testing would be too rough on the community; people simply don't have enough time to spend a full month testing a dot-release. What are you thoughts on how testing would proceed? > >>> > >>> What is it about testing for releases that takes a month to complete? > >>> Is it just coordinating with all the interested parties and making sure > >>> they are happy with the state of the code? > >>> > >> Pretty much. Basically, we allow for a two week test and fix sequence. The testers are given one week to test the newest release candidate. We could give them a shorter amount of time, but that may put too much pressure on our volunteers. (In actuality, most testers are finished quickly.) We post the binaries they build so that the community can test on their software. We then give developers a week to address any issues that came up during the testing phase. This cycle is repeated once (twice if something disastrous comes up in the second cycle). > >> > >> Two weeks is more or less an arbitrary choice, but one which allows us ample time to make sure that the release is as good as can be. > >> > >> That said, there are a couple of options that we could consider: > >> > >> 1) Shorten the release schedule for a dot-release. > >> > >> - Considering that we are working with a known good base, we could reasonably expect fewer, if any, regressions. So we wouldn't have to worry as much about allocating so much time for fixing bugs. Plus, those who created the bad patch should be able to fix it fairly quickly. > >> > > > > I think allowing one week for testing for dot-release would be fine, > > though we might be able to shorten it if we could keep track of which > > groups wanted to test the dot-release and have them report back when > > they're done. > > > > The work required to fix regressions should be much less for a > > dot-release than a standard release, and I think it might be better > > to shorten then time developers have to fix bugs rather than the time > > for testing. This could be done by overlapping the testing and bug fix > > periods and also by having a policy of reverting commits that regress a > > dot-release if they aren't fixed by the end of the bug fix period. > > > > As a previous LLVM release manager, the hardest part about releases is getting people to fix release blocker regressions. So, I would suggest that if a patch introduces new regressions, that it gets reverted if no one is actually fixing it. Otherwise, you get stuck waiting forever for a fix and then dot releases become a lot less useful. >Sure, this makes sense to me.> My advice is to start small with very concrete goals. Start with only severe and critical bug fixes and do not add features. Keep the number of changes small and figure out a system to nominate and approve bug fixes for inclusion in the dot release. Also there should be a timeframe where dot releases for a specific version are dropped. Since we release every 6 months, when do you plan to stop "maintaing" a release branch? Will there only be a 3.2.1-3.3.3 (if you release every 2 months) and then you move onto the new release? >I really only have interest in maintaining a release branch until the next release, so just 6 months. However, if someone else wanted to volunteer to maintain a specific release branch longer than that, I think that would be OK. -Tom> The one thing I want to make sure happens, is that people have the same confidence in dot release as the a new major release. So, dot releases would need to have the same qualification criteria. > > Thanks for volunteering Tom. I think this will be great. > > -Tanya > > > > -Tom > > > >> 2) Don't test quite as much, and tag the dot-release as "beta" or some other weasel wordy thing. > >> > >> - I'm not a big fan of this (even though I proposed it). I want external developers to be confident with the releases we send out. > >> > >> I'm sure there are other choices here. But I think that the greatest hurdle we will come across is finding the resources to do the testing. > >> > >> -bw > >> > > _______________________________________________ > > LLVM Developers mailing list > > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >