Hi, I would like to try again to do stable releases for LLVM 3.4. Even though we were unsuccessful with stable releases for LLVM 3.3, I learned some things going through the process, which I think will increase the chance for success with LLVM 3.4. So, here is my TODO list for a successful 3.4.1 release: 1. Get volunteers to help This is probably the most important thing on this list. Stable releases need to be a community effort, there is too much work for just a few people to do. 2. Choose a release manger. I will volunteer for this, but if someone else wants to do it, I have no problem relinquishing the role. 3. Make sure this page: http://llvm.org/docs/ReleaseProcess.html is updated and correct: It would be great if someone who just participated in the release process could look over this page and make sure it is correct and easy to follow. For example, to me it's not clear how to compare the results of the test-release.sh script from different release candidates. 4. Come up with a list of all the platforms which must be tested in order to make a release. 5. Determine whether or not stable releases should maintain a stable ABI. It seems like the preference in the community is to keep a stable ABI with stable releases. I'm curious what uses cases require a stable ABI? 6. Add patch level to LLVM version (e.g. 3.4.0) 7. Figure out the .so name to use for stable releases (i.e. Should we use libLLVM-3.4.so or libLLVM-3.4.1.so) 8. Add support for stable releases to the test-release.sh script. This may go along with adding the patch level to LLVM version. 9. Choose a release date for LLVM 3.4.1 My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it be the only stable release. Once we get better as a community at doing stable releases, then maybe we can consider doing more. I think this list is a good starting point. Anyone else have any ideas or other feedback? Thanks, Tom
On 13 January 2014 23:32, Tom Stellard <tom at stellard.net> wrote:> 1. Get volunteers to help >I'm available to do point release builds and tests, as I do for releases. 3. Make sure this page: http://llvm.org/docs/ReleaseProcess.html is updated> and >correct:>I may have to add a few things... but nothing serious. 4. Come up with a list of all the platforms which must be tested in> order to make a release. >So, this depends on who is available at the time the point release comes out. The work Bill did to organize all this was not small, and it takes time to do it all, and people take holidays, move jobs, etc. Until we have an official commitment from companies that care about LLVM, in which someone should always be available to do the release from that company, we can't commit ourselves to anything. If you remember the official release process and the number of binaries released, there was nothing "official" about that process either. 5. Determine whether or not stable releases should maintain a stable ABI.>As I said earlier, I think this is paramount. What this means is that building blocks will remain compatible with the release, as to minimise re-compilation of out-of-tree projects. I can think of things like IRBuilder, DIBuilder, APInt, StringRef, but also the Debug Information metadata format, IR semantics, pass order, command-line options, etc. In essence, release 3.4.1 should have nothing more than bug fixes to existing infrastructure, either crashes, conformance or performance, but never change the semantics. 7. Figure out the .so name to use for stable releases (i.e. Should we> use libLLVM-3.4.so or libLLVM-3.4.1.so) >AFAICR, the LSB uses symlinks like: libLLVM-3.4.so -> libLLVM-3.4.so.3 libLLVM-3.4.so.3 -> libLLVM-3.4.so.3.4.1 libLLVM-3.4.so.3.4 (deprecated) libLLVM-3.4.so.3.4.1 I'm not sure LLVM does any of that, but we'd have to do something similar if we are to add point releases. 9. Choose a release date for LLVM 3.4.1>> My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it > be the only stable release. Once we get better as a community at doing > stable releases, then maybe we can consider doing more. >I'd say we only do a release if a serious enough bug shows up after we release LLVM. I wouldn't go for performance reviews right now because that's impossible to manage at our level of commitment. We cannot test all programs in all platforms. We cannot benchmark all benchmarks on all platforms. We may not even be able to compile it for all platforms... For those reasons, it's impossible to ascertain the quality of any revision after the official release unless you do the release process every time, which won't happen because it takes three months to do so. * My Impressions If I got it right, you're referring to a different kind of point release, just checking out a release and testing it, as if it was just another release in between. I personally see no value in doing so. The only value is to backport bug-fixes into the original branch-3.4 and make a release for production systems that have tested 3.4 and need a bug fixed. My reasoning is this: 1. The LLVM community follows trunk. All developers, all out-of-tree developers and all libraries and sub-projects do so. There is no point in marking release A or B when no one even cares about releases. 2. Users that need stability (Google for Renderscript, Apple of iOS, etc) get a random revision, test it thoroughly, fix a few bugs directly, and ship. No one, AFAIK, uses or depend on any specific release of LLVM. We do releases more to track conformance and performance than anything else. 3. Users that do need a specific release (research projects, proprietary R&D) will stick to a release and not change for years. Three months won't make any difference for them, and they will only move to a newer version if there is a significant feature added or bug fixed. For those reasons, releasing anything bug bug-fixes in between releases is not just unnecessary, but a waste of time. Besides, buildbots already do most of this job (compile checks, test-suites, etc), so if you do want to release a revision, I'd advise you to do so unofficially and based on buildbot status. If you're instead, referring to bug-fix releases, than backporting specific commits to the old branch won't generally break any API or semantics, and it should be safe for us to test and ship. Also, it should only take one cycle of tests, since there won't be many changes and most of it was already tested on release anyway. Makes sense? cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140114/97ac72e1/attachment.html>
On Tue, Jan 14, 2014 at 03:57:25PM +0000, Renato Golin wrote:> On 13 January 2014 23:32, Tom Stellard <tom at stellard.net> wrote: > > > 1. Get volunteers to help > > > > I'm available to do point release builds and tests, as I do for releases. > > > 3. Make sure this page: http://llvm.org/docs/ReleaseProcess.html is updated > > and > > > correct: > > > > I may have to add a few things... but nothing serious. > > > 4. Come up with a list of all the platforms which must be tested in > > order to make a release. > > > > So, this depends on who is available at the time the point release comes > out. The work Bill did to organize all this was not small, and it takes > time to do it all, and people take holidays, move jobs, etc. > > Until we have an official commitment from companies that care about LLVM, > in which someone should always be available to do the release from that > company, we can't commit ourselves to anything. If you remember the > official release process and the number of binaries released, there was > nothing "official" about that process either. > > > 5. Determine whether or not stable releases should maintain a stable ABI. > > > > As I said earlier, I think this is paramount. > > What this means is that building blocks will remain compatible with the > release, as to minimise re-compilation of out-of-tree projects. > > I can think of things like IRBuilder, DIBuilder, APInt, StringRef, but also > the Debug Information metadata format, IR semantics, pass order, > command-line options, etc. > > In essence, release 3.4.1 should have nothing more than bug fixes to > existing infrastructure, either crashes, conformance or performance, but > never change the semantics. > > > 7. Figure out the .so name to use for stable releases (i.e. Should we > > use libLLVM-3.4.so or libLLVM-3.4.1.so) > > > > AFAICR, the LSB uses symlinks like: > > libLLVM-3.4.so -> libLLVM-3.4.so.3 > libLLVM-3.4.so.3 -> libLLVM-3.4.so.3.4.1 > libLLVM-3.4.so.3.4 (deprecated) > libLLVM-3.4.so.3.4.1 > > I'm not sure LLVM does any of that, but we'd have to do something similar > if we are to add point releases. > > > 9. Choose a release date for LLVM 3.4.1 > > > > > My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it > > be the only stable release. Once we get better as a community at doing > > stable releases, then maybe we can consider doing more. > > > > I'd say we only do a release if a serious enough bug shows up after we > release LLVM. I wouldn't go for performance reviews right now because > that's impossible to manage at our level of commitment. > > We cannot test all programs in all platforms. We cannot benchmark all > benchmarks on all platforms. We may not even be able to compile it for all > platforms... > > For those reasons, it's impossible to ascertain the quality of any revision > after the official release unless you do the release process every time, > which won't happen because it takes three months to do so. > > > * My Impressions > > If I got it right, you're referring to a different kind of point release, > just checking out a release and testing it, as if it was just another > release in between. I personally see no value in doing so. The only value > is to backport bug-fixes into the original branch-3.4 and make a release > for production systems that have tested 3.4 and need a bug fixed. >Sorry this wasn't clear. My assumption was that we would be backing-porting fixes from master to 3.4 and making a release from the 3.4 branch.> My reasoning is this: > > 1. The LLVM community follows trunk. All developers, all out-of-tree > developers and all libraries and sub-projects do so. There is no point in > marking release A or B when no one even cares about releases. > > 2. Users that need stability (Google for Renderscript, Apple of iOS, etc) > get a random revision, test it thoroughly, fix a few bugs directly, and > ship. No one, AFAIK, uses or depend on any specific release of LLVM. We do > releases more to track conformance and performance than anything else. > > 3. Users that do need a specific release (research projects, proprietary > R&D) will stick to a release and not change for years. Three months won't > make any difference for them, and they will only move to a newer version if > there is a significant feature added or bug fixed. > > For those reasons, releasing anything bug bug-fixes in between releases is > not just unnecessary, but a waste of time. Besides, buildbots already do > most of this job (compile checks, test-suites, etc), so if you do want to > release a revision, I'd advise you to do so unofficially and based on > buildbot status. > > If you're instead, referring to bug-fix releases, than backporting specific > commits to the old branch won't generally break any API or semantics, and > it should be safe for us to test and ship. Also, it should only take one > cycle of tests, since there won't be many changes and most of it was > already tested on release anyway. > > Makes sense? > > cheers, > --renato
Hi,> Am 14.01.2014 um 00:32 schrieb Tom Stellard <tom at stellard.net>: > [...] > So, here is my TODO list for a successful 3.4.1 release: > > 1. Get volunteers to help > > This is probably the most important thing on this list. Stable releases > need to be a community effort, there is too much work for just a few people to > do.I'd be in as well. Cheers, Sebastian
On 14/01/14 07:32, Tom Stellard wrote:> Hi, > > I would like to try again to do stable releases for LLVM 3.4. > Even though we were unsuccessful with stable releases for LLVM > 3.3, I learned some things going through the process, which I think > will increase the chance for success with LLVM 3.4. > > So, here is my TODO list for a successful 3.4.1 release: > > 1. Get volunteers to help > > This is probably the most important thing on this list. Stable releases > need to be a community effort, there is too much work for just a few people to > do.I'm happy to run tests. Ben
I would also be in for building & testing. Cheers, -- Arnaud On Wed, Jan 15, 2014 at 3:44 AM, Ben Pope <benpope81 at gmail.com> wrote:> On 14/01/14 07:32, Tom Stellard wrote: > >> Hi, >> >> I would like to try again to do stable releases for LLVM 3.4. >> Even though we were unsuccessful with stable releases for LLVM >> 3.3, I learned some things going through the process, which I think >> will increase the chance for success with LLVM 3.4. >> >> So, here is my TODO list for a successful 3.4.1 release: >> >> 1. Get volunteers to help >> >> This is probably the most important thing on this list. Stable releases >> need to be a community effort, there is too much work for just a few >> people to >> do. >> > > I'm happy to run tests. > > Ben > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140115/6faccc66/attachment.html>
On 14/01/2014 00:32, Tom Stellard wrote:> Hi,> 1. Get volunteers to help > > This is probably the most important thing on this list. Stable releases > need to be a community effort, there is too much work for just a few people to > do.Count me in.> > 2. Choose a release manger. > > I will volunteer for this, but if someone else wants to do it, I have no > problem relinquishing the role.I would have been interested a few months ago but since I am going to start a new job next Tuesday. I don't think that would be reasonable.> 4. Come up with a list of all the platforms which must be tested in > order to make a release. > > 5. Determine whether or not stable releases should maintain a stable ABI. > > It seems like the preference in the community is to keep a stable ABI > with stable releases. I'm curious what uses cases require a stable ABI? >From the distro pov, all libraries depending on LLVM or clang libraries.That is starting to be important. If you change the ABI, we, distribution, would have to rebuild all packages depending on LLVM & Clang. If we keep the same ABI, this work won't be necessary. Changing an ABI is really a nightmare from the distribution point of view...> 6. Add patch level to LLVM version (e.g. 3.4.0) > > 7. Figure out the .so name to use for stable releases (i.e. Should we > use libLLVM-3.4.so or libLLVM-3.4.1.so)libLLVM-3.4.so should be OK if the ABI does not change.> > 9. Choose a release date for LLVM 3.4.1 > > My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it > be the only stable release. Once we get better as a community at doing > stable releases, then maybe we can consider doing more.I am not sure that it is about date but more about the number of critical bugs in the current stable release... I would add a 10) point. Clearly document what kind of patches are acceptable from a stable release. For me, it should be only important bug fixes, no new feature. Sylvestre
----- Original Message -----> From: "Sylvestre Ledru" <sylvestre at debian.org> > To: "Tom Stellard" <tom at stellard.net>, llvmdev at cs.uiuc.edu > Cc: "erik verbruggen" <erik.verbruggen at me.com> > Sent: Thursday, January 16, 2014 12:20:38 PM > Subject: Re: [LLVMdev] LLVM 3.4 stable releases > > On 14/01/2014 00:32, Tom Stellard wrote: > > Hi, > > > 1. Get volunteers to help > > > > This is probably the most important thing on this list. Stable > > releases > > need to be a community effort, there is too much work for just a > > few people to > > do. > Count me in.Me too! Tom, thanks for doing this!> > > > 2. Choose a release manger. > > > > I will volunteer for this, but if someone else wants to do it, I > > have no > > problem relinquishing the role.I'm certainly willing to help with this (I'm a bit hesitant to say that I'll do it, because I'm not sure what's involved with actually creating the binaries). I'll certainly help with culling the bug fixes to be back ported into the stable branch.> I would have been interested a few months ago but since I am going to > start a new job next Tuesday. > I don't think that would be reasonable. > > 4. Come up with a list of all the platforms which must be tested in > > order to make a release. > > > > 5. Determine whether or not stable releases should maintain a > > stable ABI. > > > > It seems like the preference in the community is to keep a stable > > ABI > > with stable releases. I'm curious what uses cases require a stable > > ABI? > From the distro pov, all libraries depending on LLVM or clang > libraries. > That is starting to be important. > If you change the ABI, we, distribution, would have to rebuild all > packages depending on > LLVM & Clang. If we keep the same ABI, this work won't be necessary. > Changing an ABI is really a nightmare from the distribution point of > view... > > 6. Add patch level to LLVM version (e.g. 3.4.0) > > > > 7. Figure out the .so name to use for stable releases (i.e. Should > > we > > use libLLVM-3.4.so or libLLVM-3.4.1.so) > > libLLVM-3.4.so should be OK if the ABI does not change.+1 But, we might also want to symlink a libLLVM-3.4.1.so to it; that way if you ship a binary that depends on some bug fix in 3.4.1 (and explicitly link against 3.4.1), it can't accidentally be run in an environment with only 3.4 (obviously, there are other ways to do this, but that seems the simplest).> > > > > 9. Choose a release date for LLVM 3.4.1 > > > > My recommendation is to release LLVM 3.4.1 3 months after 3.4 and > > have it > > be the only stable release. Once we get better as a community at > > doing > > stable releases, then maybe we can consider doing more. > I am not sure that it is about date but more about the number of > critical bugs in the current stable release... > > > I would add a 10) point. > Clearly document what kind of patches are acceptable from a stable > release. > > For me, it should be only important bug fixes, no new feature.I think we should take bug fixes, generally, (crashes and especially miscompiles), but only those that are did not involve any significant amount of refactoring (or are tied to new features). -Hal> > Sylvestre > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory
Hi, Thanks for the feedback. Here is a summary of the responses. These items are still up for discussion, but if there are no objections in the next few days, I will add these to the release documentation: + Stable release must be ABI compatible with the current major release. + Only bug fixes will be accepted in stable releases. All changes to the stable branch must first be committed to trunk (when applicable) and approved by the code appropriate code owner. + Shorter release cycle with 1 release candidate. + Shared library name will remain libLLVM-Major.Minor.so, but a libLLVM-Major.Minor.Patch.so symlink will be added. + Supported platforms will be determined by test coverage. + The release manager will determine when to make a stable release based on input from the community. This will be a judgement call, but the basic guidelines are to do a release if there are a large number of bug-fixes sitting in the stable branch or some critical bugs are found after a release that affect a large number of users. TODO list: + Add patch level to LLVM version. + Add support for stable releases to test-release.sh script. + Update testing and release documentations. + Identify candidate patches for the stable branch. -Tom On Mon, Jan 13, 2014 at 03:32:50PM -0800, Tom Stellard wrote:> Hi, > > I would like to try again to do stable releases for LLVM 3.4. > Even though we were unsuccessful with stable releases for LLVM > 3.3, I learned some things going through the process, which I think > will increase the chance for success with LLVM 3.4. > > So, here is my TODO list for a successful 3.4.1 release: > > 1. Get volunteers to help > > This is probably the most important thing on this list. Stable releases > need to be a community effort, there is too much work for just a few people to > do. > > 2. Choose a release manger. > > I will volunteer for this, but if someone else wants to do it, I have no > problem relinquishing the role. > > 3. Make sure this page: http://llvm.org/docs/ReleaseProcess.html is updated and > correct: > > It would be great if someone who just participated in the release process could > look over this page and make sure it is correct and easy to follow. For example, > to me it's not clear how to compare the results of the test-release.sh script from > different release candidates. > > 4. Come up with a list of all the platforms which must be tested in > order to make a release. > > 5. Determine whether or not stable releases should maintain a stable ABI. > > It seems like the preference in the community is to keep a stable ABI > with stable releases. I'm curious what uses cases require a stable ABI? > > 6. Add patch level to LLVM version (e.g. 3.4.0) > > 7. Figure out the .so name to use for stable releases (i.e. Should we > use libLLVM-3.4.so or libLLVM-3.4.1.so) > > 8. Add support for stable releases to the test-release.sh script. > This may go along with adding the patch level to LLVM version. > > 9. Choose a release date for LLVM 3.4.1 > > My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it > be the only stable release. Once we get better as a community at doing > stable releases, then maybe we can consider doing more. > > > > I think this list is a good starting point. Anyone else have any ideas > or other feedback? > > Thanks, > Tom > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On 20 January 2014 15:44, Tom Stellard <tom at stellard.net> wrote:> Thanks for the feedback. Here is a summary of the responses. > These items are still up for discussion, but if there are no > objections in the next few days, I will add these to the > release documentation: >Hi Tom, Thanks for the summary, I agree with mostly everything, but I have some specific comments inline... + Shared library name will remain libLLVM-Major.Minor.so, but>a libLLVM-Major.Minor.Patch.so symlink will be added.>You mean a library M.m.p will be created and a symlink M.m will point to it, right? + Supported platforms will be determined by test coverage.>I'm in two minds about it. While it makes no sense to re-build for all archs if the bug-fix is specific to only one, it'll confuse people if you need to use the same toolchain across different architectures, and we'll end up replying on the list: "oh, 3.5.1 is only for ARM and x86, while 3.5.2 is only for Mips...", etc. I think we should release all binaries, and maybe test less extensively on the architectures that have no bug-fixes, but still build the binaries, the libraries, and symlinks. The problem arises when no one will build and test for arch A and we simply cannot release something without even knowing it runs. We shouldn't stop the release because of that, and that would be an acceptable case where there simply is no patch-release for that arch. I would expect it not to happen for the architectures listed as "supported" in the documentation, and in time, it should be "frowned upon" for "supported" architectures NOT to do a patch release. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140120/6f912306/attachment.html>
Hi, I've updated these points based on recent feedback. I'll update the documentation to reflect these if there are no objections: On Mon, Jan 20, 2014 at 07:44:12AM -0800, Tom Stellard wrote:> Hi, > > Thanks for the feedback. Here is a summary of the responses. > These items are still up for discussion, but if there are no > objections in the next few days, I will add these to the > release documentation: > > + Stable release must be ABI compatible with the current major > release. > > + Only bug fixes will be accepted in stable releases. All changes > to the stable branch must first be committed to trunk (when applicable) > and approved by the code appropriate code owner. > > + Shorter release cycle with 1 release candidate. >+ Shared library name will be libLLVM-Major.Minor.Patch.so and it will be pointed to by a symbolic link called libLLVM-Major.Minor.so. e.g. libLLVM-Major.Minor.so -> libLLVM-Major.Minor.Patch.so + Before a release can be made, the unit test suite must be run with no regressions compared to the previous release on all three "tier one" platforms: X86, X86_64, and ARM. + Candidate patches for the stable branch will be nominated by replying to the llvm-commits email generated when the patch was committed to ToT and cc'ing the code owner and release manager. If for some reason there is no corresponding fix committed to ToT, patches can be nominated by emailing them to the release manager, code owner, and llvm-commits mailing list. Patches must be approved by both the release manager and the code owner before they can be committed to the stable branch.> > + The release manager will determine when to make a stable release based > on input from the community. This will be a judgement call, but > the basic guidelines are to do a release if there are a large number > of bug-fixes sitting in the stable branch or some critical bugs are > found after a release that affect a large number of users. > > TODO list: > > + Add patch level to LLVM version. > + Add support for stable releases to test-release.sh script. > + Update testing and release documentations. > + Identify candidate patches for the stable branch.+ Explore a better method for keeping track of stable bug-fixes. A few ideas: - Add a tag to the subject of the email when submitting a patch: e.g. [PATCH 3.4.1] - Create a llvm-stable mailing list for stable patches - Add an annotation when committing fixes to master in order to indicate that they should be backported to the stable branch. -Tom