On 02/18/2015 03:01 PM, Renato Golin wrote:> So it seems that you're one of the very few people that doesn't use > ToT. Almost everyone else uses it and the progress of LLVM kind of > assume you do.My company also does not use ToT. Being able to associate a product with a well-known release is *very* important to us. It enables communication with our customers. It allows us to determine compatibility between our Clang derived product and other Clang derived products. It allows us to easily discuss groups of feature sets. I personally find it rather crazy that developers are expected to use ToT (and then determine stability on their own). Why have an RC series at all if that is the case? Tom.
On 18 February 2015 at 20:59, Tom Honermann <thonermann at coverity.com> wrote:> My company also does not use ToT. Being able to associate a product with a > well-known release is *very* important to us. It enables communication with > our customers. It allows us to determine compatibility between our Clang > derived product and other Clang derived products. It allows us to easily > discuss groups of feature sets.It's good to know that more and more people are actually interested in the releases. Am I right to assume that you test the release candidates as we put them forward with your own products? It'd be good to get that extra level of testing during releases. After all, a release is only as good as the people testing it. If it works for you, it's good for you, and so on. If there are enough people testing and reporting bugs, we'll end up with a very stable product, even if we don't realise at a first glance.> I personally find it rather crazy that developers are expected to use ToT > (and then determine stability on their own). Why have an RC series at all > if that is the case?Core developers tend to use ToT. External users tend to use stable commits (you can check the already existing buildbots for that) or releases + cherry-pick. No one I know uses pure releases and expect them to be perfect. The RC series is to fix the urgent bugs so we can call the release minimally stable. Right now it is, and the only serious problem I know is the multiple regressions in performance in that phoronix run. I can't vouch for those numbers and not many people cared about that to make it a blocker, so all in all, the release is "good". However, it's not perfect, and that's why we started doing dot-releases two years ago and why I still think it's worth. I can only see two solutions to external users that rely on releases being always better than the previous: 1. Test the full release, report all problems, wait for dot-release 1, re-base. 2. Be an active part in the release process, fix bugs (not just report it), keeping in mind that we normally don't spend more than a month during a release cycle. cheers, --renato
On Wed, Feb 18, 2015 at 4:12 PM, Renato Golin <renato.golin at linaro.org> wrote:> On 18 February 2015 at 20:59, Tom Honermann <thonermann at coverity.com> wrote: >> My company also does not use ToT. Being able to associate a product with a >> well-known release is *very* important to us. It enables communication with >> our customers. It allows us to determine compatibility between our Clang >> derived product and other Clang derived products. It allows us to easily >> discuss groups of feature sets. > > It's good to know that more and more people are actually interested in > the releases. Am I right to assume that you test the release > candidates as we put them forward with your own products? It'd be good > to get that extra level of testing during releases. > > After all, a release is only as good as the people testing it. If it > works for you, it's good for you, and so on. If there are enough > people testing and reporting bugs, we'll end up with a very stable > product, even if we don't realise at a first glance. > > >> I personally find it rather crazy that developers are expected to use ToT >> (and then determine stability on their own). Why have an RC series at all >> if that is the case? > > Core developers tend to use ToT. External users tend to use stable > commits (you can check the already existing buildbots for that) or > releases + cherry-pick. No one I know uses pure releases and expect > them to be perfect. > > The RC series is to fix the urgent bugs so we can call the release > minimally stable. Right now it is, and the only serious problem I know > is the multiple regressions in performance in that phoronix run. I > can't vouch for those numbers and not many people cared about that to > make it a blocker, so all in all, the release is "good". > > However, it's not perfect, and that's why we started doing > dot-releases two years ago and why I still think it's worth. > > I can only see two solutions to external users that rely on releases > being always better than the previous: > > 1. Test the full release, report all problems, wait for dot-release 1, re-base. > > 2. Be an active part in the release process, fix bugs (not just report > it), keeping in mind that we normally don't spend more than a month > during a release cycle. > > cheers, > --renatoRenato, Do we have any buildbots that routinely run the commercial SPEC benchmarks and flag run-time regressions for trunk and the release branches? I believe FSF gcc has that in their testing scheme. Jack
On 02/18/2015 04:12 PM, Renato Golin wrote:> On 18 February 2015 at 20:59, Tom Honermann <thonermann at coverity.com> wrote: >> My company also does not use ToT. Being able to associate a product with a >> well-known release is *very* important to us. It enables communication with >> our customers. It allows us to determine compatibility between our Clang >> derived product and other Clang derived products. It allows us to easily >> discuss groups of feature sets. > > It's good to know that more and more people are actually interested in > the releases. Am I right to assume that you test the release > candidates as we put them forward with your own products? It'd be good > to get that extra level of testing during releases.Unfortunately, we don't currently have the resources to do so. When we upgrade, we upgrade to the most recent release and perform testing on top of it. We call Clang APIs directly, so upgrades are always a bit costly as we adjust to changing APIs (btw, that isn't a complaint. We appreciate the need for APIs to evolve).> After all, a release is only as good as the people testing it. If it > works for you, it's good for you, and so on. If there are enough > people testing and reporting bugs, we'll end up with a very stable > product, even if we don't realise at a first glance.That is true, of course. The Clang community clearly takes testing seriously. I appreciate that I see push back on patch submissions that do not include a test case (even if that means some of the patches I've submitted have not been accepted because I haven't (yet) found the time to code a proper test). From my perspective, passing the test suite is a good indicator of release quality.>> I personally find it rather crazy that developers are expected to use ToT >> (and then determine stability on their own). Why have an RC series at all >> if that is the case? > > Core developers tend to use ToT. External users tend to use stable > commits (you can check the already existing buildbots for that) or > releases + cherry-pick. No one I know uses pure releases and expect > them to be perfect.We use pure releases and cherry-pick later commits for problems we encounter or see discussed on the mailing list that we think might impact us. The performance issue being discussed is one we would want to cherry-pick a fix for if/when we upgrade to a release that has the regression (assuming I remember this discussion if/when we do so). Having such issues fixed prior to the release would be a benefit to us. We don't expect the releases to be perfect.> The RC series is to fix the urgent bugs so we can call the release > minimally stable. Right now it is, and the only serious problem I know > is the multiple regressions in performance in that phoronix run. I > can't vouch for those numbers and not many people cared about that to > make it a blocker, so all in all, the release is "good".Understood.> However, it's not perfect, and that's why we started doing > dot-releases two years ago and why I still think it's worth.I agree those are worthwhile. We do monitor those.> I can only see two solutions to external users that rely on releases > being always better than the previous: > > 1. Test the full release, report all problems, wait for dot-release 1, re-base. > > 2. Be an active part in the release process, fix bugs (not just report > it), keeping in mind that we normally don't spend more than a month > during a release cycle.Agreed. Unfortunately, we are resource constrained (as is everyone) and can only do so much. Tom.
Joerg Sonnenberger
2015-Feb-18 21:46 UTC
[LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged
On Wed, Feb 18, 2015 at 03:59:15PM -0500, Tom Honermann wrote:> On 02/18/2015 03:01 PM, Renato Golin wrote: > >So it seems that you're one of the very few people that doesn't use > >ToT. Almost everyone else uses it and the progress of LLVM kind of > >assume you do. > > My company also does not use ToT. Being able to associate a product > with a well-known release is *very* important to us. It enables > communication with our customers. It allows us to determine > compatibility between our Clang derived product and other Clang > derived products. It allows us to easily discuss groups of feature > sets. > > I personally find it rather crazy that developers are expected to > use ToT (and then determine stability on their own). Why have an RC > series at all if that is the case?Let's go back a step. This started because a performance regression was reported very late in the release cycle of 3.6. A performance regression by 20% or 40% is annoying for whoever is affected by it, but it is not a critical issue like miscompilation. From the current state of the analysis, it is not even clear *why* it happens. There are various patterns like very hot inner loops where minimal changes to the resulting code can have huge impact on the performance, without being able to tell from looking at it why that is the case. The combination results in a sane decision for release management to not consider this a blocker. It doesn't mean people are ignoring it (now that it has actually been reported in the proper place, not somewhere on the internete (TM)). For me, the most critical job of a compiler is creating working output. The second most critical job is to not pointlessly create horrible code when the input was "simple enough". Everything else is nice to have. Now Renato's comment about many developers focusing on ToT is a direct result of the development pace of LLVM. It is not appropiate for everyone. If your product is mostly about clang as parser for example, code changes are not nearly as important to trace in real time. The APIs tend to be quite a bit more stable and the number of fixes and new features is also lower. A release is also a good point in time to focus on stabilizing things and shacking out bugs. It doesn't mean that's ignored the rest of the time, but development focus certainly does shift for a while around releases. That's something the github model of "just pull whatever is in trunk" ignores. Finally, the LLVM community has been starting to put more effort into release maintainance. A release two years ago was essentially a hit-and-miss effort. With 3.4 and to a much greater degree 3.5, the release branch gets further bug fixes and point releases are done. Kudos to the release manager for that, you know who you are. Joerg
On Wed, Feb 18, 2015 at 4:46 PM, Joerg Sonnenberger <joerg at britannica.bec.de> wrote:> On Wed, Feb 18, 2015 at 03:59:15PM -0500, Tom Honermann wrote: >> On 02/18/2015 03:01 PM, Renato Golin wrote: >> >So it seems that you're one of the very few people that doesn't use >> >ToT. Almost everyone else uses it and the progress of LLVM kind of >> >assume you do. >> >> My company also does not use ToT. Being able to associate a product >> with a well-known release is *very* important to us. It enables >> communication with our customers. It allows us to determine >> compatibility between our Clang derived product and other Clang >> derived products. It allows us to easily discuss groups of feature >> sets. >> >> I personally find it rather crazy that developers are expected to >> use ToT (and then determine stability on their own). Why have an RC >> series at all if that is the case? > > Let's go back a step. This started because a performance regression was > reported very late in the release cycle of 3.6. A performance regression > by 20% or 40% is annoying for whoever is affected by it, but it is not a > critical issue like miscompilation. From the current state of the > analysis, it is not even clear *why* it happens. There are various > patterns like very hot inner loops where minimal changes to the > resulting code can have huge impact on the performance, without being > able to tell from looking at it why that is the case. The combination > results in a sane decision for release management to not consider this a > blocker. It doesn't mean people are ignoring it (now that it has > actually been reported in the proper place, not somewhere on the > internete (TM)). For me, the most critical job of a compiler is creating > working output. The second most critical job is to not pointlessly > create horrible code when the input was "simple enough". Everything else > is nice to have.Are you sure that is true? My understanding is that the regression caused by r222453 and the necessary fix is in fact well understood. http://llvm.org/bugs/show_bug.cgi?id=22589#c26 http://llvm.org/bugs/show_bug.cgi?id=22589#c33 http://llvm.org/bugs/show_bug.cgi?id=22589#c36 http://llvm.org/bugs/show_bug.cgi?id=22589#c43 http://llvm.org/bugs/show_bug.cgi?id=22589#c54 http://reviews.llvm.org/D7715 Now I agree that the other 50% of the performance regression due to r217953 is not understood yet... http://llvm.org/bugs/show_bug.cgi?id=22589#c27 but that is an entirely orthogonal issue that just happens to hit the same benchmark. Jack> > Now Renato's comment about many developers focusing on ToT is a direct > result of the development pace of LLVM. It is not appropiate for > everyone. If your product is mostly about clang as parser for example, > code changes are not nearly as important to trace in real time. The APIs > tend to be quite a bit more stable and the number of fixes and new > features is also lower. A release is also a good point in time to focus > on stabilizing things and shacking out bugs. It doesn't mean that's > ignored the rest of the time, but development focus certainly does > shift for a while around releases. That's something the github model of > "just pull whatever is in trunk" ignores. > > Finally, the LLVM community has been starting to put more effort into > release maintainance. A release two years ago was essentially a > hit-and-miss effort. With 3.4 and to a much greater degree 3.5, the > release branch gets further bug fixes and point releases are done. > Kudos to the release manager for that, you know who you are. > > Joerg > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev