Jim Grosbach <grosbach at apple.com> writes:>> - How difficult do you expect a backport to 3.1 to be? We have to work >> from 3.1. Trunk is too buggy.> You've stated that trunk is too buggy for you to work from on multiple > occasions. Can you elaborate? That doesn't match my experience, as I > install a new compiler on my workstation from a trunk build every > night and have only had a problem as a result once in the last year or > more. It sounds like you've had a much different experience, which is > unfortunate, and perhaps indicates a hole in our buildbot and nightly > test infrastructure that could be fixed.We do a lot of Fortran which doesn't get covered all that well by the nightly tester. We also have a completely different frontend and optimizer meaning that we present code to LLVM that it's never seen before. We've seen major scalability problems in the past. We compile some absolutely gigantic codes here. Fortunately it appears most of these have been fixed but I think LoopStrengthReduce is still a problem (we've turned it off). Scheduling, regalloc, instcombine and dagcombine have all presented problems in the past. We've sent patches for the most egregious cases. Many of those patches get dropped, leading to less confidence on our side that issues will be fixed quickly. Others in the LLVM community have fixed other such problems. Again, these issues get fixed with new releases (and with some hacking by us in the interim) but the experience makes me very hesitant to make our development depend on trunk. We cannot afford to waste even a few days of interrupted release development work tracking down a new regression introduced in trunk. We also can't afford to have compiler builds break because someone upstream decided to change an API we rely on. We make monthly releases. We have to have some control over when we introduce such changes. We run hundreds of thousands of tests each week, both simple regression tests and large applications. I don't think the nightly tester will ever be capable of covering that. I have a long-term goal of setting up a tree that uses LLVM trunk for our builds both so I can track what's happening on trunk to more easily know what it means for us and to test out some of the new features, like the MachineInstr scheduler in this case. Unfortunately, time is limited so I have to do that piecemeal 10 minutes here and there. The git mirror will make that much easier. -Dave
On May 9, 2012, at 2:00 PM, dag at cray.com wrote:> We also can't afford to have compiler builds break because someone > upstream decided to change an API we rely on. We make monthly releases. > We have to have some control over when we introduce such changes.There's no great solution. But you'll have to make a tradeoff between being engaged in the development process vs insulating your development from external changes. I've spent plenty of time dealing with a similar situation, and I found it more efficient to keep all development upstream from integration and release as in: LLVM trunk -> Cray trunk -> Cray release. This way you're only back porting critical bug fixes and you're never reverse-engineering a pile of new features to find out why something broke. -Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20120509/919d8276/attachment.html>
On May 9, 2012, at 2:00 PM, dag at cray.com wrote:> Jim Grosbach <grosbach at apple.com> writes: > >>> - How difficult do you expect a backport to 3.1 to be? We have to work >>> from 3.1. Trunk is too buggy. > >> You've stated that trunk is too buggy for you to work from on multiple >> occasions. Can you elaborate? That doesn't match my experience, as I >> install a new compiler on my workstation from a trunk build every >> night and have only had a problem as a result once in the last year or >> more. It sounds like you've had a much different experience, which is >> unfortunate, and perhaps indicates a hole in our buildbot and nightly >> test infrastructure that could be fixed. > > We do a lot of Fortran which doesn't get covered all that well by the > nightly tester. We also have a completely different frontend and > optimizer meaning that we present code to LLVM that it's never seen > before. We've seen major scalability problems in the past. We compile > some absolutely gigantic codes here. Fortunately it appears most of > these have been fixed but I think LoopStrengthReduce is still a problem > (we've turned it off). Scheduling, regalloc, instcombine and dagcombine > have all presented problems in the past. We've sent patches for the > most egregious cases. Many of those patches get dropped, leading to > less confidence on our side that issues will be fixed quickly. Others > in the LLVM community have fixed other such problems. > > Again, these issues get fixed with new releases (and with some hacking > by us in the interim) but the experience makes me very hesitant to make > our development depend on trunk. We cannot afford to waste even a few > days of interrupted release development work tracking down a new > regression introduced in trunk. > > We also can't afford to have compiler builds break because someone > upstream decided to change an API we rely on. We make monthly releases. > We have to have some control over when we introduce such changes.This has nothing to do with the scheduler roadmap. If you want to engage the community on API compatibility discussion, it should be a different thread. But given how long you have been involved in LLVM, you should know the philosophy: LLVM development is optimized for people who stay on trunk. Evan> > We run hundreds of thousands of tests each week, both simple regression > tests and large applications. I don't think the nightly tester will > ever be capable of covering that. > > I have a long-term goal of setting up a tree that uses LLVM trunk for > our builds both so I can track what's happening on trunk to more easily > know what it means for us and to test out some of the new features, like > the MachineInstr scheduler in this case. Unfortunately, time is limited > so I have to do that piecemeal 10 minutes here and there. The git > mirror will make that much easier. > > -Dave > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On Wed, 9 May 2012 16:00:34 -0500 <dag at cray.com> wrote:> Jim Grosbach <grosbach at apple.com> writes: > > >> - How difficult do you expect a backport to 3.1 to be? We have to > >> work from 3.1. Trunk is too buggy. > > > You've stated that trunk is too buggy for you to work from on > > multiple occasions. Can you elaborate? That doesn't match my > > experience, as I install a new compiler on my workstation from a > > trunk build every night and have only had a problem as a result > > once in the last year or more. It sounds like you've had a much > > different experience, which is unfortunate, and perhaps indicates a > > hole in our buildbot and nightly test infrastructure that could be > > fixed. > > We do a lot of Fortran which doesn't get covered all that well by the > nightly tester. We also have a completely different frontend and > optimizer meaning that we present code to LLVM that it's never seen > before. We've seen major scalability problems in the past. We > compile some absolutely gigantic codes here. Fortunately it appears > most of these have been fixed but I think LoopStrengthReduce is still > a problem (we've turned it off). Scheduling, regalloc, instcombine > and dagcombine have all presented problems in the past. We've sent > patches for the most egregious cases. Many of those patches get > dropped, leading to less confidence on our side that issues will be > fixed quickly. Others in the LLVM community have fixed other such > problems. > > Again, these issues get fixed with new releases (and with some hacking > by us in the interim) but the experience makes me very hesitant to > make our development depend on trunk. We cannot afford to waste even > a few days of interrupted release development work tracking down a new > regression introduced in trunk. > > We also can't afford to have compiler builds break because someone > upstream decided to change an API we rely on. We make monthly > releases. We have to have some control over when we introduce such > changes. > > We run hundreds of thousands of tests each week, both simple > regression tests and large applications. I don't think the nightly > tester will ever be capable of covering that. > > I have a long-term goal of setting up a tree that uses LLVM trunk for > our builds both so I can track what's happening on trunk to more > easily know what it means for us and to test out some of the new > features, like the MachineInstr scheduler in this case. > Unfortunately, time is limited so I have to do that piecemeal 10 > minutes here and there. The git mirror will make that much easier.Out of curiosity, can your frontend be configured to produce LLVM IR that could be fed though the trunk backend passes, or is a tighter integration required? If it is possible to produce LLVM IR, then it might be possible to test a lot, although not all, of the upstream changes using your internal test suite in combination with the generic trunk backend in a fairly straightforward manner. It might even be possible to setup a system to warn the upstream developers when we've broken something. -Hal> > -Dave > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-- Hal Finkel Postdoctoral Appointee Leadership Computing Facility Argonne National Laboratory
Andrew Trick <atrick at apple.com> writes:> We also can't afford to have compiler builds break because someone > upstream decided to change an API we rely on. We make monthly releases. > We have to have some control over when we introduce such changes. > > I've spent plenty of time dealing with a similar situation, and I > found it more efficient to keep all development upstream from > integration and release as in: LLVM trunk -> Cray trunk -> Cray > release.No way this will work. It takes FAR too long to get any code accepted upstream. Unfortunately, the LLVM project has some pretty major barriers to contributors. These have been discussed ad nauseum in the past so I won't repeat it again. -Dave
Evan Cheng <evan.cheng at apple.com> writes:>> We also can't afford to have compiler builds break because someone >> upstream decided to change an API we rely on. We make monthly releases. >> We have to have some control over when we introduce such changes. > > This has nothing to do with the scheduler roadmap. If you want to > engage the community on API compatibility discussion, it should be a > different thread.I was asked why we don't follow trunk. I answered the question.> But given how long you have been involved in LLVM, you should know the > philosophy: LLVM development is optimized for people who stay on > trunk.It's not very optimal for people outside of Apple who have critical release schedules. -Dave
Hal Finkel <hfinkel at anl.gov> writes:> Out of curiosity, can your frontend be configured to produce LLVM IR > that could be fed though the trunk backend passes, or is a tighter > integration required?Yes, we have this capability.> If it is possible to produce LLVM IR, then it might be possible to > test a lot, although not all, of the upstream changes using your > internal test suite in combination with the generic trunk backend in a > fairly straightforward manner.Unfortuantely out test system isn't set up to accept LLVM IR files. This is something I've wanted to have implemented for a long time but we're all stretched pretty thin here.> It might even be possible to setup a system to warn the upstream > developers when we've broken something.I would like that very much. It's a resource issue, not a technical one. I think it's pretty common for teams producing production-quality code to want to base pieces off of tested and released software. I had never head of any company doing otherwise until Apple with LLVM. It's not the normal way to produce releases off an unstable trunk. -Dave