Hal Finkel <hfinkel at anl.gov> writes:>> I would like that very much. It's a resource issue, not a technical >> one. > > I think that it might be useful to think about changing the parameters > of this optimization problem. Could you release enough of the testing > system so that others could help?I can't make that decision and I very much doubt we could even do it. It's quite tied in to our entire release mechanism. Remember, this is a process that has been going for 30 years or more.> Could you release (some subset of) the tests as LLVM IR so that they > could be integrated with the existing buildbots? I understand that you > may view these things as valuable IP, but in reality, the opportunity > cost of not sharing may far outweigh any competitive advantage you get > by not sharing.Actually, we don't have any problem releasing tests. We have done so before when sending patches. The problem is the people we got the tests from. Some are from proprietary test suites, others are from sensitive codes, etc. It's often not up to us at all. A larger problem, I think, is that patches often get dropped and/or they take forever to get approved. The red tape is astounding. You ran into that with your scheduler change, which is obviously a Good Thing to us and would support users of 3.0 and 3.1 very well even if it were to be deprecated in 3.2. But yet it hasn't made it in and it looks like it would not be accepted no matter what. That is a symptom of the problem.>> I think it's pretty common for teams producing production-quality code >> to want to base pieces off of tested and released software. I had >> never head of any company doing otherwise until Apple with LLVM. >> It's not the normal way to produce releases off an unstable trunk. > > As far as I can tell, Apple tries very hard to keep trunk stable; it is > one of the most stable trunks with which I've worked. Obviously there > are regressions, but these tend to have a short lifespan. One of the > advantages of keeping a relatively stable trunk is that they get more > trunk testers, and that, in turn, help keeps trunk stable. As a result, > they can take releases from trunk, and this gives them a short > turn-around time on new features with low backporting overhead.Yes, I completely understand the reasoning and in an ideal world I would like to live off trunk. But if trunk is meant to be stable, why have releases at all? There must be value added to releases or we just shouldn't do them. It's that added value that we're counting on.> The disadvantage for outsiders is, however, that it forces your > releases to follow their releases (because of additional stabilization > activity prior to releases), and that may not be practical if your > release cycle is shorter than theirs. However, if the release cycle is > too short for the project's users, then we should think about > shortening it.As I said, we release monthy updates. Major releases are generally once a year but sometimes more frequently. LLVM doesn't do point releases and that's another issue we have to deal with. As I said, I am working on integrating trunk into our development tree via git as an experiment. Maybe it will work out really well and we'll be able to switch but I'm not counting on that. I think the LLVM project is mature enough that we really should be considering supporting release users much better. But of course that's coming from a selfish position. :) Still, I think it is a valid discussion to have. -Dave
On Thu, 10 May 2012 14:58:41 -0500 <dag at cray.com> wrote:> Hal Finkel <hfinkel at anl.gov> writes: > > >> I would like that very much. It's a resource issue, not a > >> technical one. > > > > I think that it might be useful to think about changing the > > parameters of this optimization problem. Could you release enough > > of the testing system so that others could help? > > I can't make that decision and I very much doubt we could even do it. > It's quite tied in to our entire release mechanism. Remember, this > is a process that has been going for 30 years or more. > > > Could you release (some subset of) the tests as LLVM IR so that they > > could be integrated with the existing buildbots? I understand that > > you may view these things as valuable IP, but in reality, the > > opportunity cost of not sharing may far outweigh any competitive > > advantage you get by not sharing. > > Actually, we don't have any problem releasing tests. We have done so > before when sending patches. The problem is the people we got the > tests from. Some are from proprietary test suites, others are from > sensitive codes, etc. It's often not up to us at all.I completely understand. Why don't we start by having you prepare LLVM IR files, and associated outputs, for x86_64 from your frontends only from open-source codes. As a first step, you could even just generate LLVM IR files for us from the codes in the LLVM test suite. We could setup a buildbot based on those files (which I believe would be easy to do), and then we can actively test trunk LLVM against those files.> > A larger problem, I think, is that patches often get dropped and/or > they take forever to get approved. The red tape is astounding. You > ran into that with your scheduler change, which is obviously a Good > Thing to us and would support users of 3.0 and 3.1 very well even if > it were to be deprecated in 3.2. But yet it hasn't made it in and it > looks like it would not be accepted no matter what. That is a > symptom of the problem.To be fair, the reason that my patch was not accepted was because it caused test-suite failures on x86. Does the patch work for you? If it does, then maybe the situation has changed, and we should reconsider the status of the patch. The patch actually had two parts: the IR->DAG modifications and the changes to the ILP scheduling heuristic. Changes to the ILP scheduling heuristic may be required regardless of how or where the critical chain is relaxed. Given that the patch caused test-suite failures on x86, I did not want to commit it as-is. I would have loved if someone else had worked to diagnose and/or fix the remaining problems (which may have been scattered among different backends), but it is hard to ask people to do that for a feature that would be deprecated in six months time. -Hal> > >> I think it's pretty common for teams producing production-quality > >> code to want to base pieces off of tested and released software. > >> I had never head of any company doing otherwise until Apple with > >> LLVM. It's not the normal way to produce releases off an unstable > >> trunk. > > > > As far as I can tell, Apple tries very hard to keep trunk stable; > > it is one of the most stable trunks with which I've worked. > > Obviously there are regressions, but these tend to have a short > > lifespan. One of the advantages of keeping a relatively stable > > trunk is that they get more trunk testers, and that, in turn, help > > keeps trunk stable. As a result, they can take releases from trunk, > > and this gives them a short turn-around time on new features with > > low backporting overhead. > > Yes, I completely understand the reasoning and in an ideal world I > would like to live off trunk. But if trunk is meant to be stable, > why have releases at all? There must be value added to releases or > we just shouldn't do them. It's that added value that we're counting > on. > > > The disadvantage for outsiders is, however, that it forces your > > releases to follow their releases (because of additional > > stabilization activity prior to releases), and that may not be > > practical if your release cycle is shorter than theirs. However, if > > the release cycle is too short for the project's users, then we > > should think about shortening it. > > As I said, we release monthy updates. Major releases are generally > once a year but sometimes more frequently. LLVM doesn't do point > releases and that's another issue we have to deal with. > > As I said, I am working on integrating trunk into our development tree > via git as an experiment. Maybe it will work out really well and > we'll be able to switch but I'm not counting on that. I think the > LLVM project is mature enough that we really should be considering > supporting release users much better. But of course that's coming > from a selfish position. :) Still, I think it is a valid discussion > to have. > > -Dave-- Hal Finkel Postdoctoral Appointee Leadership Computing Facility Argonne National Laboratory
Hal Finkel <hfinkel at anl.gov> writes:>> Actually, we don't have any problem releasing tests. We have done so >> before when sending patches. The problem is the people we got the >> tests from. Some are from proprietary test suites, others are from >> sensitive codes, etc. It's often not up to us at all. > > I completely understand. Why don't we start by having you prepare LLVM > IR files, and associated outputs, for x86_64 from your frontends only > from open-source codes. As a first step, you could even just generate > LLVM IR files for us from the codes in the LLVM test suite. We could > setup a buildbot based on those files (which I believe would be easy to > do), and then we can actively test trunk LLVM against those files.I like this idea. It'll work for C/C++ but not Fortran. Since there is no Fortran ABI one has to link with our Fortran compiler & libraries to get an executable that actually works. But let me think about this some more. I would really like to expand the LLVM testbase if we can. It will be a long process since I'll have to get all these tests approved for release. I can't give a timeline on that at all. I think it will be a very gradual process.> To be fair, the reason that my patch was not accepted was because it > caused test-suite failures on x86. Does the patch work for you?I'm hopefully going to try it within the next few days.> If it does, then maybe the situation has changed, and we should > reconsider the status of the patch. The patch actually had two parts: > the IR->DAG modifications and the changes to the ILP scheduling > heuristic. Changes to the ILP scheduling heuristic may be required > regardless of how or where the critical chain is relaxed.Ok, I will take a look at that.> Given that the patch caused test-suite failures on x86, I did not want > to commit it as-is.Yes, I understand that. But from the discussion I got the impression that the patch wasn't wanted because ScheduleDAG is going to be deprecated. If that's not the case I will certainly work to get it going!> I would have loved if someone else had worked to > diagnose and/or fix the remaining problems (which may have been > scattered among different backends), but it is hard to ask people to do > that for a feature that would be deprecated in six months time.Yeah, I understand. But for those of us working off releases it would not be deprecated in six months. That's probably moot now since 3.1 is almost out the door but I think the patch will still be useful for us. Believe me, I would really like to be able to work off trunk but I have to convince a lot of people here that that is possible. Starting with myself. :) -Dave