David, I got some more work on the Livermore Loops and I found out that the issue is the difference in the parameters between a single step and a multi step compilation. When you compile "clang kernel06.c" it works fine, but when you get all steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options of each and how they interact is showing a bug in the code generated. This difference is due to the fact that I'm running the test-suite using LNT, while the build bots are running it using Make directly. I'd expect them both to be the same, but apparently they're quite different in what kind of parameters they use, passes they test and results they get. I think there are two courses of action here: 1. Identify the issue, isolate the case and create a bug to resolve later. 2. Make sure LNT does exactly what the build bots are doing I'm working on item 1 right now, not sure how item 2 can be solved... Of course, the fact that it's the not same flow meant we caught a bug in LLVM, but that's bound to create more confusion and broken commits, which is worse in the long run. Also, if we're not running LNT as often as buildbots, the benefit of having them different is sporadic at best. When I set up some tests to run on ARM I have done both direct and multi-step, to make sure they were generating the same code and in many cases I found that the order in which the passes were executed was breaking some tests. We managed to get the EDG bridge to set it up in the same way as the multi-pass would, so we would get similar results, but it doesn't seem to be the case with clang. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130103/9e65e767/attachment.html>
FYI, attached is a way to reproduce the error without the test-suite paraphernalia. cheers, --renato On 3 January 2013 12:05, Renato Golin <renato.golin at linaro.org> wrote:> David, > > I got some more work on the Livermore Loops and I found out that the issue > is the difference in the parameters between a single step and a multi step > compilation. > > When you compile "clang kernel06.c" it works fine, but when you get all > steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options of > each and how they interact is showing a bug in the code generated. > > This difference is due to the fact that I'm running the test-suite using > LNT, while the build bots are running it using Make directly. I'd expect > them both to be the same, but apparently they're quite different in what > kind of parameters they use, passes they test and results they get. > > I think there are two courses of action here: > > 1. Identify the issue, isolate the case and create a bug to resolve later. > 2. Make sure LNT does exactly what the build bots are doing > > I'm working on item 1 right now, not sure how item 2 can be solved... > > Of course, the fact that it's the not same flow meant we caught a bug in > LLVM, but that's bound to create more confusion and broken commits, which > is worse in the long run. > > Also, if we're not running LNT as often as buildbots, the benefit of > having them different is sporadic at best. > > When I set up some tests to run on ARM I have done both direct and > multi-step, to make sure they were generating the same code and in many > cases I found that the order in which the passes were executed was breaking > some tests. > > We managed to get the EDG bridge to set it up in the same way as the > multi-pass would, so we would get similar results, but it doesn't seem to > be the case with clang. > > cheers, > --renato >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130103/ff32b382/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: livermore-llvm-bug.zip Type: application/zip Size: 5933 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130103/ff32b382/attachment.zip>
Forgot to mention, commenting out the line that runs opt with link time opts solves the problem! ;) On 3 January 2013 15:08, Renato Golin <renato.golin at linaro.org> wrote:> FYI, attached is a way to reproduce the error without the > test-suite paraphernalia. > > cheers, > --renato > > > On 3 January 2013 12:05, Renato Golin <renato.golin at linaro.org> wrote: > >> David, >> >> I got some more work on the Livermore Loops and I found out that the >> issue is the difference in the parameters between a single step and a multi >> step compilation. >> >> When you compile "clang kernel06.c" it works fine, but when you get all >> steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options of >> each and how they interact is showing a bug in the code generated. >> >> This difference is due to the fact that I'm running the test-suite using >> LNT, while the build bots are running it using Make directly. I'd expect >> them both to be the same, but apparently they're quite different in what >> kind of parameters they use, passes they test and results they get. >> >> I think there are two courses of action here: >> >> 1. Identify the issue, isolate the case and create a bug to resolve later. >> 2. Make sure LNT does exactly what the build bots are doing >> >> I'm working on item 1 right now, not sure how item 2 can be solved... >> >> Of course, the fact that it's the not same flow meant we caught a bug in >> LLVM, but that's bound to create more confusion and broken commits, which >> is worse in the long run. >> >> Also, if we're not running LNT as often as buildbots, the benefit of >> having them different is sporadic at best. >> >> When I set up some tests to run on ARM I have done both direct and >> multi-step, to make sure they were generating the same code and in many >> cases I found that the order in which the passes were executed was breaking >> some tests. >> >> We managed to get the EDG bridge to set it up in the same way as the >> multi-pass would, so we would get similar results, but it doesn't seem to >> be the case with clang. >> >> cheers, >> --renato >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130103/3f628525/attachment.html>
+Daniel & Michael who work on the LNT infrastructure & might have some thoughts on the differences & their merits & motivations. On Thu, Jan 3, 2013 at 4:05 AM, Renato Golin <renato.golin at linaro.org> wrote:> David, > > I got some more work on the Livermore Loops and I found out that the issue > is the difference in the parameters between a single step and a multi step > compilation.Thanks for the investigation.> When you compile "clang kernel06.c" it works fine, but when you get all > steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options of > each and how they interact is showing a bug in the code generated.Sounds quite plausible.> This difference is due to the fact that I'm running the test-suite using > LNT, while the build bots are running it using Make directly. I'd expect > them both to be the same, but apparently they're quite different in what > kind of parameters they use, passes they test and results they get. > > I think there are two courses of action here: > > 1. Identify the issue, isolate the case and create a bug to resolve later. > 2. Make sure LNT does exactly what the build bots are doingPart of the issue here is whether or not the Make-based execution is still maintained/valued. I'm getting the impression that the LNT execution may be already, or be becoming, the standard way to run the test suite even when not gathering perf statistics. Michael/Daniel - is that the case? If so, should we rip out the direct Make execution, or do something to otherwise warn/disable it?> I'm working on item 1 right now, not sure how item 2 can be solved... > > Of course, the fact that it's the not same flow meant we caught a bug in > LLVM, but that's bound to create more confusion and broken commits, which is > worse in the long run.Yeah, unless there's some strong/specific motivation for this I'd be in favor of removing the difference (or removing the Make-based execution entirely)> Also, if we're not running LNT as often as buildbots, the benefit of having > them different is sporadic at best.we're running both pretty regularly, I think - if anything I suspect we might be running LNT on more configurations than the Make-based execution (except that on some LNT runners we're multisampling, so it's slower)> When I set up some tests to run on ARM I have done both direct and > multi-step, to make sure they were generating the same code and in many > cases I found that the order in which the passes were executed was breaking > some tests. > > We managed to get the EDG bridge to set it up in the same way as the > multi-pass would, so we would get similar results, but it doesn't seem to be > the case with clang. > > cheers, > --renato
On 3 January 2013 16:15, David Blaikie <dblaikie at gmail.com> wrote:> Part of the issue here is whether or not the Make-based execution is > still maintained/valued. I'm getting the impression that the LNT > execution may be already, or be becoming, the standard way to run the > test suite even when not gathering perf statistics. Michael/Daniel - > is that the case? >The main issue here is that Clang seems not to be choosing link time optimizations by default, while the make-based run calls it explicitly. So it is possible to achieve the same effect (ie. cover LTO) by turning them on on some runs (for all types of tests on all hardware configurations).> If so, should we rip out the direct Make execution, or do something to > otherwise warn/disable it? >I'd strongly recommend that we use only one test style (LNT) everywhere, and that we should test LTO more effectively. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130103/1386cdd8/attachment.html>
To weigh in here... On Thu, Jan 3, 2013 at 8:15 AM, David Blaikie <dblaikie at gmail.com> wrote:> +Daniel & Michael who work on the LNT infrastructure & might have some > thoughts on the differences & their merits & motivations. > > On Thu, Jan 3, 2013 at 4:05 AM, Renato Golin <renato.golin at linaro.org> > wrote: > > David, > > > > I got some more work on the Livermore Loops and I found out that the > issue > > is the difference in the parameters between a single step and a multi > step > > compilation. > > Thanks for the investigation. > > > When you compile "clang kernel06.c" it works fine, but when you get all > > steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options > of > > each and how they interact is showing a bug in the code generated. > > Sounds quite plausible. > > > This difference is due to the fact that I'm running the test-suite using > > LNT, while the build bots are running it using Make directly. I'd expect > > them both to be the same, but apparently they're quite different in what > > kind of parameters they use, passes they test and results they get. > > > > I think there are two courses of action here: > > > > 1. Identify the issue, isolate the case and create a bug to resolve > later. > > 2. Make sure LNT does exactly what the build bots are doing > > Part of the issue here is whether or not the Make-based execution is > still maintained/valued. I'm getting the impression that the LNT > execution may be already, or be becoming, the standard way to run the > test suite even when not gathering perf statistics. Michael/Daniel - > is that the case? >Well, the distinction isn't really between LNT and non-LNT, its between the TEST=nightly and TEST=simple style supported by the Makefiles. LNT uses the TEST=simple style and that is all I care to support. Historically, the old way of testing (TEST=nightly) used the various LLVM tools to effect a compilation because there weren't compilers that worked. However, this is a bad way to "test" the product that most users actually care about, which is the compiler. With TEST=simple, all the compilation is done using the compiler just as an end user would. If you want LTO, the right way to get it is to use the compilers support for LTO. This is how we test LTO internally. I've never tried to get LTO working on Linux, but it should be possible using the gold plugin and passing the right compiler options. If so, should we rip out the direct Make execution, or do something to> otherwise warn/disable it? >Per my other thread polling users of the test-suite, there are still people who use the Makefiles to do more custom things. I personally would love to deprecate them completely, but they do support some useful workflows. My ideal would be: 1. Migrate LNT to drive the test-suite using a more sane mechanism (not a glob of Makefiles). I would like the "more sane mechanism" to be lit-based. 2. Maybe do some work to make using lit to drive the test-suite more convenient and hopefully support some of the useful workflows the Makefiles support with less of the crap. 3. Deprecate the Makefiles, or at least let the die through lack of maintenance. Does that answer the parts you wanted my input on? - Daniel> > I'm working on item 1 right now, not sure how item 2 can be solved... > > > > Of course, the fact that it's the not same flow meant we caught a bug in > > LLVM, but that's bound to create more confusion and broken commits, > which is > > worse in the long run. > > Yeah, unless there's some strong/specific motivation for this I'd be > in favor of removing the difference (or removing the Make-based > execution entirely) > > > Also, if we're not running LNT as often as buildbots, the benefit of > having > > them different is sporadic at best. > > we're running both pretty regularly, I think - if anything I suspect > we might be running LNT on more configurations than the Make-based > execution (except that on some LNT runners we're multisampling, so > it's slower) > > > When I set up some tests to run on ARM I have done both direct and > > multi-step, to make sure they were generating the same code and in many > > cases I found that the order in which the passes were executed was > breaking > > some tests. > > > > We managed to get the EDG bridge to set it up in the same way as the > > multi-pass would, so we would get similar results, but it doesn't seem > to be > > the case with clang. > > > > cheers, > > --renato >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130107/50df24a4/attachment.html>