I am very interested in seeing a qualification plan for ARM given that it is a widely used target with several combinations of options/modes to be tested. I & my team use ARM hardware for running tests and we run all test LLVM test suite tests as part of qualification process. I had started a similar conversation in llvm-commits, but this is probably the right forum. It will save everyone a lot of time if we can all agree on qualification tests and options for ARM and would be happy to initiate the discussion and process for this. --Raja> ------------------------------ > > Message: 3 > Date: Mon, 10 Oct 2011 17:13:49 -0700 > From: Tanya Lattner <lattner at apple.com> > Subject: Re: [LLVMdev] LLC ARM Backend maintainer > To: Jim Grosbach <grosbach at apple.com> > Cc: "llvmdev at cs.uiuc.edu" <llvmdev at cs.uiuc.edu> > Message-ID: <A8FD895A-6307-402D-A38D-A3BF39FF43C6 at apple.com> > Content-Type: text/plain; CHARSET=US-ASCII > > Exactly as Jim said :) > > I'm only talking about releases and release blockers. Its also a bit of a > grey area when you are talking about just ARM codgen tests in the > regression test suite because those should always be passing regardless > of what target/os you are testing on (assuming you built the arm backend > which the release team typically does). As it stands now we don't have > ARM as a supported target just because there is no qualification plan for > it (ie. no one running the test suite or some other benchmarks on an ARM > device for example) and no real support to qualify it. If someone wants > to initiate this process, it can be talked about post-3.0. > > -Tanya > > On Oct 10, 2011, at 8:47 AM, Jim Grosbach wrote: > > > No. Note the qualifying phrase "for releases" on Tanya's statement. If, > during release testing, a regression is found on ARM compared to 2.9 > results, it is not required by process to be considered a release > blocker. That does not mean features can or should be enabled which > knowingly break ARM. That's an entirely different situation. > > > > -Jim > > > > > > On Oct 8, 2011, at 9:59 AM, Rotem, Nadav wrote: > > > >> Hi Tanya, > >> > >> The new type-legalization mode (-promote-elements) which enables > vector-select in LLVM (and a nice perf boost for several workloads), is > currently disabled because of a _single_ bug in the ARM codegen which > makes a few tests fail. If ARM is not a supported target, can I mark > these tests as 'XFAIL' and enable vector-select support in LLVM ? > >> > >> Thanks, > >> Nadav > >> > >> > >> -----Original Message----- > >> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] > On Behalf Of Tanya Lattner > >> Sent: Saturday, October 08, 2011 01:10 > >> To: Seb > >> Cc: llvmdev at cs.uiuc.edu > >> Subject: Re: [LLVMdev] LLC ARM Backend maintainer > >> > >> > >> On Oct 7, 2011, at 1:07 AM, Seb wrote: > >> > >>> Hi all, > >>> > >>> To answer Eli question, I wanted to know who is actively working on > ARM because I submitted some bug report (#11029, #9905) and don't know if > someone is working on them, if/when the will be fixed. Maybe I just need > to better understand LLVM release process, I've seen a mail in this list > about it. > >> > >> Bugs get fixed if there are people to fix them. There are numerous > people working on ARM and patches are also accepted. > >> > >> ARM is not currently a target that we support for releases. So those > bugs are not release blockers. > >> > >> -Tanya > >> > >>> > >>> -- Seb > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >> > >> _______________________________________________ > >> LLVM Developers mailing list > >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >> --------------------------------------------------------------------- > >> Intel Israel (74) Limited > >> > >> This e-mail and any attachments may contain confidential material for > >> the sole use of the intended recipient(s). Any review or distribution > >> by others is strictly prohibited. If you are not the intended > >> recipient, please contact the sender and delete all copies. > >> > >> > >> _______________________________________________ > >> LLVM Developers mailing list > >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > > > >
Hi Raja, I'm open to suggestions. Our current release qualification is to bootstrap the compiler (similar to how GCC does their bootstrapping), run the test suites, and verify that there are no regressions. Improving the test suite is always welcome. In addition, we send out pre-release tarballs and have people in the community build and test their programs with it. This is not a perfect system, but it's one which works for us given the number of testers available, the amount of time and resources they have, and whatever fixes need to be merged into the release. ARM qualification is a bit trickier, because of the different specific chips out there, different OSes, and having to verify ARM, Thumb1, and Thumb2 for the same configurations. And the tests tend to run a bit slower than, say, an x86 chip. So it's mostly a matter of time and resources. Unless we can get people who are willing to perform these tests, we won't be able to release ARM as an official supported platform. -bw On Oct 11, 2011, at 9:44 AM, Raja Venkateswaran wrote:> I am very interested in seeing a qualification plan for ARM given that it is > a widely used target with several combinations of options/modes to be > tested. I & my team use ARM hardware for running tests and we run all test > LLVM test suite tests as part of qualification process. I had started a > similar conversation in llvm-commits, but this is probably the right forum. > It will save everyone a lot of time if we can all agree on qualification > tests and options for ARM and would be happy to initiate the discussion and > process for this. > > --Raja > >> ------------------------------ >> >> Message: 3 >> Date: Mon, 10 Oct 2011 17:13:49 -0700 >> From: Tanya Lattner <lattner at apple.com> >> Subject: Re: [LLVMdev] LLC ARM Backend maintainer >> To: Jim Grosbach <grosbach at apple.com> >> Cc: "llvmdev at cs.uiuc.edu" <llvmdev at cs.uiuc.edu> >> Message-ID: <A8FD895A-6307-402D-A38D-A3BF39FF43C6 at apple.com> >> Content-Type: text/plain; CHARSET=US-ASCII >> >> Exactly as Jim said :) >> >> I'm only talking about releases and release blockers. Its also a bit of a >> grey area when you are talking about just ARM codgen tests in the >> regression test suite because those should always be passing regardless >> of what target/os you are testing on (assuming you built the arm backend >> which the release team typically does). As it stands now we don't have >> ARM as a supported target just because there is no qualification plan for >> it (ie. no one running the test suite or some other benchmarks on an ARM >> device for example) and no real support to qualify it. If someone wants >> to initiate this process, it can be talked about post-3.0. >> >> -Tanya >> >> On Oct 10, 2011, at 8:47 AM, Jim Grosbach wrote: >> >>> No. Note the qualifying phrase "for releases" on Tanya's statement. If, >> during release testing, a regression is found on ARM compared to 2.9 >> results, it is not required by process to be considered a release >> blocker. That does not mean features can or should be enabled which >> knowingly break ARM. That's an entirely different situation. >>> >>> -Jim >>> >>> >>> On Oct 8, 2011, at 9:59 AM, Rotem, Nadav wrote: >>> >>>> Hi Tanya, >>>> >>>> The new type-legalization mode (-promote-elements) which enables >> vector-select in LLVM (and a nice perf boost for several workloads), is >> currently disabled because of a _single_ bug in the ARM codegen which >> makes a few tests fail. If ARM is not a supported target, can I mark >> these tests as 'XFAIL' and enable vector-select support in LLVM ? >>>> >>>> Thanks, >>>> Nadav >>>> >>>> >>>> -----Original Message----- >>>> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] >> On Behalf Of Tanya Lattner >>>> Sent: Saturday, October 08, 2011 01:10 >>>> To: Seb >>>> Cc: llvmdev at cs.uiuc.edu >>>> Subject: Re: [LLVMdev] LLC ARM Backend maintainer >>>> >>>> >>>> On Oct 7, 2011, at 1:07 AM, Seb wrote: >>>> >>>>> Hi all, >>>>> >>>>> To answer Eli question, I wanted to know who is actively working on >> ARM because I submitted some bug report (#11029, #9905) and don't know if >> someone is working on them, if/when the will be fixed. Maybe I just need >> to better understand LLVM release process, I've seen a mail in this list >> about it. >>>> >>>> Bugs get fixed if there are people to fix them. There are numerous >> people working on ARM and patches are also accepted. >>>> >>>> ARM is not currently a target that we support for releases. So those >> bugs are not release blockers. >>>> >>>> -Tanya >>>> >>>>> >>>>> -- Seb >>>>> _______________________________________________ >>>>> LLVM Developers mailing list >>>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> --------------------------------------------------------------------- >>>> Intel Israel (74) Limited >>>> >>>> This e-mail and any attachments may contain confidential material for >>>> the sole use of the intended recipient(s). Any review or distribution >>>> by others is strictly prohibited. If you are not the intended >>>> recipient, please contact the sender and delete all copies. >>>> >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >> >> >> > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Bill Wendling <wendling at apple.com> writes:> Improving the test suite is always welcome.Do we have an idea of what sorts of improvements we'd like? Any codes that we want to add, for example? What would be useful for ARM?> In addition, we send out pre-release tarballs and have people in the > community build and test their programs with it. This is not a perfect > system, but it's one which works for us given the number of testers > available, the amount of time and resources they have, and whatever > fixes need to be merged into the release. > > ARM qualification is a bit trickier, because of the different specific > chips out there, different OSes, and having to verify ARM, Thumb1, and > Thumb2 for the same configurations. And the tests tend to run a bit > slower than, say, an x86 chip. So it's mostly a matter of time and > resources. Unless we can get people who are willing to perform these > tests, we won't be able to release ARM as an official supported > platform.Resources isn't the only problem. I've asked several times about adding my personal machines to the testing pool but I never get a reply. So there they sit idle every day, processing the occasional e-mail when they could be chewing LLVM tests. It is in fact highly in my own interest to get them running. I just need to be pointed to the document that tells me what the buildbot master expects to see and defines a procedure for adding them as slaves. One thing that could help these situations is virtualization. I've toyed with the idea of setting up various virtual machines to test various OS/architecture combinations. With QEMU I can imagine testing various ISAs as well. If there are any ARM full-system simulators we could use those as well. I'd be happy to run them. -Dave