Hi Raja, I'm open to suggestions. Our current release qualification is to bootstrap the compiler (similar to how GCC does their bootstrapping), run the test suites, and verify that there are no regressions. Improving the test suite is always welcome. In addition, we send out pre-release tarballs and have people in the community build and test their programs with it. This is not a perfect system, but it's one which works for us given the number of testers available, the amount of time and resources they have, and whatever fixes need to be merged into the release. ARM qualification is a bit trickier, because of the different specific chips out there, different OSes, and having to verify ARM, Thumb1, and Thumb2 for the same configurations. And the tests tend to run a bit slower than, say, an x86 chip. So it's mostly a matter of time and resources. Unless we can get people who are willing to perform these tests, we won't be able to release ARM as an official supported platform. -bw On Oct 11, 2011, at 9:44 AM, Raja Venkateswaran wrote:> I am very interested in seeing a qualification plan for ARM given that it is > a widely used target with several combinations of options/modes to be > tested. I & my team use ARM hardware for running tests and we run all test > LLVM test suite tests as part of qualification process. I had started a > similar conversation in llvm-commits, but this is probably the right forum. > It will save everyone a lot of time if we can all agree on qualification > tests and options for ARM and would be happy to initiate the discussion and > process for this. > > --Raja > >> ------------------------------ >> >> Message: 3 >> Date: Mon, 10 Oct 2011 17:13:49 -0700 >> From: Tanya Lattner <lattner at apple.com> >> Subject: Re: [LLVMdev] LLC ARM Backend maintainer >> To: Jim Grosbach <grosbach at apple.com> >> Cc: "llvmdev at cs.uiuc.edu" <llvmdev at cs.uiuc.edu> >> Message-ID: <A8FD895A-6307-402D-A38D-A3BF39FF43C6 at apple.com> >> Content-Type: text/plain; CHARSET=US-ASCII >> >> Exactly as Jim said :) >> >> I'm only talking about releases and release blockers. Its also a bit of a >> grey area when you are talking about just ARM codgen tests in the >> regression test suite because those should always be passing regardless >> of what target/os you are testing on (assuming you built the arm backend >> which the release team typically does). As it stands now we don't have >> ARM as a supported target just because there is no qualification plan for >> it (ie. no one running the test suite or some other benchmarks on an ARM >> device for example) and no real support to qualify it. If someone wants >> to initiate this process, it can be talked about post-3.0. >> >> -Tanya >> >> On Oct 10, 2011, at 8:47 AM, Jim Grosbach wrote: >> >>> No. Note the qualifying phrase "for releases" on Tanya's statement. If, >> during release testing, a regression is found on ARM compared to 2.9 >> results, it is not required by process to be considered a release >> blocker. That does not mean features can or should be enabled which >> knowingly break ARM. That's an entirely different situation. >>> >>> -Jim >>> >>> >>> On Oct 8, 2011, at 9:59 AM, Rotem, Nadav wrote: >>> >>>> Hi Tanya, >>>> >>>> The new type-legalization mode (-promote-elements) which enables >> vector-select in LLVM (and a nice perf boost for several workloads), is >> currently disabled because of a _single_ bug in the ARM codegen which >> makes a few tests fail. If ARM is not a supported target, can I mark >> these tests as 'XFAIL' and enable vector-select support in LLVM ? >>>> >>>> Thanks, >>>> Nadav >>>> >>>> >>>> -----Original Message----- >>>> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] >> On Behalf Of Tanya Lattner >>>> Sent: Saturday, October 08, 2011 01:10 >>>> To: Seb >>>> Cc: llvmdev at cs.uiuc.edu >>>> Subject: Re: [LLVMdev] LLC ARM Backend maintainer >>>> >>>> >>>> On Oct 7, 2011, at 1:07 AM, Seb wrote: >>>> >>>>> Hi all, >>>>> >>>>> To answer Eli question, I wanted to know who is actively working on >> ARM because I submitted some bug report (#11029, #9905) and don't know if >> someone is working on them, if/when the will be fixed. Maybe I just need >> to better understand LLVM release process, I've seen a mail in this list >> about it. >>>> >>>> Bugs get fixed if there are people to fix them. There are numerous >> people working on ARM and patches are also accepted. >>>> >>>> ARM is not currently a target that we support for releases. So those >> bugs are not release blockers. >>>> >>>> -Tanya >>>> >>>>> >>>>> -- Seb >>>>> _______________________________________________ >>>>> LLVM Developers mailing list >>>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> --------------------------------------------------------------------- >>>> Intel Israel (74) Limited >>>> >>>> This e-mail and any attachments may contain confidential material for >>>> the sole use of the intended recipient(s). Any review or distribution >>>> by others is strictly prohibited. If you are not the intended >>>> recipient, please contact the sender and delete all copies. >>>> >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >> >> >> > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On Oct 11, 2011, at 2:43 PM, David A. Greene wrote:> Bill Wendling <wendling at apple.com> writes: > >> Improving the test suite is always welcome. > > Do we have an idea of what sorts of improvements we'd like? Any codes > that we want to add, for example? What would be useful for ARM? > >> In addition, we send out pre-release tarballs and have people in the >> community build and test their programs with it. This is not a perfect >> system, but it's one which works for us given the number of testers >> available, the amount of time and resources they have, and whatever >> fixes need to be merged into the release. >> >> ARM qualification is a bit trickier, because of the different specific >> chips out there, different OSes, and having to verify ARM, Thumb1, and >> Thumb2 for the same configurations. And the tests tend to run a bit >> slower than, say, an x86 chip. So it's mostly a matter of time and >> resources. Unless we can get people who are willing to perform these >> tests, we won't be able to release ARM as an official supported >> platform. > > Resources isn't the only problem. I've asked several times about adding > my personal machines to the testing pool but I never get a reply. So > there they sit idle every day, processing the occasional e-mail when > they could be chewing LLVM tests. > > It is in fact highly in my own interest to get them running. I just > need to be pointed to the document that tells me what the buildbot > master expects to see and defines a procedure for adding them as slaves. >Daniel may have the details you need to get this up and running. -bw> One thing that could help these situations is virtualization. > > I've toyed with the idea of setting up various virtual machines to test > various OS/architecture combinations. With QEMU I can imagine testing > various ISAs as well. > > If there are any ARM full-system simulators we could use those as well. > I'd be happy to run them. > > -Dave
I think we need to think along two dimensions - Breadth of testing and depth
of testing
1. Breadth: What the best supported ARM ISA versions in LLVM ARM? Say its
armv6 and armv7; We need to
- regression test ARM mode, Thumb-2 and Thumb-1 mode (armv6)
- Performance/code-size test ARM mode, Thumb-2 and Thumb-1 modes
We need to agree on an optimization level for regression as well as
performance (such as -O3 for performance and -Os for code-size)
2. Depth:
(a) Adding more regression tests: Every new commit comes with a set of
tests, but these are just regression tests. We need global access to
validation suites; unfortunately most validation suites are commercial and
their licensing prohibits even proxy public use. What about leveraging some
other open source test suites?
(b) Adding more performance tests: We need to identify performance and
code-size regressions before committing. Currently there are wrappers for
SPEC. What other performance/code-size suites can we get? Should there be
guidelines for performance reporting on SPEC and/or other suites? A lot of
users depend on LLVM ARM performance/codesize remaining stable or getting
better, so any degradation will trigger extra work for all consumers.
3. Reporting:
We need a more formal reporting process of validation done for a commit.
Currently, the validation process for ARM is same as x86 (just run the tests
and make sure they pass). We need to expand reporting to include breadth and
depth above to ensure reduced work for community in tracking down
regressions.
Of course, all this is going to increase the threshold for committing.
Either the committer pays early by running all these tests or the community
pays late by fixing them. The risk of paying late is also an unstable LLVM
tip for ARM.
Availability of ARM hardware is certainly an issue. I can make available ARM
hardware to run regressions through buildbot (or some other bot mechanism),
but making login access available to ARM hardware (for debugging) raises
firewall and security issues. I would like to hear community's thoughts on
these.
Thanks
--Raja
> -----Original Message-----
> From: David A. Greene [mailto:greened at obbligato.org]
> Sent: Tuesday, October 11, 2011 2:43 PM
> To: Bill Wendling
> Cc: rajav at codeaurora.org; llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] ARM Qualification
>
> Bill Wendling <wendling at apple.com> writes:
>
> > Improving the test suite is always welcome.
>
> Do we have an idea of what sorts of improvements we'd like? Any codes
> that we want to add, for example? What would be useful for ARM?
>
> > In addition, we send out pre-release tarballs and have people in the
> > community build and test their programs with it. This is not a perfect
> > system, but it's one which works for us given the number of
testers
> > available, the amount of time and resources they have, and whatever
> > fixes need to be merged into the release.
> >
> > ARM qualification is a bit trickier, because of the different specific
> > chips out there, different OSes, and having to verify ARM, Thumb1, and
> > Thumb2 for the same configurations. And the tests tend to run a bit
> > slower than, say, an x86 chip. So it's mostly a matter of time and
> > resources. Unless we can get people who are willing to perform these
> > tests, we won't be able to release ARM as an official supported
> > platform.
>
> Resources isn't the only problem. I've asked several times about
adding
> my personal machines to the testing pool but I never get a reply. So
> there they sit idle every day, processing the occasional e-mail when
> they could be chewing LLVM tests.
>
> It is in fact highly in my own interest to get them running. I just
> need to be pointed to the document that tells me what the buildbot
> master expects to see and defines a procedure for adding them as slaves.
>
> One thing that could help these situations is virtualization.
>
> I've toyed with the idea of setting up various virtual machines to test
> various OS/architecture combinations. With QEMU I can imagine testing
> various ISAs as well.
>
> If there are any ARM full-system simulators we could use those as well.
> I'd be happy to run them.
>
> -Dave
On Oct 11, 2011, at 2:43 PM, David A. Greene wrote:> Bill Wendling <wendling at apple.com> writes: > >> Improving the test suite is always welcome. > > Do we have an idea of what sorts of improvements we'd like? Any codes > that we want to add, for example? What would be useful for ARM? > >> In addition, we send out pre-release tarballs and have people in the >> community build and test their programs with it. This is not a perfect >> system, but it's one which works for us given the number of testers >> available, the amount of time and resources they have, and whatever >> fixes need to be merged into the release. >> >> ARM qualification is a bit trickier, because of the different specific >> chips out there, different OSes, and having to verify ARM, Thumb1, and >> Thumb2 for the same configurations. And the tests tend to run a bit >> slower than, say, an x86 chip. So it's mostly a matter of time and >> resources. Unless we can get people who are willing to perform these >> tests, we won't be able to release ARM as an official supported >> platform. > > Resources isn't the only problem. I've asked several times about adding > my personal machines to the testing pool but I never get a reply. So > there they sit idle every day, processing the occasional e-mail when > they could be chewing LLVM tests. >I'm pretty sure this has been answered: http://lists.cs.uiuc.edu/pipermail/llvmdev/2010-September/034555.html But now the master has moved (this week), so its no longer osuosl. However, I do agree that it should be put into an appropriate place on the website (if its there, I can't easily find it). -Tanya> It is in fact highly in my own interest to get them running. I just > need to be pointed to the document that tells me what the buildbot > master expects to see and defines a procedure for adding them as slaves. > > One thing that could help these situations is virtualization. > > I've toyed with the idea of setting up various virtual machines to test > various OS/architecture combinations. With QEMU I can imagine testing > various ISAs as well. > > If there are any ARM full-system simulators we could use those as well. > I'd be happy to run them. > > -Dave > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev