Kaylor, Andrew via llvm-dev
2020-Apr-16 22:44 UTC
[llvm-dev] [cfe-dev] Adding SYCL tests in test-suite
Thanks, Johannes! It would be nice to have some additional infrastructure to control execution of tests that have special resource requirements like this. We've seen some problems in our internal testing with parallel test execution causing system gridlock. Having a common way to address that would be great. One reason I thought separate SYCL folders (either at the top level or elsewhere in the tree) would be useful is that I think we'll probably want a single option to turn these tests on or off as a group. A "parallel" folder may make sense for the same reason. I see your point about mixed languages, but perhaps we could add a "parallel/mixed" subfolder if/when such a test is added. Any other opinions? -Andy -----Original Message----- From: Johannes Doerfert <johannesdoerfert at gmail.com> Sent: Wednesday, April 15, 2020 11:45 PM To: Kaylor, Andrew <andrew.kaylor at intel.com> Cc: LLVM Developers <llvm-dev at lists.llvm.org>; cfe-dev at lists.llvm.org Subject: Re: [cfe-dev] Adding SYCL tests in test-suite Hi Andy, On 4/15/20 7:17 PM, Kaylor, Andrew via cfe-dev wrote: > We'd like to some SYCL tests to LLVM's test suite. The SYCL support in > the LLVM repo is still very much a work-in-progress, but since the > test-suite is supposed to be able to support compilers other than > clang, I thought it would be reasonable to start adding the tests > there now, disabled by default, rather than maintaining a fork of the > test-suite repo until SYCL support is fully in place in the main repo. I think adding such tests is generally a good idea and I do think we could have them in the test suite while SYCL support in clang is still maturing. > These tests would involve compiling one or more source files > containing SYCL kernels and then executing the resulting binaries > using either host, CPU, or accelerator devices, based on the user's > configuration. Running these tests in some configurations would > require an OpenCL runtime and, of course, the chosen hardware. Now this is where the dicey part begins. The test suite, as of now, is not well equipped for tests "competing for shared resources". I mean, parallel tests or tests running on accelerators. To be fair, I am assuming we don't want the test suite to be run completely sequential if the SYCL parts are run, potentially with the non-SYCL parts. I'm saying this because OpenMP is in a similar position right now. Our goal is to run OpenMP tests as part of the test suite but that requires infrastructure we only started to build (downstream so far). I think the goals of basically all parallel (and offloading) extensions are pretty much the same so we should work together on them. > Am I over-reaching? No. > If not, I'd like some feedback on where the tests should go. For the > most part, I think these will be correctness tests, though I expect > we'll want to add benchmarks at some point. For the correctness tests, > I thought it would make sense to either (a) create a top-level SYCL > folder with SingleSource and MultiSource folders under it, or (b) > create SYCL folders in appropriate locations under the existing > SingleSource and MultiSource folders (e.g. > llvm-test-suite/SingleSource/UnitTests/SYCL). I would favor (b) but I > wasn't sure if SYCL is enough of a departure from the normal C/C++ > tests to push it into its own location. Option (c) would be an parallel or accelerator subfolder in which we nest things. Howe we nest things is again the question you asked. While I can see the appeal of a (almost) top-level SYCL folder it also has drawbacks, mostly when it comes to these really cool applications that come in 2+ different languages. Keeping these sources together as they are seems pretty important. In that spirit, I am unsure why we should do any of the options (a-c) at all. We could just put a test, let's say XYZ, into the existing structure SingleSource/Benchmarks/XYZ regardless of the language XYZ is written in. We mixed C and C++, why stop there. To be honest, I don't really care too much and the above is mostly to start a discussion on the pros and cons of the different options. > I haven't added anything to test-suite before, so if I'm approaching > this in completely the wrong way don't be shy about telling me so. I think writing an email like this is exactly the right way :) Cheers, Johannes > Thanks, > Andy > > > > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Johannes Doerfert via llvm-dev
2020-Jun-08 21:00 UTC
[llvm-dev] [cfe-dev] Adding SYCL tests in test-suite
Hi Andy, [-cfe-dev, llvm-dev seems sufficient for this] First, apologies for not responding earlier. I CC'ed a few people that might be interested in this (or at least in the path we'll take). On 4/16/20 5:44 PM, Kaylor, Andrew wrote:> Thanks, Johannes! > > It would be nice to have some additional infrastructure to control execution of tests that have special resource requirements like this. We've seen some problems in our internal testing with parallel test execution causing system gridlock. Having a common way to address that would be great.@Neeraj started to work on this, I'm not sure where exactly we are though.> One reason I thought separate SYCL folders (either at the top level or elsewhere in the tree) would be useful is that I think we'll probably want a single option to turn these tests on or off as a group. A "parallel" folder may make sense for the same reason. I see your point about mixed languages, but perhaps we could add a "parallel/mixed" subfolder if/when such a test is added.Agreed, something like this would make sense to me: llvm-test-suite/SingleSource/... llvm-test-suite/parallel/SYCL/... llvm-test-suite/parallel/OpenMP/... llvm-test-suite/parallel/multiple/proxyapps/... llvm-test-suite/parallel/multiple/cloverleaf/... Thoughts? Cheers, Johannes> Any other opinions? > > -Andy > > > -----Original Message----- > From: Johannes Doerfert <johannesdoerfert at gmail.com> > Sent: Wednesday, April 15, 2020 11:45 PM > To: Kaylor, Andrew <andrew.kaylor at intel.com> > Cc: LLVM Developers <llvm-dev at lists.llvm.org>; cfe-dev at lists.llvm.org > Subject: Re: [cfe-dev] Adding SYCL tests in test-suite > > > Hi Andy, > > On 4/15/20 7:17 PM, Kaylor, Andrew via cfe-dev wrote: > > We'd like to some SYCL tests to LLVM's test suite. The SYCL support in > the LLVM repo is still very much a work-in-progress, but since the > test-suite is supposed to be able to support compilers other than > clang, I thought it would be reasonable to start adding the tests > there now, disabled by default, rather than maintaining a fork of the > test-suite repo until SYCL support is fully in place in the main repo. > > I think adding such tests is generally a good idea and I do think we could have them in the test suite while SYCL support in clang is still maturing. > > > > These tests would involve compiling one or more source files > containing SYCL kernels and then executing the resulting binaries > using either host, CPU, or accelerator devices, based on the user's > configuration. Running these tests in some configurations would > require an OpenCL runtime and, of course, the chosen hardware. > > Now this is where the dicey part begins. The test suite, as of now, is not well equipped for tests "competing for shared resources". I mean, parallel tests or tests running on accelerators. To be fair, I am assuming we don't want the test suite to be run completely sequential if the SYCL parts are run, potentially with the non-SYCL parts. I'm saying this because OpenMP is in a similar position right now. > > Our goal is to run OpenMP tests as part of the test suite but that requires infrastructure we only started to build (downstream so far). I think the goals of basically all parallel (and offloading) extensions are pretty much the same so we should work together on them. > > > Am I over-reaching? > > No. > > > > If not, I'd like some feedback on where the tests should go. For the > most part, I think these will be correctness tests, though I expect > we'll want to add benchmarks at some point. For the correctness tests, > I thought it would make sense to either (a) create a top-level SYCL > folder with SingleSource and MultiSource folders under it, or (b) > create SYCL folders in appropriate locations under the existing > SingleSource and MultiSource folders (e.g. > > llvm-test-suite/SingleSource/UnitTests/SYCL). I would favor (b) but I > wasn't sure if SYCL is enough of a departure from the normal C/C++ > tests to push it into its own location. > > Option (c) would be an parallel or accelerator subfolder in which we nest things. Howe we nest things is again the question you asked. While I can see the appeal of a (almost) top-level SYCL folder it also has drawbacks, mostly when it comes to these really cool applications that come in 2+ different languages. Keeping these sources together as they are seems pretty important. In that spirit, I am unsure why we should do any of the options (a-c) at all. We could just put a test, let's say XYZ, into the existing structure SingleSource/Benchmarks/XYZ regardless of the language XYZ is written in. We mixed C and C++, why stop there. > > To be honest, I don't really care too much and the above is mostly to start a discussion on the pros and cons of the different options. > > > I haven't added anything to test-suite before, so if I'm approaching > this in completely the wrong way don't be shy about telling me so. > > I think writing an email like this is exactly the right way :) > > Cheers, > Johannes > > > > Thanks, > > Andy > > > > > > > > _______________________________________________ > > cfe-dev mailing list > > cfe-dev at lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200608/e384c734/attachment.html>
Kaylor, Andrew via llvm-dev
2020-Jun-08 21:53 UTC
[llvm-dev] [cfe-dev] Adding SYCL tests in test-suite
Hi Johannes, The structure you suggested makes sense to me. Vladimir Lazarev has been working on moving some end-to-end tests out of the source tree (in the intel/llvm GitHub branch where the parts of our SYCL development that aren’t ready to be included in the main LLVM repo are being shared). He has a local working copy that can run the tests with various hardware and device runtimes. The last version of this that I saw put the tests in llvm-test-suite/SYCL, but it should be easy enough to move them another level deeper to create a parallel structure to be shared with OpenMP. Vladimir is on vacation right now, but I believe he’ll be preparing a patch for review shortly after he returns. Thanks, Andy From: Johannes Doerfert <johannesdoerfert at gmail.com> Sent: Monday, June 08, 2020 2:00 PM To: Kaylor, Andrew <andrew.kaylor at intel.com> Cc: LLVM Developers <llvm-dev at lists.llvm.org>; bhomerding at anl.gov; Finkel, Hal J. <hfinkel at anl.gov>; Kruse, Michael <michael.kruse at anl.gov>; Malik,Abid <amalik at bnl.gov>; Clement, Valentin <clementv at ornl.gov>; Neeraj Ganu <neeraj.ganu at stonybrook.edu> Subject: Re: [cfe-dev] Adding SYCL tests in test-suite Hi Andy, [-cfe-dev, llvm-dev seems sufficient for this] First, apologies for not responding earlier. I CC'ed a few people that might be interested in this (or at least in the path we'll take). On 4/16/20 5:44 PM, Kaylor, Andrew wrote: Thanks, Johannes! It would be nice to have some additional infrastructure to control execution of tests that have special resource requirements like this. We've seen some problems in our internal testing with parallel test execution causing system gridlock. Having a common way to address that would be great. @Neeraj started to work on this, I'm not sure where exactly we are though. One reason I thought separate SYCL folders (either at the top level or elsewhere in the tree) would be useful is that I think we'll probably want a single option to turn these tests on or off as a group. A "parallel" folder may make sense for the same reason. I see your point about mixed languages, but perhaps we could add a "parallel/mixed" subfolder if/when such a test is added. Agreed, something like this would make sense to me: llvm-test-suite/SingleSource/... llvm-test-suite/parallel/SYCL/... llvm-test-suite/parallel/OpenMP/... llvm-test-suite/parallel/multiple/proxyapps/... llvm-test-suite/parallel/multiple/cloverleaf/... Thoughts? Cheers, Johannes Any other opinions? -Andy -----Original Message----- From: Johannes Doerfert <johannesdoerfert at gmail.com><mailto:johannesdoerfert at gmail.com> Sent: Wednesday, April 15, 2020 11:45 PM To: Kaylor, Andrew <andrew.kaylor at intel.com><mailto:andrew.kaylor at intel.com> Cc: LLVM Developers <llvm-dev at lists.llvm.org><mailto:llvm-dev at lists.llvm.org>; cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org> Subject: Re: [cfe-dev] Adding SYCL tests in test-suite Hi Andy, On 4/15/20 7:17 PM, Kaylor, Andrew via cfe-dev wrote: > We'd like to some SYCL tests to LLVM's test suite. The SYCL support in > the LLVM repo is still very much a work-in-progress, but since the > test-suite is supposed to be able to support compilers other than > clang, I thought it would be reasonable to start adding the tests > there now, disabled by default, rather than maintaining a fork of the > test-suite repo until SYCL support is fully in place in the main repo. I think adding such tests is generally a good idea and I do think we could have them in the test suite while SYCL support in clang is still maturing. > These tests would involve compiling one or more source files > containing SYCL kernels and then executing the resulting binaries > using either host, CPU, or accelerator devices, based on the user's > configuration. Running these tests in some configurations would > require an OpenCL runtime and, of course, the chosen hardware. Now this is where the dicey part begins. The test suite, as of now, is not well equipped for tests "competing for shared resources". I mean, parallel tests or tests running on accelerators. To be fair, I am assuming we don't want the test suite to be run completely sequential if the SYCL parts are run, potentially with the non-SYCL parts. I'm saying this because OpenMP is in a similar position right now. Our goal is to run OpenMP tests as part of the test suite but that requires infrastructure we only started to build (downstream so far). I think the goals of basically all parallel (and offloading) extensions are pretty much the same so we should work together on them. > Am I over-reaching? No. > If not, I'd like some feedback on where the tests should go. For the > most part, I think these will be correctness tests, though I expect > we'll want to add benchmarks at some point. For the correctness tests, > I thought it would make sense to either (a) create a top-level SYCL > folder with SingleSource and MultiSource folders under it, or (b) > create SYCL folders in appropriate locations under the existing > SingleSource and MultiSource folders (e.g. > llvm-test-suite/SingleSource/UnitTests/SYCL). I would favor (b) but I > wasn't sure if SYCL is enough of a departure from the normal C/C++ > tests to push it into its own location. Option (c) would be an parallel or accelerator subfolder in which we nest things. Howe we nest things is again the question you asked. While I can see the appeal of a (almost) top-level SYCL folder it also has drawbacks, mostly when it comes to these really cool applications that come in 2+ different languages. Keeping these sources together as they are seems pretty important. In that spirit, I am unsure why we should do any of the options (a-c) at all. We could just put a test, let's say XYZ, into the existing structure SingleSource/Benchmarks/XYZ regardless of the language XYZ is written in. We mixed C and C++, why stop there. To be honest, I don't really care too much and the above is mostly to start a discussion on the pros and cons of the different options. > I haven't added anything to test-suite before, so if I'm approaching > this in completely the wrong way don't be shy about telling me so. I think writing an email like this is exactly the right way :) Cheers, Johannes > Thanks, > Andy > > > > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200608/e3a1d9ba/attachment.html>