On 9/8/14, 12:39 PM, Greg Fitzgerald wrote:> Hi Jon, > > Compiler-rt, libcxx, libcxxabi should all use the same > cross-compilation testing strategy. Compiler-rt already has a working > solution. If it is inadequate for libcxx or libcxxabi, can we start > by addressing those deficiencies?One issue to start with is that none of the libcxx/libcxxabi tests have RUN lines. All of the tests are either: *.fail.cpp ones that fail to compile, or *.pass.cpp ones that are expected to exit with a return status of 0 without failing an asserts. Granted, this is more of a test semantic issue, but there are a *lot* of tests to update in order to converge on compiler_rt's 'RUN: ' format. [Sidenote: I don't see any 'RUN:' lines on the builtins unit tests. How does it work for those ones?] Another issue, which Dan Albert recently pointed out to me, is that there are several libcxx tests which would require copying in of test data (in addition to test executables). I think that this necessitates inventing a new LIT syntax, something like '// DEPENDS-ON: filename.dat' in order to inform the lit.cfg that the dependent file also needs to be copied to the remote target. A third issue is that I don't think the current '%run' solution works for full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever on HOST, and the tests themselves on TARGET, where {BUILD,HOST,TARGET} are all different machines). This use case is of particular interest to me. I suppose %run could be renamed %run_target, add %run_host, and then have an implied no-op %run_build on commands that don't have %run_target or %run_host. Perhaps someone might want to have BUILD be a different machine than where LIT runs, in which case %run_build would have to be explicit (but I'm not really interested in that use case). Cheers, Jon> > Thanks, > Greg > >-- Jon Roelofs jonathan at codesourcery.com CodeSourcery / Mentor Embedded
Jon,> One issue to start with is that none of the libcxx/libcxxabi tests have RUN lines.Sorry, I didn't mean to imply that 'RUN:' lines should be added to libcxx tests. It's fine that every one is implicitly: // RUN: %clang %s -o %t && %run %t> I think that this necessitates inventing a new LIT syntax, something like '// DEPENDS-ON: filename.dat'If you didn't want to change lit, this would be a good place to add a 'RUN:' line. Something like: // RUN: %clang %s -o %t && %run %t filename.dat and then your %run implementation would see the dependency on the local file and ship it over to TARGET.> A third issue is that I don't think the current '%run' solution works for > full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever > on HOST, and the tests themselves on TARGET, where {BUILD, > HOST,TARGET} are all different machines).In compiler-rt, I'd configure such a build with the following CMake variables: LLVM_LIT=$BUILD/lit CMAKE_C_COMPILER=$HOST/clang COMPILER_RT_EMULATOR=$TARGET/qemu For libcxx, how about adding LIBCXX_EMULATOR, and then changing lit.cfg to acknowledge it when invoking executables? You'd then put your ssh functionality in a standalone script. If a command-line interface is too inflexible (or painful), perhaps add a LIBCXX_PY_EMULATOR variable that points to a Python module. -Greg On Mon, Sep 8, 2014 at 12:49 PM, Jonathan Roelofs <jonathan at codesourcery.com> wrote:> > > On 9/8/14, 12:39 PM, Greg Fitzgerald wrote: >> >> Hi Jon, >> >> Compiler-rt, libcxx, libcxxabi should all use the same >> cross-compilation testing strategy. Compiler-rt already has a working >> solution. If it is inadequate for libcxx or libcxxabi, can we start >> by addressing those deficiencies? > > One issue to start with is that none of the libcxx/libcxxabi tests have RUN > lines. All of the tests are either: *.fail.cpp ones that fail to compile, or > *.pass.cpp ones that are expected to exit with a return status of 0 without > failing an asserts. Granted, this is more of a test semantic issue, but > there are a *lot* of tests to update in order to converge on compiler_rt's > 'RUN: ' format. > > [Sidenote: I don't see any 'RUN:' lines on the builtins unit tests. How does > it work for those ones?] > > Another issue, which Dan Albert recently pointed out to me, is that there > are several libcxx tests which would require copying in of test data (in > addition to test executables). I think that this necessitates inventing a > new LIT syntax, something like '// DEPENDS-ON: filename.dat' in order to > inform the lit.cfg that the dependent file also needs to be copied to the > remote target. > > A third issue is that I don't think the current '%run' solution works for > full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever on > HOST, and the tests themselves on TARGET, where {BUILD,HOST,TARGET} are all > different machines). This use case is of particular interest to me. I > suppose %run could be renamed %run_target, add %run_host, and then have an > implied no-op %run_build on commands that don't have %run_target or > %run_host. Perhaps someone might want to have BUILD be a different machine > than where LIT runs, in which case %run_build would have to be explicit (but > I'm not really interested in that use case). > > > Cheers, > Jon >> >> >> Thanks, >> Greg >> >> > > -- > Jon Roelofs > jonathan at codesourcery.com > CodeSourcery / Mentor Embedded
On 9/8/14, 3:31 PM, Greg Fitzgerald wrote:> Jon, > >> One issue to start with is that none of the libcxx/libcxxabi tests have RUN lines. > > Sorry, I didn't mean to imply that 'RUN:' lines should be added to > libcxx tests. It's fine that every one is implicitly: > > // RUN: %clang %s -o %t && %run %t >... and then when an explicit 'RUN' line is provided, use that instead? That sounds reasonable.> >> I think that this necessitates inventing a new LIT syntax, something like '// DEPENDS-ON: filename.dat' > > If you didn't want to change lit, this would be a good place to add a > 'RUN:' line. Something like: > > // RUN: %clang %s -o %t && %run %t filename.dat > > and then your %run implementation would see the dependency on the > local file and ship it over to TARGET.This also sounds reasonable.> > >> A third issue is that I don't think the current '%run' solution works for >> full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever >> on HOST, and the tests themselves on TARGET, where {BUILD, >> HOST,TARGET} are all different machines). > > In compiler-rt, I'd configure such a build with the following CMake variables:I'm not sure we're on the same page... Your example, and the way that lit is currently set up, assumes BUILD and HOST are the same machine.> LLVM_LIT=$BUILD/lit > CMAKE_C_COMPILER=$HOST/clang > COMPILER_RT_EMULATOR=$TARGET/qemuLet me give a concrete example: BUILD=i686-pc-linux-gnu HOST=i386-pc-win32 TARGET=arm-elf (on say, real hardware, or i686-pc-linux/bin/qemu-system-arm, or i386-pc-win32/bin/qemu-system-arm) Here I'd like to run lit on the linux machine such that test compilation happens on the windows machine and test execution happens on arm hardware or qemu (where qemu is running on either the windows machine, or the linux machine, depending on how the lit.site.cfg is set up). Jon> > For libcxx, how about adding LIBCXX_EMULATOR, and then changing > lit.cfg to acknowledge it when invoking executables? You'd then put > your ssh functionality in a standalone script. If a command-line > interface is too inflexible (or painful), perhaps add a > LIBCXX_PY_EMULATOR variable that points to a Python module. > > -Greg > > > On Mon, Sep 8, 2014 at 12:49 PM, Jonathan Roelofs > <jonathan at codesourcery.com> wrote: >> >> >> On 9/8/14, 12:39 PM, Greg Fitzgerald wrote: >>> >>> Hi Jon, >>> >>> Compiler-rt, libcxx, libcxxabi should all use the same >>> cross-compilation testing strategy. Compiler-rt already has a working >>> solution. If it is inadequate for libcxx or libcxxabi, can we start >>> by addressing those deficiencies? >> >> One issue to start with is that none of the libcxx/libcxxabi tests have RUN >> lines. All of the tests are either: *.fail.cpp ones that fail to compile, or >> *.pass.cpp ones that are expected to exit with a return status of 0 without >> failing an asserts. Granted, this is more of a test semantic issue, but >> there are a *lot* of tests to update in order to converge on compiler_rt's >> 'RUN: ' format. >> >> [Sidenote: I don't see any 'RUN:' lines on the builtins unit tests. How does >> it work for those ones?] >> >> Another issue, which Dan Albert recently pointed out to me, is that there >> are several libcxx tests which would require copying in of test data (in >> addition to test executables). I think that this necessitates inventing a >> new LIT syntax, something like '// DEPENDS-ON: filename.dat' in order to >> inform the lit.cfg that the dependent file also needs to be copied to the >> remote target. >> >> A third issue is that I don't think the current '%run' solution works for >> full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever on >> HOST, and the tests themselves on TARGET, where {BUILD,HOST,TARGET} are all >> different machines). This use case is of particular interest to me. I >> suppose %run could be renamed %run_target, add %run_host, and then have an >> implied no-op %run_build on commands that don't have %run_target or >> %run_host. Perhaps someone might want to have BUILD be a different machine >> than where LIT runs, in which case %run_build would have to be explicit (but >> I'm not really interested in that use case). >> >> >> Cheers, >> Jon >>> >>> >>> Thanks, >>> Greg >>> >>> >> >> -- >> Jon Roelofs >> jonathan at codesourcery.com >> CodeSourcery / Mentor Embedded-- Jon Roelofs jonathan at codesourcery.com CodeSourcery / Mentor Embedded