John Criswell via llvm-dev
2016-Nov-24 15:02 UTC
[llvm-dev] Running "different tests" on cmake based test-suite build
On 11/24/16 3:12 AM, Arnaud De Grandmaison via llvm-dev wrote:> > Hi Sandeep, > > The CMake version of the test-suite has been improved (see > https://reviews.llvm.org/D21360) so that it’s easy to plug additional > test suites. A starting point would be to look at how the > TEST_SUITE_SUBDIRS variable in the top level CMakeLists.txt is used. >I think this bug report is about adding new programs to the test suite. I don't think Sandeep is asking how to add more test programs to the test suite; he is asking how to add new Makefiles to the test suite to make it run new experiments that it did not run previously. As context, several previous research projects used an interface within the old autoconf test suite to extend it to compile programs with different compiler options and run performance experiments on them. For example, the SAFECode and Automatic Pool Allocation projects added rules to compile and run programs with additional transforms and to record the results in CSV files which could be imported into other programs for creating graphs for papers. All of this could be done without modifying any files within the test-suite project; one simply created additional Makefiles, set come variables on the make command line, and fired off the experiments with "make." I believe Sandeep is asking whether similar functionality exists within the new Cmake build system and, if so, how to use it. I am curious, too. It would make running some experiments for my research projects more convenient. I just assumed this feature was deprecated as I got the impression that I was the only one still using it. Regards, John Criswell> Kind regards, > > Arnaud > > *From:*llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of > *Sandeep Dasgupta via llvm-dev > *Sent:* 24 November 2016 02:32 > *To:* llvm-dev at lists.llvm.org > *Subject:* [llvm-dev] Running "different tests" on cmake based > test-suite build > > Hello Friends, > > With configure based build deprecated on llvm, how can I run different > tests/pass on the suite > (http://llvm.org/docs/TestSuiteMakefileGuide.html#running-different-tests)? > > > Right now I am having a hacky solution, which includes running the > "still present" configure file of test-suite to generate the > MAKE.<MYPASS>.Makefile; > > cd external-testsuite-build-loc > > <llvmsrc>/project/test-suite/configure --enable-optimized > -disable-assertions > > but while running 'make TEST=PASS' I am getting error for trying to > include $(LLVM_OBJ_ROOT)/Makefile.config which is obviously not there > on a cmake based llvm build. I have to edit the generated makefile to > bypass this. > > Can you please point me to some clean solution or let me know if am > missing something? > > -- > *With Thanks and Regards* > Sandeep Dasgupta > Research Assistant CS @UIUC > (http://web.engr.illinois.edu/~sdasgup3/ > <http://web.engr.illinois.edu/%7Esdasgup3/>) > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- John Criswell Assistant Professor Department of Computer Science, University of Rochester http://www.cs.rochester.edu/u/criswell -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161124/4287ae88/attachment.html>
Dasgupta, Sandeep via llvm-dev
2016-Nov-24 22:57 UTC
[llvm-dev] Running "different tests" on cmake based test-suite build
Hello All As John rightly pointed out, my question is about adding the capability to build the testsuite differently and not about how to add other testsuites. For example before configure build were deprecated, we can do something like make TEST=mypass, to build the testsuite according to the makefile rules in TEST.mypass.Makefile and that was handy if we want to test out own passes on the testsuite. To achieve the same in the current cmake build system, I need to do a hack solution as mentioned in my first post. Is there any clean way to achieve the same. Thanks Sandeep Sent via the Samsung Galaxy Note® 3, an AT&T 4G LTE smartphone -------- Original message -------- From: John Criswell <jtcriswel at gmail.com> Date: 11/24/2016 9:02 AM (GMT-06:00) To: Arnaud De Grandmaison <Arnaud.DeGrandmaison at arm.com>, "Dasgupta, Sandeep" <sdasgup3 at illinois.edu> Cc: llvm-dev at lists.llvm.org, nd <nd at arm.com> Subject: Re: [llvm-dev] Running "different tests" on cmake based test-suite build On 11/24/16 3:12 AM, Arnaud De Grandmaison via llvm-dev wrote: Hi Sandeep, The CMake version of the test-suite has been improved (see https://reviews.llvm.org/D21360<https://urldefense.proofpoint.com/v2/url?u=https-3A__reviews.llvm.org_D21360&d=DQMDaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=at_zHdaJ5pMt_A20sKa_JLbvjeXndNos3PveVfgUF0o&m=elm0JHrSir46GeQfeVXpM9G9IL8qP9v_jqIz9nqEDrI&s=69KaBU9uj9j_L8_uw9gE8Vcq2CN4j7BH0vab0K97pI8&e=>) so that it’s easy to plug additional test suites. A starting point would be to look at how the TEST_SUITE_SUBDIRS variable in the top level CMakeLists.txt is used. I think this bug report is about adding new programs to the test suite. I don't think Sandeep is asking how to add more test programs to the test suite; he is asking how to add new Makefiles to the test suite to make it run new experiments that it did not run previously. As context, several previous research projects used an interface within the old autoconf test suite to extend it to compile programs with different compiler options and run performance experiments on them. For example, the SAFECode and Automatic Pool Allocation projects added rules to compile and run programs with additional transforms and to record the results in CSV files which could be imported into other programs for creating graphs for papers. All of this could be done without modifying any files within the test-suite project; one simply created additional Makefiles, set come variables on the make command line, and fired off the experiments with "make." I believe Sandeep is asking whether similar functionality exists within the new Cmake build system and, if so, how to use it. I am curious, too. It would make running some experiments for my research projects more convenient. I just assumed this feature was deprecated as I got the impression that I was the only one still using it. Regards, John Criswell Kind regards, Arnaud From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Sandeep Dasgupta via llvm-dev Sent: 24 November 2016 02:32 To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [llvm-dev] Running "different tests" on cmake based test-suite build Hello Friends, With configure based build deprecated on llvm, how can I run different tests/pass on the suite (http://llvm.org/docs/TestSuiteMakefileGuide.html#running-different-tests<https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_docs_TestSuiteMakefileGuide.html-23running-2Ddifferent-2Dtests&d=DQMDaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=at_zHdaJ5pMt_A20sKa_JLbvjeXndNos3PveVfgUF0o&m=elm0JHrSir46GeQfeVXpM9G9IL8qP9v_jqIz9nqEDrI&s=Jykf6DJSLcnbNS1O4aEwAFiUFeyMcoQb8n0qOZfhnSc&e=>)? Right now I am having a hacky solution, which includes running the "still present" configure file of test-suite to generate the MAKE.<MYPASS>.Makefile; cd external-testsuite-build-loc <llvmsrc>/project/test-suite/configure --enable-optimized -disable-assertions but while running 'make TEST=PASS' I am getting error for trying to include $(LLVM_OBJ_ROOT)/Makefile.config which is obviously not there on a cmake based llvm build. I have to edit the generated makefile to bypass this. Can you please point me to some clean solution or let me know if am missing something? -- With Thanks and Regards Sandeep Dasgupta Research Assistant CS @UIUC (http://web.engr.illinois.edu/~sdasgup3/<http://web.engr.illinois.edu/%7Esdasgup3/>) _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=DQMDaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=at_zHdaJ5pMt_A20sKa_JLbvjeXndNos3PveVfgUF0o&m=elm0JHrSir46GeQfeVXpM9G9IL8qP9v_jqIz9nqEDrI&s=-MxG49PzHWKO5EeBSgjRewqderemUUsFzJPt5vmmSOE&e=> -- John Criswell Assistant Professor Department of Computer Science, University of Rochester http://www.cs.rochester.edu/u/criswell<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cs.rochester.edu_u_criswell&d=DQMDaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=at_zHdaJ5pMt_A20sKa_JLbvjeXndNos3PveVfgUF0o&m=elm0JHrSir46GeQfeVXpM9G9IL8qP9v_jqIz9nqEDrI&s=b5hyuBgRB-MigEMyXeeDgzgAG8FwHHfVrL1oFF-d0Yw&e=> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161124/2d89a6f7/attachment.html>
Sandeep Dasgupta via llvm-dev
2016-Nov-25 02:58 UTC
[llvm-dev] Running "different tests" on cmake based test-suite build
Just as an aside, as far as running a custom pass on testsuite, one solution that I was using is to configure testsuite using cmake CC=<path to my_own_script> where my_own_script might contain something like: clang -Xclang -load -Xclang mypass.so "$@" Nevertheless, the previous "make TEST=mypass" solution was much more handy in term of the different ways you can build or report the results. - Sandeep On 11/24/2016 04:57 PM, Dasgupta, Sandeep via llvm-dev wrote:> Hello All > As John rightly pointed out, my question is about adding the > capability to build the testsuite differently and not about how to add > other testsuites. > For example before configure build were deprecated, we can do > something like make TEST=mypass, to build the testsuite according to > the makefile rules in TEST.mypass.Makefile and that was handy if we > want to test out own passes on the testsuite. To achieve the same in > the current cmake build system, I need to do a hack solution as > mentioned in my first post. Is there any clean way to achieve the same. > > Thanks > Sandeep > > > > Sent via the Samsung Galaxy Note® 3, an AT&T 4G LTE smartphone > > > -------- Original message -------- > From: John Criswell <jtcriswel at gmail.com> > Date: 11/24/2016 9:02 AM (GMT-06:00) > To: Arnaud De Grandmaison <Arnaud.DeGrandmaison at arm.com>, "Dasgupta, > Sandeep" <sdasgup3 at illinois.edu> > Cc: llvm-dev at lists.llvm.org, nd <nd at arm.com> > Subject: Re: [llvm-dev] Running "different tests" on cmake based > test-suite build > > On 11/24/16 3:12 AM, Arnaud De Grandmaison via llvm-dev wrote: >> >> Hi Sandeep, >> >> The CMake version of the test-suite has been improved (see >> https://reviews.llvm.org/D21360) so that it’s easy to plug additional >> test suites. A starting point would be to look at how the >> TEST_SUITE_SUBDIRS variable in the top level CMakeLists.txt is used. >> > > I think this bug report is about adding new programs to the test > suite. I don't think Sandeep is asking how to add more test programs > to the test suite; he is asking how to add new Makefiles to the test > suite to make it run new experiments that it did not run previously. > > As context, several previous research projects used an interface > within the old autoconf test suite to extend it to compile programs > with different compiler options and run performance experiments on > them. For example, the SAFECode and Automatic Pool Allocation > projects added rules to compile and run programs with additional > transforms and to record the results in CSV files which could be > imported into other programs for creating graphs for papers. All of > this could be done without modifying any files within the test-suite > project; one simply created additional Makefiles, set come variables > on the make command line, and fired off the experiments with "make." > > I believe Sandeep is asking whether similar functionality exists > within the new Cmake build system and, if so, how to use it. I am > curious, too. It would make running some experiments for my research > projects more convenient. I just assumed this feature was deprecated > as I got the impression that I was the only one still using it. > > Regards, > > John Criswell > >> Kind regards, >> >> Arnaud >> >> *From:*llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf >> Of *Sandeep Dasgupta via llvm-dev >> *Sent:* 24 November 2016 02:32 >> *To:* llvm-dev at lists.llvm.org >> *Subject:* [llvm-dev] Running "different tests" on cmake based >> test-suite build >> >> Hello Friends, >> >> With configure based build deprecated on llvm, how can I run >> different tests/pass on the suite >> (http://llvm.org/docs/TestSuiteMakefileGuide.html#running-different-tests >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_docs_TestSuiteMakefileGuide.html-23running-2Ddifferent-2Dtests&d=DQMDaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=at_zHdaJ5pMt_A20sKa_JLbvjeXndNos3PveVfgUF0o&m=elm0JHrSir46GeQfeVXpM9G9IL8qP9v_jqIz9nqEDrI&s=Jykf6DJSLcnbNS1O4aEwAFiUFeyMcoQb8n0qOZfhnSc&e=>)? >> >> >> Right now I am having a hacky solution, which includes running the >> "still present" configure file of test-suite to generate the >> MAKE.<MYPASS>.Makefile; >> >> cd external-testsuite-build-loc >> >> <llvmsrc>/project/test-suite/configure --enable-optimized >> -disable-assertions >> >> but while running 'make TEST=PASS' I am getting error for trying to >> include $(LLVM_OBJ_ROOT)/Makefile.config which is obviously not there >> on a cmake based llvm build. I have to edit the generated makefile to >> bypass this. >> >> Can you please point me to some clean solution or let me know if am >> missing something? >> >> -- >> *With Thanks and Regards* >> Sandeep Dasgupta >> Research Assistant CS @UIUC >> (http://web.engr.illinois.edu/~sdasgup3/ >> <http://web.engr.illinois.edu/%7Esdasgup3/>) >> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > -- > John Criswell > Assistant Professor > Department of Computer Science, University of Rochester > http://www.cs.rochester.edu/u/criswell > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- *With Thanks and Regards* Sandeep Dasgupta Research Assistant CS @UIUC (http://web.engr.illinois.edu/~sdasgup3/) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161124/d6eb680c/attachment-0001.html>