Tobias Grosser
2013-Dec-19 09:08 UTC
[LLVMdev] How to XFAIL test cases with buildbot LNTFactory
Hi,
I am currently trying to set up new performance and regression testers
for Polly and LLVM and would like to XFAIL two test cases. I am using
the LNTBuilder instead of the NightlyTestBuilder out of the assumption
that the LNTBuilder is the more modern solution.
However, when trying to xfail test cases I realized the xfail=[]
parameter of getLNTFactor is ignored. Previously this was not an issue,
as LNT would automatically XFAIL test cases that failed in the previous
run, but it seems Daniel removed this 'feature' a year ago:
commit 11987a4615f2a751c30ad6de29e1c277446a940c
Author: ddunbar <ddunbar at 91177308-0d34-0410-b5e6-96231b3b80d8>
Date: Fri Oct 26 18:52:03 2012 +0000
lnt runtest: Don't use XFAIL just because a test was previously
failing.
Does anyone know what is the canonical way to mark xfailures when
running nightly tests on the buildbots?
Should I switch to the NightlyTestBuilder? Or is there a way to convince
the LNTBuilder to accept XFAILures e.g. by allowing
the LitTestCommand to take an optional xfails argument according
to which test results will be relabeled for the buildbot?
Cheers,
Tobias
Daniel Dunbar
2013-Dec-19 18:10 UTC
[LLVMdev] How to XFAIL test cases with buildbot LNTFactory
There isn't a good way to mark XFAILs in the nightly test suite. The current way is to do it manually in the Makefiles (detecting the situation that causes the test to XFAIL). See SingleSource/Regression/C/Makefile and search for EXEC_XFAILS for an example. - Daniel On Thu, Dec 19, 2013 at 1:08 AM, Tobias Grosser <tobias at grosser.es> wrote:> Hi, > > I am currently trying to set up new performance and regression testers for > Polly and LLVM and would like to XFAIL two test cases. I am using the > LNTBuilder instead of the NightlyTestBuilder out of the assumption > that the LNTBuilder is the more modern solution. > > However, when trying to xfail test cases I realized the xfail=[] parameter > of getLNTFactor is ignored. Previously this was not an issue, > as LNT would automatically XFAIL test cases that failed in the previous > run, but it seems Daniel removed this 'feature' a year ago: > > > commit 11987a4615f2a751c30ad6de29e1c277446a940c > Author: ddunbar <ddunbar at 91177308-0d34-0410-b5e6-96231b3b80d8> > Date: Fri Oct 26 18:52:03 2012 +0000 > > lnt runtest: Don't use XFAIL just because a test was previously > failing. > > Does anyone know what is the canonical way to mark xfailures when running > nightly tests on the buildbots? > > Should I switch to the NightlyTestBuilder? Or is there a way to convince > the LNTBuilder to accept XFAILures e.g. by allowing > the LitTestCommand to take an optional xfails argument according > to which test results will be relabeled for the buildbot? > > Cheers, > Tobias > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131219/ac3e3b7d/attachment.html>
Pawel Worach
2014-Aug-04 15:07 UTC
[LLVMdev] How to XFAIL test cases with buildbot LNTFactory
Hi, Would it be possible to add the xfail logic to the compile step too ? Is there a better way to do this now ? Index: src-test-suite/TEST.simple.Makefile ==================================================================--- src-test-suite/TEST.simple.Makefile (revision 214709) +++ src-test-suite/TEST.simple.Makefile (working copy) @@ -21,7 +21,16 @@ $(PROGRAMS_TO_TEST:%=Output/%.simple.compile.report.txt): \ Output/%.simple.compile.report.txt: Output/%.out-simple @echo > $@ - @-if test -f Output/$*.simple; then \ + @-is_xfail=0; \ + for i in $(COMP_XFAILS); do \ + if test "$*" = $$i; then \ + is_xfail=1; \ + fi; \ + done; \ + if test $$is_xfail -eq 1; then \ + echo "TEST-XFAIL: compile $(RELDIR)/$*" >> $@;\ + echo "TEST-RESULT-compile-success: xfail" >> $@;\ + elif test -f Output/$*.simple; then \ echo "TEST-PASS: compile $(RELDIR)/$*" >> $@; \ echo "TEST-RESULT-compile-success: pass" >> $@;\ else \ -- Pawel> There isn't a good way to mark XFAILs in the nightly test suite. The > current way is to do it manually in the Makefiles (detecting the situation > that causes the test to XFAIL). See SingleSource/Regression/C/Makefile and > search for EXEC_XFAILS for an example. > > - Daniel > > > On Thu, Dec 19, 2013 at 1:08 AM, Tobias Grosser <tobias at grosser.es> wrote: > >> Hi, >> >> I am currently trying to set up new performance and regression testers for >> Polly and LLVM and would like to XFAIL two test cases. I am using the >> LNTBuilder instead of the NightlyTestBuilder out of the assumption >> that the LNTBuilder is the more modern solution. >> >> However, when trying to xfail test cases I realized the xfail=[] parameter >> of getLNTFactor is ignored. Previously this was not an issue, >> as LNT would automatically XFAIL test cases that failed in the previous >> run, but it seems Daniel removed this 'feature' a year ago: >> >> >> commit 11987a4615f2a751c30ad6de29e1c277446a940c >> Author: ddunbar <ddunbar at 91177308-0d34-0410-b5e6-96231b3b80d8> >> Date: Fri Oct 26 18:52:03 2012 +0000 >> >> lnt runtest: Don't use XFAIL just because a test was previously >> failing. >> >> Does anyone know what is the canonical way to mark xfailures when running >> nightly tests on the buildbots? >> >> Should I switch to the NightlyTestBuilder? Or is there a way to convince >> the LNTBuilder to accept XFAILures e.g. by allowing >> the LitTestCommand to take an optional xfails argument according >> to which test results will be relabeled for the buildbot? >> >> Cheers, >> Tobias >> >>