> > One thing we could do to remove fragility in the test is to remove the > > running of `short.py` in the test. This is only invoked to check that > > it's possible for a command to run to completion in the presence of a > > fixed timeout. If we can live without testing that part (i.e. we only > > test that a timeout can be reached) then the test should be much more > > robust. > > If you're on board with that, it's a tradeoff I think is probably > reasonable from a test coverage V reliability V development time > tradeoff.Sorry for the delay here. I've put a patch up for review that goes with this approach: https://reviews.llvm.org/D88807
Looks like there might still be some issues with the timeout tests? http://lab.llvm.org:8011/#/builders/126/builds/226/steps/13/logs/FAIL__lit___shtest-timeout_py On Sun, Oct 4, 2020 at 2:44 PM Dan Liew <dan at su-root.co.uk> wrote:> > > One thing we could do to remove fragility in the test is to remove the > > > running of `short.py` in the test. This is only invoked to check that > > > it's possible for a command to run to completion in the presence of a > > > fixed timeout. If we can live without testing that part (i.e. we only > > > test that a timeout can be reached) then the test should be much more > > > robust. > > > > If you're on board with that, it's a tradeoff I think is probably > > reasonable from a test coverage V reliability V development time > > tradeoff. > > Sorry for the delay here. I've put a patch up for review that goes > with this approach: https://reviews.llvm.org/D88807 >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201022/a2afd6a5/attachment.html>
Another case: http://lab.llvm.org:8011/#/builders/43/builds/810 shtest-timeout.py seems to be fairly flaky on the clang-cmake-aarch64-quick bot: http://lab.llvm.org:8011/#/builders/43, I get notifications from it fairly often On Thu, Oct 22, 2020 at 7:15 PM David Blaikie via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Looks like there might still be some issues with the timeout tests? > http://lab.llvm.org:8011/#/builders/126/builds/226/steps/13/logs/FAIL__lit___shtest-timeout_py > > On Sun, Oct 4, 2020 at 2:44 PM Dan Liew <dan at su-root.co.uk> wrote: > >> > > One thing we could do to remove fragility in the test is to remove the >> > > running of `short.py` in the test. This is only invoked to check that >> > > it's possible for a command to run to completion in the presence of a >> > > fixed timeout. If we can live without testing that part (i.e. we only >> > > test that a timeout can be reached) then the test should be much more >> > > robust. >> > >> > If you're on board with that, it's a tradeoff I think is probably >> > reasonable from a test coverage V reliability V development time >> > tradeoff. >> >> Sorry for the delay here. I've put a patch up for review that goes >> with this approach: https://reviews.llvm.org/D88807 >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201109/a015aad4/attachment.html>