Folks, Some of our CMake buildbots are failing to timeout, and I believe it's something to do with how the output comes from the LIT tests, even though we add -v to LIT_ARGS. When the "check-all" stage runs, the output stops at the "Running tests" message and only prints the rest of the output (including all tests that pass, fail, etc) at the end. I believe this has to do with ninja holding off the stream until the job actually finishes, but I'm not seeing this when I run ninja manually, which hints as to how the terminal is setup for buildbot. Does anyone have a good hint for me to fix this? cheers, --renato
On 12/18/14 3:55 AM, Renato Golin wrote:> Folks, > > Some of our CMake buildbots are failing to timeout, and I believe it's > something to do with how the output comes from the LIT tests, even > though we add -v to LIT_ARGS.I think this will help one facet of your problem: http://reviews.llvm.org/D6584 Jon -- Jon Roelofs jonathan at codesourcery.com CodeSourcery / Mentor Embedded
If I remember correctly, there was some talk in ninja's github issues about not buffering when there's only one command to run. Is it possible that your manual runs are using a newer version of ninja? Also, I remember there being some talk about printing 'still waiting...' messages when a command hasn't finished for a while. That would stop the timeouts but it would also remove the protection against infinite loops. ________________________________________ From: llvmdev-bounces at cs.uiuc.edu [llvmdev-bounces at cs.uiuc.edu] on behalf of Renato Golin [renato.golin at linaro.org] Sent: 18 December 2014 10:55 To: LLVM Dev Subject: [LLVMdev] LIT Verbose Folks, Some of our CMake buildbots are failing to timeout, and I believe it's something to do with how the output comes from the LIT tests, even though we add -v to LIT_ARGS. When the "check-all" stage runs, the output stops at the "Running tests" message and only prints the rest of the output (including all tests that pass, fail, etc) at the end. I believe this has to do with ninja holding off the stream until the job actually finishes, but I'm not seeing this when I run ninja manually, which hints as to how the terminal is setup for buildbot. Does anyone have a good hint for me to fix this? cheers, --renato _______________________________________________ LLVM Developers mailing list LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On 18 December 2014 at 21:32, Jonathan Roelofs <jonathan at codesourcery.com> wrote:> I think this will help one facet of your problem: > http://reviews.llvm.org/D6584I don't think so, because the tests don't time out, it's a buffering issue... --renato
On 18 December 2014 at 21:34, Daniel Sanders <Daniel.Sanders at imgtec.com> wrote:> If I remember correctly, there was some talk in ninja's github issues about not buffering when there's only one command to run. Is it possible that your manual runs are using a newer version of ninja?My laptop version is actually older... :) 1.3.0.git against 1.3.4 It's strange, as LIT actually prints the whole thing, so I can't fix it by adding more verbosity to lit, I have to see what ninja is doing wrong. I'll have a look on github, thanks!> Also, I remember there being some talk about printing 'still waiting...' messages when a command hasn't finished for a while. That would stop the timeouts but it would also remove the protection against infinite loops.Yeah, I'd rather fix the buffering issue, not create another one. :) cheers, --renato
+pcc Renato Golin <renato.golin at linaro.org> writes:> Folks, > > Some of our CMake buildbots are failing to timeout, and I believe it's > something to do with how the output comes from the LIT tests, even > though we add -v to LIT_ARGS. > > When the "check-all" stage runs, the output stops at the "Running > tests" message and only prints the rest of the output (including all > tests that pass, fail, etc) at the end.IIUC, the problem is that cmake+ninja doesn't actually output anything while tests are running. I think what you're hoping for is r222181, which teaches cmake that the test targets output arbitrary stuff on the terminal, and that ninja should show that output. Unfortunately, this needs a really new (unreleased?) version of cmake to work. If you don't have a new enough cmake, you might be better off running the lit command directly for now, though I realize this is a bit of a pain for check-all since the list of targets depends on what exactly you've checked out.
On 19 December 2014 at 06:34, Justin Bogner <mail at justinbogner.com> wrote:> IIUC, the problem is that cmake+ninja doesn't actually output anything > while tests are running. I think what you're hoping for is r222181, > which teaches cmake that the test targets output arbitrary stuff on the > terminal, and that ninja should show that output.Hum, I'm using CMake 2.8...> Unfortunately, this needs a really new (unreleased?) version of cmake to > work. If you don't have a new enough cmake, you might be better off > running the lit command directly for now, though I realize this is a bit > of a pain for check-all since the list of targets depends on what > exactly you've checked out.I don't mind updating the cmake by hand (compile+install) if that's what it takes. I'll try. thanks! --renato