Stella Stamenova via llvm-dev
2022-Jan-11 01:47 UTC
[llvm-dev] [EXTERNAL] Re: Responsibilities of a buildbot owner
1) Watchpoint Support: I disabled a couple of the watchpoint tests that are occasionally failing this morning. I think there may be one or two more that fail as well and we could disable those also. I am not sure whether the issue here is with watchpoint support or lldb-server. I actually think the issue is with lldb-server, but I haven't worked on lldb in years (besides the buildbot), so I haven't investigated in more details. I think some of these tests became flaky recently (possibly since the upgrade to VS2019?) 2) Random mysterious failure: I've noticed a class of failures in llvm, lld, clang, and lldb (mostly lldb and lld) that have to do with running multiple threads on Windows. I think the underlying issue is that code in the product as well as the tests doesn't account for the way Windows behaves with regards to new threads and the order of events ends up being non-deterministic. The lld failure in particular was incredibly frustrating because it would only occur occasionally, never on a buildbot as far as I could tell, and the comments in the code seem to indicate that it should work (but it doesn't): llvm-project/Parallel.cpp at e356027016c6365b3d8924f54c33e2c63d931492 * llvm/llvm-project (github.com)<https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d931492/llvm/lib/Support/Parallel.cpp>. Ideally, someone familiar with Windows threading would address the issue across the board. 3) lldb-server for Windows test failures: tools/lldb-server/tests/./LLDBServerTests.exe/StandardStartupTest.TestStopReplyContainsThreadPcs This particular failure is definitely more recent (in the last couple of months) and I would hate for us to disable this test instead of having someone who works on lldb-server investigate. Thanks, -Stella From: Jim Ingham <jingham at apple.com> Sent: Monday, January 10, 2022 5:26 PM To: Philip Reames <listmail at philipreames.com> Cc: Stella Stamenova <stilis at microsoft.com>; llvm-dev <llvm-dev at lists.llvm.org>; clayborg at gmail.com; jmolenda at apple.com; zturner at google.com Subject: [EXTERNAL] Re: [llvm-dev] Responsibilities of a buildbot owner This situation is somewhat complicated by the fact that Zachary - the only listed code owner for Windows support - hasn't worked on lldb for quite a while now. Various people have been helping with the Windows port, but I'm not sure that there's someone taking overall responsibility for the Windows port. Greg may have access to a Windows system, but neither Jason nor I work on Windows at all. In fact, I don't think anybody listed in the Code Owner's file for lldb does much work on Windows. For the health of that port, we probably do need someone to organize the effort and help sort out this sort of thing. Anyway, looking at the current set of bot failures for this Windows bot, I saw three basic classes of failures (besides the build breaks). 1) Watchpoint Support: TestWatchLocation.py wasn't the only or even the most common Watchpoint failure in these test runs: For instance in: https://lab.llvm.org/buildbot/#/builders/83/builds/13600<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13600&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884787489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=en7t1pHwgIzShGF0G4azkpwW06xfxCmJpxhWiOFjiZg%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13543<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13543&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884787489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=d64KQWUAUBt1dyWV4E1mhdJfraNRUPSqAKbSaJvWxUM%3D&reserved=0> The failing test is TestWatchpointMultipleThreads.py. On: https://lab.llvm.org/buildbot/#/builders/83/builds/13579<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13579&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6QuRoDFgy89SSDxT%2FtHIshqSdZEFkCJQ1btOgLXZ2U4%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13576<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13576&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gcDA1Z75UuhoeqMoCTcC%2B2OJIsRGkBfO6icOruuYyiM%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13565<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13565&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=c3ielcfvwh5hNJBjTQlOJqAqLQ7vDgm58B5STsabncE%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13538<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13538&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cz82bluf1Gn3ED4s8oBX1Wq3oIixTqqtKpfhLqNPjX8%3D&reserved=0> it's TestSetWatchlocation.py On: https://lab.llvm.org/buildbot/#/builders/83/builds/13550<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13550&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DoU09empZFVXcSdEqWLTAKJeqavyisnM3%2ByyRsQpfAg%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13508<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13508&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TJfw9KplEcyRe7CKHZAp0Zz8mgSup%2Fg0pQ8LELXjbQ4%3D&reserved=0> It's TestWatchLocationWithWatchSet.py On: https://lab.llvm.org/buildbot/#/builders/83/builds/13528<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13528&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yKD1Z1DjFPby7cf8DKI5YE8fK1e%2F7bSFOmbFwrXx0UA%3D&reserved=0> It's TestTargetWatchAddress.py These are all in one way or another failing because we set a watchpoint, and expected to hit it, and did not. In the failing tests, we do verify that we got a valid watchpoint back. We just "continue" expecting to hit it and don't. The tests don't seem to be doing anything suspicious that would cause inconsistent behavior, and they aren't failing on other systems. It sounds more like the way lldb-server for Windows implements watchpoint setting is flakey in some way. So these really are "tests correctly showing flakey behavior in the underlying code". We could just skip all these watchpoint tests, but we already have 268-some odd tests that are marked as skipIfWindows, most with annotations that some behavior or other is flakey on Windows. It is not great for the platform support to just keep adding to that count, but if nobody is available to dig into the Windows watchpoint code, we probably need to declare Watchpoint support "in a beta state" and turn off all the tests for it. But that seems like a decision that should be made by someone with more direct responsibility for the Windows port. Does our bot strategy cover how to deal with incomplete platform support on some particular platform? Is the only choice really just turning off all the tests that are uncovering flaws in the underlying implementation? 2) Random mysterious failure: I also saw one failure here: https://lab.llvm.org/buildbot/#/builders/83/builds/13513<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13513&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=b8c0i1EgqGLm61xrB33VRpO4EasdIO4PcnKbDGRNRhQ%3D&reserved=0> functionalities/load_after_attach/TestLoadAfterAttach.py In that one, lldb sets a breakpoint, confirms that the breakpoint got a valid location, then continues and runs to completion w/o hitting the breakpoint. Again, that test is quite straightforward, and it looks like the underlying implementation, not the test, is what is at fault. 3) lldb-server for Windows test failures: In these runs: https://lab.llvm.org/buildbot/#/builders/83/builds/13594<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13594&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=koR%2BxDxC6iNFhWGNisjIKp63kyZNkHih4IM5GHnyWaw%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13580<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13580&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8YlK7yOUVSNO9y0FJvu6becIIoLn5oElVER3eE8u3H4%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13550<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13550&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DoU09empZFVXcSdEqWLTAKJeqavyisnM3%2ByyRsQpfAg%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13535<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13535&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YqDDr9M%2F%2FjB6tN8iE6zO9264UyWlq497aMYWGtOUqvc%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13526<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13526&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FBsta7PI%2FSygHD4RGcZQN6VsIEwr0usEhjO%2BwuwFG5s%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13525<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13525&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PpX0hb1SN5xAw4zzmODk91NIR%2F3gSk7fXQaLiQ0wWbE%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13511<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13511&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Q%2FFySuLlWeKbprM8pz7G7BWc0D6AEUqQnPden3dIx50%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13498<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13498&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=a6pTlD31U2gQ6K0XDOqdY7Hyb4cr40M93XDBpO42fHg%3D&reserved=0> The failure was in the Windows' lldb-server implementation here: tools/lldb-server/tests/./LLDBServerTests.exe/StandardStartupTest.TestStopReplyContainsThreadPcs And there were a couple more lldb-server test fails: https://lab.llvm.org/buildbot/#/builders/83/builds/13527<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13527&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oBGs6jpECuwfge9zqvuVFxQoYzVkzxov8bNoJQHbqq0%3D&reserved=0> https://lab.llvm.org/buildbot/#/builders/83/builds/13524<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13524&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZcjSxB%2FI2S5D6RZfrhVhuPd9oIRgC%2Fb1FdAdpbwGhY0%3D&reserved=0> Where the failure is: tools/lldb-server/TestGdbRemoteExpeditedRegisters.py MacOS doesn't use lldb-server, so I am not particularly familiar with it, and didn't look into these failures further. Jim On Jan 10, 2022, at 3:33 PM, Philip Reames <listmail at philipreames.com<mailto:listmail at philipreames.com>> wrote: +CC lldb code owners This bot appears to have been restored to the primary buildmaster, but is failing something like 1 in 5 builds due to lldb tests which are flaky. https://lab.llvm.org/buildbot/#/builders/83<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sMLjLgtzcxeqBDUwdHPbVA%2F7RN9VE0ACvg%2FHetKXjhU%3D&reserved=0> Specifically, this test is the one failing: commands/watchpoints/hello_watchlocation/TestWatchLocation.py Can someone with LLDB context please either a) address the cause of the flakiness or b) disable the test? Philip p.s. Please restrict this sub-thread to the topic of stabilizing this bot. Policy questions can be addressed in the other sub-threads to keep this vaguely understandable. On 1/8/22 1:01 PM, Philip Reames via llvm-dev wrote: In this particular example, we appear to have a bunch of flaky lldb tests. I personally know absolutely nothing about lldb. I have no idea whether the tests are badly designed, the system they're being run on isn't yet supported by lldb, or if there's some recent code bug introduced which causes the failure. "Someone" needs to take the responsibility of figuring that out, and in the meantime spaming developers with inactionable failure notices seems undesirable. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20220111/f3765feb/attachment.html>
Pavel Labath via llvm-dev
2022-Jan-11 11:32 UTC
[llvm-dev] [EXTERNAL] Re: Responsibilities of a buildbot owner
I am afraid I too have to say that I believe the real problem here is the lack active developers with interest in/commitment to the windows port of lldb. While I appreciate having Stella's windows buildbot around, and it prevents windows from bitrotting completely, it would take a much more active involvement to resolve the multitude of systemic issues affecting windows support. Like, if we tried to apply the current llvm support policy guidelines to the windows (host-side, at least) support code, I don't think it would even meet the criteria for inclusion in the peripheral tier (active sub-community). Now for something slightly more constructive: While I am not familiar with the windows-specific parts of the watchpoint code, I think I can say without exaggerating that I have a *lot* of experience in fixing flaky tests. That experience tells me that flaky watchpoint tests are often/usually caused by factors outside lldb. (due to watchpoints being a global, scarce, hardware resource). Virtualization is particularly tricky here -- every virtualization technology that I've tried has had (at some point in time at least) a watchpoint-related bug. The problem described here sounds a lot like the issue I observed on Google Compute Engine, which could also miss some watchpoints "randomly". So, if this bot is running in any kind of a virtualized environment, the first thing I'd do is check whether the issue happens on physical hardware. Relatedly to that, I also want to mention that we also have the ability to skip categories of tests in lldb. All the watchpoint tests are (should be) annotated by the watchpoint category, and so you can easily skip all of them, either by hard-disabling the category for windows in the source code (if this is an lldb issue) or externally through the buildbot config (if this is due to the bot environment => LLDB_TEST_USER_ARGS="--skip-category watchpoint"). hope that helps, pl On 11/01/2022 02:47, Stella Stamenova via llvm-dev wrote:> 1) Watchpoint Support: > > I disabled a couple of the watchpoint tests that are occasionally > failing this morning. I think there may be one or two more that fail as > well and we could disable those also. I am not sure whether the issue > here is with watchpoint support or lldb-server. I actually think the > issue is with lldb-server, but I haven’t worked on lldb in years > (besides the buildbot), so I haven’t investigated in more details. I > think some of these tests became flaky recently (possibly since the > upgrade to VS2019?) > > 2) Random mysterious failure: > > I’ve noticed a class of failures in llvm, lld, clang, and lldb (mostly > lldb and lld) that have to do with running multiple threads on Windows. > I think the underlying issue is that code in the product as well as the > tests doesn’t account for the way Windows behaves with regards to new > threads and the order of events ends up being non-deterministic. The lld > failure in particular was incredibly frustrating because it would only > occur occasionally, never on a buildbot as far as I could tell, and the > comments in the code seem to indicate that it should work (but it > doesn’t): llvm-project/Parallel.cpp at > e356027016c6365b3d8924f54c33e2c63d931492 · llvm/llvm-project > (github.com) > <https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d931492/llvm/lib/Support/Parallel.cpp>. > Ideally, someone familiar with Windows threading would address the issue > across the board. > > 3) lldb-server for Windows test failures: > > tools/lldb-server/tests/./LLDBServerTests.exe/StandardStartupTest.TestStopReplyContainsThreadPcs > > This particular failure is definitely more recent (in the last couple of > months) and I would hate for us to disable this test instead of having > someone who works on lldb-server investigate. > > Thanks, > > -Stella > > *From:* Jim Ingham <jingham at apple.com> > *Sent:* Monday, January 10, 2022 5:26 PM > *To:* Philip Reames <listmail at philipreames.com> > *Cc:* Stella Stamenova <stilis at microsoft.com>; llvm-dev > <llvm-dev at lists.llvm.org>; clayborg at gmail.com; jmolenda at apple.com; > zturner at google.com > *Subject:* [EXTERNAL] Re: [llvm-dev] Responsibilities of a buildbot owner > > This situation is somewhat complicated by the fact that Zachary - the > only listed code owner for Windows support - hasn’t worked on lldb for > quite a while now. Various people have been helping with the Windows > port, but I’m not sure that there’s someone taking overall > responsibility for the Windows port. > > Greg may have access to a Windows system, but neither Jason nor I work > on Windows at all. In fact, I don’t think anybody listed in the Code > Owner’s file for lldb does much work on Windows. For the health of that > port, we probably do need someone to organize the effort and help sort > out this sort of thing. > > Anyway, looking at the current set of bot failures for this Windows bot, > I saw three basic classes of failures (besides the build breaks). > > 1) Watchpoint Support: > > TestWatchLocation.py wasn’t the only or even the most common Watchpoint > failure in these test runs: > > For instance in: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13600 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13600&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884787489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=en7t1pHwgIzShGF0G4azkpwW06xfxCmJpxhWiOFjiZg%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13543 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13543&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884787489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=d64KQWUAUBt1dyWV4E1mhdJfraNRUPSqAKbSaJvWxUM%3D&reserved=0> > > The failing test is TestWatchpointMultipleThreads.py. > > On: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13579 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13579&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6QuRoDFgy89SSDxT%2FtHIshqSdZEFkCJQ1btOgLXZ2U4%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13576 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13576&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gcDA1Z75UuhoeqMoCTcC%2B2OJIsRGkBfO6icOruuYyiM%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13565 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13565&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=c3ielcfvwh5hNJBjTQlOJqAqLQ7vDgm58B5STsabncE%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13538 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13538&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cz82bluf1Gn3ED4s8oBX1Wq3oIixTqqtKpfhLqNPjX8%3D&reserved=0> > > it’s TestSetWatchlocation.py > > On: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13550 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13550&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DoU09empZFVXcSdEqWLTAKJeqavyisnM3%2ByyRsQpfAg%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13508 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13508&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TJfw9KplEcyRe7CKHZAp0Zz8mgSup%2Fg0pQ8LELXjbQ4%3D&reserved=0> > > It’s TestWatchLocationWithWatchSet.py > > On: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13528 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13528&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yKD1Z1DjFPby7cf8DKI5YE8fK1e%2F7bSFOmbFwrXx0UA%3D&reserved=0> > > It’s TestTargetWatchAddress.py > > These are all in one way or another failing because we set a watchpoint, > and expected to hit it, and did not. In the failing tests, we do verify > that we got a valid watchpoint back. We just “continue” expecting to > hit it and don't. The tests don’t seem to be doing anything suspicious > that would cause inconsistent behavior, and they aren’t failing on other > systems. It sounds more like the way lldb-server for Windows implements > watchpoint setting is flakey in some way. > > So these really are “tests correctly showing flakey behavior in the > underlying code”. We could just skip all these watchpoint tests, but we > already have 268-some odd tests that are marked as skipIfWindows, most > with annotations that some behavior or other is flakey on Windows. It > is not great for the platform support to just keep adding to that count, > but if nobody is available to dig into the Windows watchpoint code, we > probably need to declare Watchpoint support “in a beta state” and turn > off all the tests for it. But that seems like a decision that should be > made by someone with more direct responsibility for the Windows port. > > Does our bot strategy cover how to deal with incomplete platform support > on some particular platform? Is the only choice really just turning off > all the tests that are uncovering flaws in the underlying implementation? > > 2) Random mysterious failure: > > I also saw one failure here: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13513 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13513&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=b8c0i1EgqGLm61xrB33VRpO4EasdIO4PcnKbDGRNRhQ%3D&reserved=0> > > functionalities/load_after_attach/TestLoadAfterAttach.py > > In that one, lldb sets a breakpoint, confirms that the breakpoint got a > valid location, then continues and runs to completion w/o hitting the > breakpoint. Again, that test is quite straightforward, and it looks > like the underlying implementation, not the test, is what is at fault. > > 3) lldb-server for Windows test failures: > > In these runs: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13594 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13594&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=koR%2BxDxC6iNFhWGNisjIKp63kyZNkHih4IM5GHnyWaw%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13580 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13580&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8YlK7yOUVSNO9y0FJvu6becIIoLn5oElVER3eE8u3H4%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13550 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13550&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DoU09empZFVXcSdEqWLTAKJeqavyisnM3%2ByyRsQpfAg%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13535 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13535&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YqDDr9M%2F%2FjB6tN8iE6zO9264UyWlq497aMYWGtOUqvc%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13526 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13526&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FBsta7PI%2FSygHD4RGcZQN6VsIEwr0usEhjO%2BwuwFG5s%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13525 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13525&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PpX0hb1SN5xAw4zzmODk91NIR%2F3gSk7fXQaLiQ0wWbE%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13511 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13511&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Q%2FFySuLlWeKbprM8pz7G7BWc0D6AEUqQnPden3dIx50%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13498 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13498&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=a6pTlD31U2gQ6K0XDOqdY7Hyb4cr40M93XDBpO42fHg%3D&reserved=0> > > The failure was in the Windows’ lldb-server implementation here: > > tools/lldb-server/tests/./LLDBServerTests.exe/StandardStartupTest.TestStopReplyContainsThreadPcs > > And there were a couple more lldb-server test fails: > > https://lab.llvm.org/buildbot/#/builders/83/builds/13527 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13527&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oBGs6jpECuwfge9zqvuVFxQoYzVkzxov8bNoJQHbqq0%3D&reserved=0> > > https://lab.llvm.org/buildbot/#/builders/83/builds/13524 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13524&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZcjSxB%2FI2S5D6RZfrhVhuPd9oIRgC%2Fb1FdAdpbwGhY0%3D&reserved=0> > > Where the failure is: > > tools/lldb-server/TestGdbRemoteExpeditedRegisters.py > > MacOS doesn’t use lldb-server, so I am not particularly familiar with > it, and didn’t look into these failures further. > > Jim > > > > On Jan 10, 2022, at 3:33 PM, Philip Reames > <listmail at philipreames.com <mailto:listmail at philipreames.com>> wrote: > > +CC lldb code owners > > This bot appears to have been restored to the primary buildmaster, > but is failing something like 1 in 5 builds due to lldb tests which > are flaky. > > https://lab.llvm.org/buildbot/#/builders/83 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sMLjLgtzcxeqBDUwdHPbVA%2F7RN9VE0ACvg%2FHetKXjhU%3D&reserved=0> > > Specifically, this test is the one failing: > > commands/watchpoints/hello_watchlocation/TestWatchLocation.py > > Can someone with LLDB context please either a) address the cause of > the flakiness or b) disable the test? > > Philip > > p.s. Please restrict this sub-thread to the topic of stabilizing > this bot. Policy questions can be addressed in the other > sub-threads to keep this vaguely understandable. > > On 1/8/22 1:01 PM, Philip Reames via llvm-dev wrote: > > In this particular example, we appear to have a bunch of flaky > lldb tests. I personally know absolutely nothing about lldb. I > have no idea whether the tests are badly designed, the system > they're being run on isn't yet supported by lldb, or if there's > some recent code bug introduced which causes the failure. > "Someone" needs to take the responsibility of figuring that out, > and in the meantime spaming developers with inactionable failure > notices seems undesirable. > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev