Even if it were under llvm, that doesn't mean it would have to be part of ninja check-all. Currently we have lldb/test and lldb/unittests, what about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest. Although now that I think about it, we would want it to be able to invoke clang-cl directly, so I guess that does necessitate it *not* being under llvm. On Thu, Sep 7, 2017 at 11:26 AM David Blaikie <dblaikie at gmail.com> wrote:> It's used, but not a huge repository of things, as you can see. I think > I've run it once or twice, but a long time ago. It was introduced for/by > Apple/LLDB stuff, so it's not something I've paid much attention to. > > It's probably not suitable as part of llvm tests directly. Those tests are > designed to be shorter/narrower/more focussed than full integration tests > (we don't execute any compiled programs under test there, for example). > > Porting to lit seems probably fine/good. > > On Thu, Sep 7, 2017 at 11:23 AM Zachary Turner <zturner at google.com> wrote: > >> What is the status of debuginfo-tests? Is it actively supported? How do >> you run it? It doesn't appear to be based on lit, any particular reason? >> Why is it its own repo instead of being part of llvm repo? >> >> I'd like improve this to support CodeView and PDB, such that it would >> only run on Windows and only if a suitable debugger was found (probably >> WinDbg). WinDbg supports a JavaScript-based scripting model, similar to >> how LLDB supports a Python based model, so my thoughts were to have a >> lit-based runner that scans for .js files that contain a test script >> alongside some source, then build the program, run it in WinDbg with some >> script that does various things, and exits the debugger, moving on to the >> next test. >> >> Anything I should be aware of / careful of when messing around in here? >> And any reason it can't be moved to llvm/tests and ported to lit? >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170907/2e3b5413/attachment.html>
On Thu, Sep 7, 2017 at 11:34 AM Zachary Turner <zturner at google.com> wrote:> Even if it were under llvm, that doesn't mean it would have to be part of > ninja check-all. Currently we have lldb/test and lldb/unittests, what > about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest. >Having them in lldb seems like an interesting question - not sure what the value is in having them in a separate repo compared to having them in the LLDB project. (I mean, long term, I think LLDB testing should probably have fewer of these end-to-end tests (ideally few tests that actually run a compiler/execute a resulting program (avoiding that so the tests are more portable, reliable, etc)) - but it's reasonable to have some & probably OK that they live in the LLDB repo (for LLVM these sort of tests live separately in the test-suite repo) Maybe because it makes the tests more likely to be run, more reliable (so they don't fail due to LLDB changes - makes it easier for Clang/llvm developers to act on regressions, etc)> > Although now that I think about it, we would want it to be able to invoke > clang-cl directly, so I guess that does necessitate it *not* being under > llvm. > > On Thu, Sep 7, 2017 at 11:26 AM David Blaikie <dblaikie at gmail.com> wrote: > >> It's used, but not a huge repository of things, as you can see. I think >> I've run it once or twice, but a long time ago. It was introduced for/by >> Apple/LLDB stuff, so it's not something I've paid much attention to. >> >> It's probably not suitable as part of llvm tests directly. Those tests >> are designed to be shorter/narrower/more focussed than full integration >> tests (we don't execute any compiled programs under test there, for >> example). >> >> Porting to lit seems probably fine/good. >> >> On Thu, Sep 7, 2017 at 11:23 AM Zachary Turner <zturner at google.com> >> wrote: >> >>> What is the status of debuginfo-tests? Is it actively supported? How >>> do you run it? It doesn't appear to be based on lit, any particular >>> reason? Why is it its own repo instead of being part of llvm repo? >>> >>> I'd like improve this to support CodeView and PDB, such that it would >>> only run on Windows and only if a suitable debugger was found (probably >>> WinDbg). WinDbg supports a JavaScript-based scripting model, similar to >>> how LLDB supports a Python based model, so my thoughts were to have a >>> lit-based runner that scans for .js files that contain a test script >>> alongside some source, then build the program, run it in WinDbg with some >>> script that does various things, and exits the debugger, moving on to the >>> next test. >>> >>> Anything I should be aware of / careful of when messing around in here? >>> And any reason it can't be moved to llvm/tests and ported to lit? >>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170907/581e8f04/attachment.html>
On Thu, Sep 7, 2017 at 11:40 AM David Blaikie <dblaikie at gmail.com> wrote:> On Thu, Sep 7, 2017 at 11:34 AM Zachary Turner <zturner at google.com> wrote: > >> Even if it were under llvm, that doesn't mean it would have to be part of >> ninja check-all. Currently we have lldb/test and lldb/unittests, what >> about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest. >> > > Having them in lldb seems like an interesting question - not sure what the > value is in having them in a separate repo compared to having them in the > LLDB project. (I mean, long term, I think LLDB testing should probably have > fewer of these end-to-end tests (ideally few tests that actually run a > compiler/execute a resulting program (avoiding that so the tests are more > portable, reliable, etc)) - but it's reasonable to have some & probably OK > that they live in the LLDB repo (for LLVM these sort of tests live > separately in the test-suite repo) > > Maybe because it makes the tests more likely to be run, more reliable (so > they don't fail due to LLDB changes - makes it easier for Clang/llvm > developers to act on regressions, etc) >Sorry, I had a bad typo. I meant we have llvm/test, llvm/unittests, so what about llvm/dbgtests. I definitely woudln't want them in LLDB, as my entire motivation is to write tests against a non-LLDB debugger. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170907/f3b35d14/attachment-0001.html>
> On Sep 7, 2017, at 11:34 AM, Zachary Turner <zturner at google.com> wrote: > > Even if it were under llvm, that doesn't mean it would have to be part of ninja check-all. Currently we have lldb/test and lldb/unittests, what about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest.They are not LLDB tests. They end-to-end test that the debug info produced by clang works with the *host* debugger, which can be gdb or lldb. They are part of the CFE tests for this reason. -- adrian> > Although now that I think about it, we would want it to be able to invoke clang-cl directly, so I guess that does necessitate it *not* being under llvm. > > On Thu, Sep 7, 2017 at 11:26 AM David Blaikie <dblaikie at gmail.com <mailto:dblaikie at gmail.com>> wrote: > It's used, but not a huge repository of things, as you can see. I think I've run it once or twice, but a long time ago. It was introduced for/by Apple/LLDB stuff, so it's not something I've paid much attention to. > > It's probably not suitable as part of llvm tests directly. Those tests are designed to be shorter/narrower/more focussed than full integration tests (we don't execute any compiled programs under test there, for example). > > Porting to lit seems probably fine/good. > > On Thu, Sep 7, 2017 at 11:23 AM Zachary Turner <zturner at google.com <mailto:zturner at google.com>> wrote: > What is the status of debuginfo-tests? Is it actively supported? How do you run it? It doesn't appear to be based on lit, any particular reason? Why is it its own repo instead of being part of llvm repo? > > I'd like improve this to support CodeView and PDB, such that it would only run on Windows and only if a suitable debugger was found (probably WinDbg). WinDbg supports a JavaScript-based scripting model, similar to how LLDB supports a Python based model, so my thoughts were to have a lit-based runner that scans for .js files that contain a test script alongside some source, then build the program, run it in WinDbg with some script that does various things, and exits the debugger, moving on to the next test. > > Anything I should be aware of / careful of when messing around in here? And any reason it can't be moved to llvm/tests and ported to lit?-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170907/eefdf349/attachment.html>