Renato Golin wrote:> On 18 October 2012 02:53, Robinson, Paul <Paul.Robinson at am.sony.com> wrote: > > I had a "quality suite" at a previous job; it was the result of many PY > > of effort. It was also debugger-based, which is a mixed blessing; you > > get a lot of DWARF-parsing code for free, but then you get a lot of > > debugger bugs for free too! And you don't get to test the DWARF > > directly, you get to test how the debugger uses the DWARF. Not really > > optimal, but still--a whole lot better than nothing. > > The trade off also goes in the other direction. If you had a strict > Dwarf parser green, that would mean next to nothing as to what that > Dwarf would represent in the debugger(s).Well, having IR-level testing tells you next to nothing as to what your program would actually do when you compile and run it. But it seems to me that we have a huge pile of IR-level tests, so _somebody_ must think they are useful. :-) Sure, the acid test is whether the debugger does the right thing. I'm not saying debugger-based tests are worthless, I'm saying that _just_ having debugger-based tests is not _optimal_. DWARF-level testing would let you do things that debugger-based tests would find anywhere from awkward to impossible. That said, what's easiest is probably to get some form of GDB bot up and running, and the benefit is likely to be worth the pain.> AFAIK, most Dwarf compatible debuggers are also GDB compatible, which > means that even the idiotic things that GDB does will probably be > understood by other debuggers.I don't think what GDB _does_ will be understood by other debuggers. :-) And the debugger for some of the platforms I have to deal with is certainly not GDB, so I care more about generating valid DWARF than I do about getting GDB running smoothly. I do recognize that the majority of LLVM targets care more about GDB, though. Cheers, --paulr
On Thu, Oct 18, 2012 at 1:36 PM, Robinson, Paul <Paul.Robinson at am.sony.com> wrote:> Renato Golin wrote: >> On 18 October 2012 02:53, Robinson, Paul <Paul.Robinson at am.sony.com> wrote: >> > I had a "quality suite" at a previous job; it was the result of many PY >> > of effort. It was also debugger-based, which is a mixed blessing; you >> > get a lot of DWARF-parsing code for free, but then you get a lot of >> > debugger bugs for free too! And you don't get to test the DWARF >> > directly, you get to test how the debugger uses the DWARF. Not really >> > optimal, but still--a whole lot better than nothing. >> >> The trade off also goes in the other direction. If you had a strict >> Dwarf parser green, that would mean next to nothing as to what that >> Dwarf would represent in the debugger(s). > > Well, having IR-level testing tells you next to nothing as to what your > program would actually do when you compile and run it. But it seems > to me that we have a huge pile of IR-level tests, so _somebody_ must > think they are useful. :-)It's not so much an argument from correctness as convenience. IR level regression tests test features in relative isolation and with relative speed compared to full scenario tests. That means faster dev iteration with higher confidence (you can run all the regression tests quickly - so you can run them on every commit without slowing down too much, as well as investigate failures quickly because they point to specific issues directly rather than "something is broken in this big sequence of steps") They have the drawback that they can be overconstrained (the most obvious examples of this include ordering (there's been some discussion recently about FileCheck allowing unordered sequences of lines) but possibly other situations where emitting different information (debug info or machine code, etc) might not have a noticable effect on the output (maybe the debugger doesn't care about that particular thing being A or B or first or last, etc))> > Sure, the acid test is whether the debugger does the right thing. I'm > not saying debugger-based tests are worthless, I'm saying that _just_ > having debugger-based tests is not _optimal_. DWARF-level testing > would let you do things that debugger-based tests would find anywhere > from awkward to impossible. > > That said, what's easiest is probably to get some form of GDB bot up > and running, and the benefit is likely to be worth the pain.This is certainly true - even if we had a great debug info regression suite we'd still want this. Given that we don't have a great debug info regression suite we /really/ want this. But we also want the infrastructure to write good debug info regression tests - sooner the better, so we can have a slightly less bad debug info regression suite as we go along :)> >> AFAIK, most Dwarf compatible debuggers are also GDB compatible, which >> means that even the idiotic things that GDB does will probably be >> understood by other debuggers. > > I don't think what GDB _does_ will be understood by other debuggers. :-) > And the debugger for some of the platforms I have to deal with is > certainly not GDB, so I care more about generating valid DWARF than > I do about getting GDB running smoothly. I do recognize that the > majority of LLVM targets care more about GDB, though. > > Cheers, > --paulr > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On 18 October 2012 21:36, Robinson, Paul <Paul.Robinson at am.sony.com> wrote:> Well, having IR-level testing tells you next to nothing as to what your > program would actually do when you compile and run it. But it seems > to me that we have a huge pile of IR-level tests, so _somebody_ must > think they are useful. :-)When creating Dwarf tests I did it at all levels: IR checking for metadata, ELF checking for Dwarf and GDB execution checking for correct behaviour. All that as LIT driven, so a "make check" would give me the results in a few seconds, with the benefit of a good dev iteration, as David mentioned. Unfortunately, none of that was in the open, so I can't share... :(> That said, what's easiest is probably to get some form of GDB bot up > and running, and the benefit is likely to be worth the pain.Since GDB already has a good and standard test infrastructure, it'd likely get a good chunk of bad Dwarf our of the way before you start worrying about Lauterbach's specifics. Do you have a good (maybe open) Dwarf validation suite available? There is no such thing as too many tests... ;) -- cheers, --renato http://systemcall.org/
> When creating Dwarf tests I did it at all levels: IR checking for > metadata, ELF checking for Dwarf and GDB execution checking for > correct behaviour. >Agreed. This is how I've been doing it for a while now.> All that as LIT driven, so a "make check" would give me the results in > a few seconds, with the benefit of a good dev iteration, as David > mentioned. >Quite.> Since GDB already has a good and standard test infrastructure, it'd > likely get a good chunk of bad Dwarf our of the way before you start > worrying about Lauterbach's specifics.The gdb testsuite is pretty good as a "what's expected" set of tests, however, one thing to keep in mind is that a lot of the checks aren't particularly fuzzy. I.e. it checks what's expected but it's not necessarily valid dwarf that it's looking for but a particular behavior. -eric