llvm dev - Nov 2007 - [LLVMdev] Problem with regression tests using stderr

I was attempting to write a test that involves grepping though the
stderr produced by opt -analyze, but found that my test was passing
even before I fixed the bug I was writing the test for!

I found that this one-line sure-fail test:

; RUN: echo hi |& false

passes. I also tried 2>&1, because I've seen some tests do that,
though that doesn't appear to work in this context either. It didn't
seem to create a file nameed &1 though, as the LLVM test-writing
documentation says.

After some investigation, I believe the problem is due to the regexp
on this line in test/lib/llvm.exp:

      } elseif {[regexp {RUN: *([^&]+)(&&)?} $line match oneline
suffix]} {

I don't understand what this is trying to do. What is intended by
the special handling of && here?

I found only one test that uses && on a RUN line,
test/CodeGen/X86/2007-07-03-GR64ToVR64.ll, and it's not clear to
me what it's doing there.

Dan

-- 
Dan Gohman, Cray Inc.

On Nov 20, 2007, at 8:22 AM, Dan Gohman wrote:
> I was attempting to write a test that involves grepping though the
> stderr produced by opt -analyze, but found that my test was passing
> even before I fixed the bug I was writing the test for!
>
> I found that this one-line sure-fail test:
>
> ; RUN: echo hi |& false
>
> passes. I also tried 2>&1, because I've seen some tests do that,
> though that doesn't appear to work in this context either. It
didn't
> seem to create a file nameed &1 though, as the LLVM test-writing
> documentation says.
>
> After some investigation, I believe the problem is due to the regexp
> on this line in test/lib/llvm.exp:
>
>       } elseif {[regexp {RUN: *([^&]+)(&&)?} $line match
oneline
> suffix]} {
>
> I don't understand what this is trying to do. What is intended by
> the special handling of && here?
>
> I found only one test that uses && on a RUN line,
> test/CodeGen/X86/2007-07-03-GR64ToVR64.ll, and it's not clear to
> me what it's doing there.
>
I think that it's a hold-over to how things used to be done. IIRC,  
you had to have the && at the end of the RUN line if you had another  
RUN line that needed to be executed. That's no longer the case, of  
course.

If you want to catch stderr, there is one test doing that:

Analysis/ScalarEvolution/2007-07-15-NegativeStride.ll:

; RUN: llvm-as < %s | opt -analyze -scalar-evolution 2>&1 | grep  
"Loop bb:  100 iterations"

-bw

On Tue, Nov 20, 2007 at 07:01:25PM -0800, Bill Wendling
wrote:
> I think that it's a hold-over to how things used to be done. IIRC,  
> you had to have the && at the end of the RUN line if you had
another
> RUN line that needed to be executed. That's no longer the case, of  
> course.
Thanks. I'll remove the && from the one test that still has it then.

Then we can look at removing the associated code from llvm.exp and fixing
the bug. This will take a while though, at it exposes a large number of
failures that are currently hidden.
> If you want to catch stderr, there is one test doing that:
> 
> Analysis/ScalarEvolution/2007-07-15-NegativeStride.ll:
> 
> ; RUN: llvm-as < %s | opt -analyze -scalar-evolution 2>&1 | grep
> "Loop bb:  100 iterations"
2>&1 doesn't evoke the desired behavior; the RUN line is
unfortunately
interpreted by TCL rather than a regular shell. The llvm.exp bug has
allowed this and a variety of other common bourne-shell-isms to slip by
unnoticed.

Dan

-- 
Dan Gohman, Cray Inc.