llvm dev - Sep 2010 - [LLVMdev] LLVM make check-lit results *not reproducible* in certain cases WAS: [llvm-commits]: Initial cut of ARM MC ELF emitter (PATCH)

Hi everyone,

I am sending this email after some time consuming exploration into how
LLVM executes its tests.  I first noticed this issue when Rafael
noticed that my patch I sent in to llvm-commits broke some tests.

I was initially very skeptical, as I had run the make check test from
the build directory before sending in the patch, but as it turns out,
several issues came together in a perfect storm to give a royal
punkin' to yours truly.

After several time-consuming make clean/make check cycles among my
various llvm build directories, I began to doubt my sanity when across
certain recompile make clean/make check operation pairs on the SAME
SOURCE directory, different tests began to fail from the list by
Rafael, or sometimes, NO tests failed (and in one case, 100% of tests
failed!)

The answer turned out to be both simpler and more complex than I had
imagined. Whenever you have complex integration tests that call out to
various utilities, it is wise to make sure that at the very least, the
locations of said utilities are anchored to specific directories.

At least at first it seemed to be the case, according to the
autogenerated test/lit.site.config file in the  individual llvm build
directories, recreated here for your reading pleasure:

-----
## Autogenerated by LLVM/Clang configuration.
# Do not edit!
config.llvm_src_root = "{my directory}/llvm-tip-mc-hg"
config.llvm_obj_root = "{my directory}/BUILD/llvm-tip-mc"
config.llvm_tools_dir = "{my
directory}/BUILD/llvm-tip-mc/Debug+Asserts/bin"
------

After seeing this, I definitely thought I was going mad, as the
contents config.llvm_tools_dir seems to indicate, the executables
being used by the tests are from the freshly built artifacts dir, and
initial foray into to the utils/lit directory seem to confirm this,
but there is a wrinkle here, in the form of the next line:

config.llvmgcc_dir = "/pkgs/llvm-tip"

I certainly did NOT tell llvm configure where to find llvm-gcc, but
its still okay right? I mean, its just llvm-gcc, and that's the
directory that I had told llvm-gcc configure to place itself, and even
better! I was following directions!

Please take a brief look at the directions posted on steps 4-8 on
http://www.llvm.org/docs/GCCFEBuildInstrs.html

The key thing here is that the directions show the commingling of llvm
install artifacts AND the llvm-gcc install artifacts into the same
directory, which I happily followed, as it seemed like a good idea,
except it was most emphatically NOT a good idea when the build/testing
harness "guesses" what your path is and hallucinates new directories
on top of the directory which contains the actual utilities that
should be tested.

After jumping thorough some 7 or 8 layers of python calls in the
llvm-lit package to trace through what was getting called how,  I
discovered that indeed what I suspected was the case, that the PATH
was improperly sanitized. The directory config.llvmgcc_dir was getting
prepended ahead of llvm_tools_dir, but verifying this took some effort
as well.

Here are the issues that bit me, in random order:

1. Most importantly, the make check should anchor the utilities it uses
2. LLVM's configure should not assume existance of llvm-gcc if the
configure command line did not tell it so
3. The test script places that path ahead of the first-hand place
where the build just placed the files. I MEAN COME ON!! What was the
point of the build if you weren't going to use the artifacts just
built!!?? We're just gonna wear out the bits on the hard disk and on
the ram chips from overuse!! GOODNESS!! :-) :-) !!
4. Uhhh,, Silly me, I guess I shouldn't follow directions TOO closely,
I mean, they're mostly advisory! :-)

AARR (many many hundred 'R's) GGH!  :-)

Obviously, the short term solution to this is not to have llvm-gcc on
the path, or at least not commingle llvm-gcc and llvm in the same
install dir (as implied "GOOD THING" on the docs).

All in all, it was a great punk'in. I really really wish I hadn't gone
through it, but otherwise, you would not be reading this long-winded
entertaining missive!

I think the long term solution is to make sure that each llvm
executable in the script somehow be automagically anchored to the
llvm_tools_dir, I'd submiit a patch for that, but its late, and I need
a drink! :-)

Thanks for reading!

-jason

p.s.  YES! I'm not crazy! Wo0T!

---------- Forwarded message ----------
From: Rafael Espindola <espindola at google.com>
Date: Tue, Sep 21, 2010 at 7:49 AM
Subject: Re: [llvm-commits] Fwd: Initial cut of ARM MC ELF emitter (PATCH)
To: Jason Kim <jasonwkim at google.com>
Cc: llvm-commits at cs.uiuc.edu


On 20 September 2010 16:45, Jason Kim <jasonwkim at google.com>
wrote:
> Apologies once again. The whitespace issue in arm-mc-elf-s01.patch4
> has been fixed.
Hello Jason. With this patch I got the following failures in "make
check-lit"

  LLVM :: CodeGen/ARM/2009-10-27-double-align.ll
   LLVM :: CodeGen/ARM/align.ll
   LLVM :: CodeGen/ARM/arguments-nosplit-double.ll
   LLVM :: CodeGen/ARM/arguments-nosplit-i64.ll
   LLVM :: CodeGen/ARM/arguments.ll
   LLVM :: CodeGen/ARM/ldrd.ll
   LLVM :: CodeGen/ARM/va_arg.ll
   LLVM :: CodeGen/Thumb2/thumb2-ldrd.ll

Cheers,
--
Rafael Ávila de Espíndola

So, if I understand you correctly the summary is that doing

* Build llvm and install it in /foo/bar
* Build llvm-gcc  and install it in /foo/bar
* export PATH=/foo/bar/bin:$PATH
* Build llvm again in a unrelated build directory

and then running "make check-lit" in the second directory causes
/foo/bar/bin/llc and similar tools to be used instead of the recently
build tools? If that is the case, that really looks like a bug. Could
you please report it?
> Thanks for reading!
>
> -jason

Thanks,
-- 
Rafael Ávila de Espíndola

On Tue, Sep 21, 2010 at 9:20 PM, Rafael Espindola <espindola at
google.com> wrote:
> So, if I understand you correctly the summary is that doing
>
> * Build llvm and install it in /foo/bar
> * Build llvm-gcc  and install it in /foo/bar
> * export PATH=/foo/bar/bin:$PATH
> * Build llvm again in a unrelated build directory
>
> and then running "make check-lit" in the second directory causes
> /foo/bar/bin/llc and similar tools to be used instead of the recently
> build tools? If that is the case, that really looks like a bug. Could
> you please report it?
Just did.  PR8199

Thanks!

-jason

>
>> Thanks for reading!
>>
>> -jason
>
>
> Thanks,
> --
> Rafael Ávila de Espíndola
>