llvm dev - Jun 2019 - Help required on running the regression tests

Please, can anybody show me the required steps for running the LLVM regression
tests, or point me to a detailed doc?

I read the available documentation in https://llvm.org/docs/TestingGuide.html
<https://llvm.org/docs/TestingGuide.html>  but that's mostly a general
description on what’s available rather than an “guide”.

This is what I tried:

- Installed both LLVM and CLANG using CMAKE on a Mac OS X computer. (Sources
compiled fine, and everything seems to work as I’m already at the final steps of
a custom LLVM target implementation)
- Python 2.7.10 is already installed.
- From the command line (terminal app) I tried to execute python lit.site.cfg.py
, however I get the following error:

File "/Users/joan/LLVM+CLANG/build/test/lit.site.cfg.py", line 6, in
<module>
    config.host_triple = "x86_64-apple-darwin16.7.0"
NameError: name 'config' is not defined

There seem to be something obvious that I miss, so any help is appreciated.

Please be aware that I’m an experienced C, C++, Obj-C programmer (now retired)
but I do not speak a word of python.

Thanks,

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190614/04aac232/attachment.html>

On Fri, 14 Jun 2019 at 10:10, Joan Lluch via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> I read the available documentation in
https://llvm.org/docs/TestingGuide.html  but that's mostly a general
description on what’s available rather than an “guide”.
It mentions the main commands: "make check-all" will run the
regression tests for all built components. There are also more
specific targets like "check" (LLVM only), "check-clang",
...

If you're using ninja you can get it to list them with something like
"ninja -t targets | grep check".
> - From the command line (terminal app) I tried to execute python
lit.site.cfg.py , however I get the following error:
>
> File "/Users/joan/LLVM+CLANG/build/test/lit.site.cfg.py", line 6,
in <module>
>     config.host_triple = "x86_64-apple-darwin16.7.0"
> NameError: name 'config' is not defined
A tool bin/llvm-lit is generated for running tests either individually
or when pointed at a test directory. It takes care of the needed
imports, PYTHONPATHs and so on. It's invoked by the "check"
targets in
make. That's the only real way to interact with lit stuff directly.

It's mostly useful when debugging a single test that's failed
"bin/llvm-lit -v /path/to/test.ll"  quickly tells you if your
attempted fix has worked. I occasionally use it to test all CodeGen
for a particular target without running other things that I don't care
about right now ("bin/llvm-lit -v
/path/to/llvm/test/CodeGen/AArch64").

Cheers.

Tim.

Hi Tim,

When I started with this, I just downloaded the sources for the LLVM site, I did
not even created a git for it. Then ran the CMAKE app (not from the command
line) with the default settings other than also adding the ‘experimental’
targets, and boom a fully functional XCode project was created, so I gave no
further thought on that. I am not even aware of what was involved under the
hood, but this has worked fine since I have been able to create a totally new
custom target from that install with no issues. That’s why now I’m surprised
that the tests doesn’t seem to even be there. I think that CMAKE is one of the
major ways to install LLVM, and I may try to install it and ‘clean’ compile from
the ground up again, but I would need to understand what’s required to actually
get the tests available.

Thanks,

> On 14 Jun 2019, at 12:25, Tim Northover <t.p.northover at gmail.com>
wrote:
> 
> Hi Joan,
> 
> You should probably reply with this on the list, and mention the exact
> commands you *are* using to configure and build etc. It's been ages
> since I've built with make, but I've never seen the testing options
> missing in any configuration.
> 
> The only thing I can think of is that the tests have to be run from
> the build directory as far as I know. If you're trying to do it on an
> already-installed LLVM, or run "make check" from a subdirectory
of the
> build path, that might go badly.
> 
> Cheers.
> 
> Tim.
> 
> On Fri, 14 Jun 2019 at 11:21, Joan Lluch <joan.lluch at icloud.com>
wrote:
>> 
>> Hi Tim, Thanks for the kick reply, I read that
>> 
>> but when I try make check-llm, or make check-all or make check  I
always get messages similar to this one:
>> 
>> make: *** No rule to make target `check-llvm'.  Stop.
>> 
>> I’m not using ninja, but I assume I do not need it for running the
tests.
>> 
>> I think, there’s something very basic that I might be missing, because
as said the ‘make’ tool doesn’t even work (?).
>> 
>> Maybe, Do I need to compile (or make) LLVM with a particular settings
to get the test code ready to run?
>> 
>> John
>> 
>> 
>> On 14 Jun 2019, at 12:11, Tim Northover <t.p.northover at
gmail.com> wrote:
>> 
>> On Fri, 14 Jun 2019 at 10:10, Joan Lluch via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> 
>> I read the available documentation in
https://llvm.org/docs/TestingGuide.html  but that's mostly a general
description on what’s available rather than an “guide”.
>> 
>> 
>> It mentions the main commands: "make check-all" will run the
>> regression tests for all built components. There are also more
>> specific targets like "check" (LLVM only),
"check-clang", ...
>> 
>> If you're using ninja you can get it to list them with something
like
>> "ninja -t targets | grep check".
>> 
>> - From the command line (terminal app) I tried to execute python
lit.site.cfg.py , however I get the following error:
>> 
>> File "/Users/joan/LLVM+CLANG/build/test/lit.site.cfg.py",
line 6, in <module>
>>   config.host_triple = "x86_64-apple-darwin16.7.0"
>> NameError: name 'config' is not defined
>> 
>> 
>> A tool bin/llvm-lit is generated for running tests either individually
>> or when pointed at a test directory. It takes care of the needed
>> imports, PYTHONPATHs and so on. It's invoked by the
"check" targets in
>> make. That's the only real way to interact with lit stuff directly.
>> 
>> It's mostly useful when debugging a single test that's failed
>> "bin/llvm-lit -v /path/to/test.ll"  quickly tells you if your
>> attempted fix has worked. I occasionally use it to test all CodeGen
>> for a particular target without running other things that I don't
care
>> about right now ("bin/llvm-lit -v
>> /path/to/llvm/test/CodeGen/AArch64").
>> 
>> Cheers.
>> 
>> Tim.
>> 
>>

Hi Joan,

On Fri, 14 Jun 2019 at 11:45, Joan Lluch <joan.lluch at icloud.com>
wrote:
> When I started with this, I just downloaded the sources for the LLVM site,
I did not even created a git for it. Then ran the CMAKE app (not from the
command line) with the default settings other than also adding the
‘experimental’ targets, and boom a fully functional XCode project was created,
so I gave no further thought on that.
Oh, Xcode. Now that's a large wrinkle in matters. I take it that means
you used "cmake -G Xcode"? I think most people who work on LLVM use
either the "Unix Makefiles" or he "Ninja" backends for
CMake, which do
generate the check destinations needed. The xcode backend would
completely replace any make/ninja files CMake would generate so that
would explain what you're seeing.

Some more people (I think) have a separate xcode build directory
purely for indexing/IDE but do their main builds in a make/ninja
environment.

Anyway, I've just generated an Xcode project myself and (by running
"xcodebuild -list") there does appear to be an "xcodebuild
-target
check" (etc). I've not actually run them because at the moment I only
have access to a laptop that would take quite a while to finish, but I
think that's probably your best bet.

Cheers.

Tim.