llvm dev - Jun 2018 - Using lnt to run SPEC tests: possible or not?

It appears that lnt has the capability to run the 
SPEC2000/2006/2017(maybe) tests from looking at the options and from 
poking around in the mailing list archives.  Or does it?  I experimented 
a bit and cannot get it to work right.  However, it isn't really 
documented anywhere so maybe I didn't set it up properly.  Plus there's 
all the mention of llvmgcc...

So is anyone doing this?  Is there some setup information that I missed? 
  Thanks for any tips on how to do this!
-- 

-Bill Seurer

Hi Bill,

Yes, it is possible to run SPEC tests through LNT/test-suite.
You won’t get official reportable SPEC numbers as you don’t use the official
build-and-run framework from SPEC, but OTOH you get the extra features in
LNT/test-suite working out of the box, e.g. the ones documented at
http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.

To make use of this, I think you need to use the cmake driver in the test-suite.
Which, if you use the LNT wrapper to drive runs, is invoked using “lnt runtest
test-suite”.

A few example of the important options on the command line that I hope will get
you started:

  *   Running SPEC2K6int reference inputs: lnt runtest test-suite
--only-test=External/SPEC/CINT2006 --cmake-define
TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
TEST_SUITE_RUN_TYPE=ref

  *   Running SPEC2K6int train inputs: lnt runtest test-suite
--only-test=External/SPEC/CINT2006 --cmake-define
TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
TEST_SUITE_RUN_TYPE=train
  *   Running SPEC2Kfp reference inputs:  lnt runtest test-suite
--only-test=External/SPEC/CFP2000 --cmake-define
TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define
TEST_SUITE_RUN_TYPE=ref


Thanks,

Kristof


On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

It appears that lnt has the capability to run the SPEC2000/2006/2017(maybe)
tests from looking at the options and from poking around in the mailing list
archives.  Or does it?  I experimented a bit and cannot get it to work right. 
However, it isn't really documented anywhere so maybe I didn't set it up
properly.  Plus there's all the mention of llvmgcc...

So is anyone doing this?  Is there some setup information that I missed?  Thanks
for any tips on how to do this!
--

-Bill Seurer

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180620/980d44bc/attachment.html>

Thanks Kristof.

I am hitting some odd cmake issues (it isn't handing pthread stuff 
properly for some reason) that I haven't seen before but this should be 
me going.  I've used lnt for running the llvm test suite many times 
using "runtest nt" without problem.

Hmmm.  powerpc64 isn't listed in the known architectures in 
test_suite.py so I may need to update some of these files.

On 06/20/18 02:02, Kristof Beyls wrote:
> Hi Bill,
> 
> Yes, it is possible to run SPEC tests through LNT/test-suite.
> You won’t get official reportable SPEC numbers as you don’t use the 
> official build-and-run framework from SPEC, but OTOH you get the extra 
> features in LNT/test-suite working out of the box, e.g. the ones 
> documented at 
> http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.
> 
> To make use of this, I think you need to use the cmake driver in the 
> test-suite.
> Which, if you use the LNT wrapper to drive runs, is invoked using “lnt 
> runtest test-suite”.
> 
> A few example of the important options on the command line that I hope 
> will get you started:
> 
>   * Running SPEC2K6int reference inputs:lnt runtest test-suite
>     --only-test=External/SPEC/CINT2006 --cmake-define
>     TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>     TEST_SUITE_RUN_TYPE=ref
> 
>   * Running SPEC2K6int train inputs:lnt runtest test-suite
>     --only-test=External/SPEC/CINT2006 --cmake-define
>     TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>     TEST_SUITE_RUN_TYPE=train
>   * Running SPEC2Kfp reference inputs: lnt runtest test-suite
>     --only-test=External/SPEC/CFP2000 --cmake-define
>     TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define
>     TEST_SUITE_RUN_TYPE=ref
> 
> Thanks,
> 
> Kristof
> 
> 
>> On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev 
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>
>> It appears that lnt has the capability to run the 
>> SPEC2000/2006/2017(maybe) tests from looking at the options and from 
>> poking around in the mailing list archives.  Or does it?  I 
>> experimented a bit and cannot get it to work right.  However, it
isn't
>> really documented anywhere so maybe I didn't set it up properly.
 Plus
>> there's all the mention of llvmgcc...
>>
>> So is anyone doing this?  Is there some setup information that I 
>> missed?  Thanks for any tips on how to do this!
>> -- 
>>
>> -Bill Seurer
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 

-- 

-Bill Seurer