[LLVMdev] [RFC] Benchmarking subset of the test suite

Hal Finkel
2014-05-04T20:33:35Z

----- Original Message -----
> From: "Chris Lattner" <clattner at apple.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Sunday, May 4, 2014 1:37:42 PM
> Subject: Re: [LLVMdev] [RFC] Benchmarking subset of the test suite
>
> On May 4, 2014, at 5:39 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> > Without further ado, I propose that a test-suite configuration
> > designed for benchmarking exclude the following:
>
> Unrelatedly to your point, I think we should just outright remove the
> bulk of the SingleSource/Regression / SingleSource/UnitTests tests.
>  They largely date back to before Clang existed and aren’t serving
> much useful value anymore.
>
> Any of them that are useful (e.g. the EH tests?) could be saved or
> moved somewhere else.

I have mixed feelings about this. On the one hand, the regression and unit tests
don't belong with the others, and if we start running applications multiple
times for benchmarking, we'd likely want to exclude these correctness-only
tests. On the other hand, I've found some of these tests quite useful in
catching miscompiles (the EH tests, the bitfield tests, vector intrinsics tests
and the va_args tests specifically come to mind). It might be useful to separate
out some set of 'ABI' tests, even if we remove the others.

 -Hal

>
> -Chris
>
>
> >
> > SingleSource/Regression/C++/2003-05-14-array-init
> > SingleSource/Regression/C++/2003-05-14-expr_stmt
> > SingleSource/Regression/C/2003-05-14-initialize-string
> > SingleSource/Regression/C/2003-05-21-BitfieldHandling
> > SingleSource/Regression/C/2003-05-21-UnionBitfields
> > SingleSource/Regression/C/2003-05-21-UnionTest
> > SingleSource/Regression/C/2003-05-22-LocalTypeTest
> > SingleSource/Regression/C/2003-05-22-VarSizeArray
> > SingleSource/Regression/C/2003-05-23-TransparentUnion
> > SingleSource/Regression/C++/2003-06-08-BaseType
> > SingleSource/Regression/C++/2003-06-08-VirtualFunctions
> > SingleSource/Regression/C++/2003-06-13-Crasher
> > SingleSource/Regression/C/2003-06-16-InvalidInitializer
> > SingleSource/Regression/C/2003-06-16-VolatileError
> > SingleSource/Regression/C++/2003-08-20-EnumSizeProblem
> > SingleSource/Regression/C++/2003-09-29-NonPODsByValue
> > SingleSource/Regression/C/2003-10-12-GlobalVarInitializers
> > SingleSource/Regression/C/2004-02-03-AggregateCopy
> > SingleSource/Regression/C/2004-03-15-IndirectGoto
> > SingleSource/Regression/C/2005-05-06-LongLongSignedShift
> > SingleSource/Regression/C/2008-01-07-LongDouble
> > SingleSource/Regression/C++/2008-01-29-ParamAliasesReturn
> > SingleSource/Regression/C++/2011-03-28-Bitfield
> > SingleSource/Regression/C/badidx
> > SingleSource/Regression/C/bigstack
> > SingleSource/Regression/C++/BuiltinTypeInfo
> > SingleSource/Regression/C/callargs
> > SingleSource/Regression/C/casts
> > SingleSource/Regression/C/compare
> > SingleSource/Regression/C/ConstructorDestructorAttributes
> > SingleSource/Regression/C/DuffsDevice
> > SingleSource/Regression/C++/EH/class_hierarchy
> > SingleSource/Regression/C++/EH/ConditionalExpr
> > SingleSource/Regression/C++/EH/ctor_dtor_count
> > SingleSource/Regression/C++/EH/ctor_dtor_count-2
> > SingleSource/Regression/C++/EH/dead_try_block
> > SingleSource/Regression/C++/EH/exception_spec_test
> > SingleSource/Regression/C++/EH/function_try_block
> > SingleSource/Regression/C++/EH/inlined_cleanup
> > SingleSource/Regression/C++/EH/recursive-throw
> > SingleSource/Regression/C++/EH/simple_rethrow
> > SingleSource/Regression/C++/EH/simple_throw
> > SingleSource/Regression/C++/EH/throw_rethrow_test
> > SingleSource/Regression/C++/fixups
> > SingleSource/Regression/C++/global_ctor
> > SingleSource/Regression/C/globalrefs
> > SingleSource/Regression/C++/global_type
> > SingleSource/Regression/C++/ofstream_ctor
> > SingleSource/Regression/C/pointer_arithmetic
> > SingleSource/Regression/C++/pointer_member
> > SingleSource/Regression/C++/pointer_method
> > SingleSource/Regression/C++/pointer_method2
> > SingleSource/Regression/C/PR10189
> > SingleSource/Regression/C/PR1386
> > SingleSource/Regression/C/PR491
> > SingleSource/Regression/C/PR640
> > SingleSource/Regression/C++/short_circuit_dtor
> > SingleSource/Regression/C/sumarray
> > SingleSource/Regression/C/sumarraymalloc
> > SingleSource/Regression/C/testtrace
> > SingleSource/UnitTests/2002-04-17-PrintfChar
> > SingleSource/UnitTests/2002-05-02-ArgumentTest
> > SingleSource/UnitTests/2002-05-02-CastTest
> > SingleSource/UnitTests/2002-05-02-CastTest1
> > SingleSource/UnitTests/2002-05-02-CastTest2
> > SingleSource/UnitTests/2002-05-02-CastTest3
> > SingleSource/UnitTests/2002-05-02-ManyArguments
> > SingleSource/UnitTests/2002-05-03-NotTest
> > SingleSource/UnitTests/2002-05-19-DivTest
> > SingleSource/UnitTests/2002-08-02-CastTest
> > SingleSource/UnitTests/2002-08-02-CastTest2
> > SingleSource/UnitTests/2002-08-19-CodegenBug
> > SingleSource/UnitTests/2002-10-09-ArrayResolution
> > SingleSource/UnitTests/2002-10-12-StructureArgs
> > SingleSource/UnitTests/2002-10-12-StructureArgsSimple
> > SingleSource/UnitTests/2002-10-13-BadLoad
> > SingleSource/UnitTests/2002-12-13-MishaTest
> > SingleSource/UnitTests/2003-04-22-Switch
> > SingleSource/UnitTests/2003-05-02-DependentPHI
> > SingleSource/UnitTests/2003-05-07-VarArgs
> > SingleSource/UnitTests/2003-05-12-MinIntProblem
> > SingleSource/UnitTests/2003-05-14-AtExit
> > SingleSource/UnitTests/2003-05-26-Shorts
> > SingleSource/UnitTests/2003-05-31-CastToBool
> > SingleSource/UnitTests/2003-05-31-LongShifts
> > SingleSource/UnitTests/2003-07-06-IntOverflow
> > SingleSource/UnitTests/2003-07-08-BitOpsTest
> > SingleSource/UnitTests/2003-07-09-LoadShorts
> > SingleSource/UnitTests/2003-07-09-SignedArgs
> > SingleSource/UnitTests/2003-07-10-SignConversions
> > SingleSource/UnitTests/2003-08-05-CastFPToUint
> > SingleSource/UnitTests/2003-08-11-VaListArg
> > SingleSource/UnitTests/2003-08-20-FoldBug
> > SingleSource/UnitTests/2003-09-18-BitFieldTest
> > SingleSource/UnitTests/2003-10-13-SwitchTest
> > SingleSource/UnitTests/2003-10-29-ScalarReplBug
> > SingleSource/UnitTests/2004-02-02-NegativeZero
> > SingleSource/UnitTests/2004-06-20-StaticBitfieldInit
> > SingleSource/UnitTests/2004-11-28-GlobalBoolLayout
> > SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls
> > SingleSource/UnitTests/2005-05-12-Int64ToFP
> > SingleSource/UnitTests/2005-05-13-SDivTwo
> > SingleSource/UnitTests/2005-07-17-INT-To-FP
> > SingleSource/UnitTests/2005-11-29-LongSwitch
> > SingleSource/UnitTests/2006-01-29-SimpleIndirectCall
> > SingleSource/UnitTests/2006-02-04-DivRem
> > SingleSource/UnitTests/2006-12-01-float_varg
> > SingleSource/UnitTests/2006-12-04-DynAllocAndRestore
> > SingleSource/UnitTests/2006-12-07-Compare64BitConstant
> > SingleSource/UnitTests/2006-12-11-LoadConstants
> > SingleSource/UnitTests/2007-01-04-KNR-Args
> > SingleSource/UnitTests/2007-03-02-VaCopy
> > SingleSource/UnitTests/2007-04-25-weak
> > SingleSource/UnitTests/2008-04-18-LoopBug
> > SingleSource/UnitTests/2008-04-20-LoopBug2
> > SingleSource/UnitTests/2008-07-13-InlineSetjmp
> > SingleSource/UnitTests/2009-04-16-BitfieldInitialization
> > SingleSource/UnitTests/2009-12-07-StructReturn
> > SingleSource/UnitTests/2010-05-24-BitfieldTest
> > SingleSource/UnitTests/AtomicOps
> > SingleSource/UnitTests/block-byref-cxxobj-test
> > SingleSource/UnitTests/block-byref-test
> > SingleSource/UnitTests/block-call-r7674133
> > SingleSource/UnitTests/block-copied-in-cxxobj
> > SingleSource/UnitTests/block-copied-in-cxxobj-1
> > SingleSource/UnitTests/blockstret
> > SingleSource/UnitTests/byval-alignment
> > SingleSource/UnitTests/conditional-gnu-ext
> > SingleSource/UnitTests/conditional-gnu-ext-cxx
> > SingleSource/UnitTests/DefaultInitDynArrays
> > SingleSource/UnitTests/FloatPrecision
> > SingleSource/UnitTests/initp1
> > SingleSource/UnitTests/member-function-pointers
> > SingleSource/UnitTests/printargs
> > SingleSource/UnitTests/SignlessTypes/cast2
> > SingleSource/UnitTests/SignlessTypes/cast-bug
> > SingleSource/UnitTests/SignlessTypes/ccc
> > SingleSource/UnitTests/SignlessTypes/div
> > SingleSource/UnitTests/SignlessTypes/factor
> > SingleSource/UnitTests/SignlessTypes/Large/cast
> > SingleSource/UnitTests/SignlessTypes/shr
> > SingleSource/UnitTests/stmtexpr
> > SingleSource/UnitTests/StructModifyTest
> > SingleSource/UnitTests/TestLoop
> > SingleSource/UnitTests/Threads/2010-12-08-tls
> > SingleSource/UnitTests/Threads/tls
> > SingleSource/UnitTests/Vector/build
> > SingleSource/UnitTests/Vector/divides
> > SingleSource/UnitTests/Vector/sumarray
> > SingleSource/UnitTests/Vector/sumarray-dbl
> > SingleSource/UnitTests/vla
> >
> > SingleSource/UnitTests/Vector/Altivec/2007-01-07-lvsl-lvsr-Regression
> > SingleSource/UnitTests/Vector/Altivec/alti.sdot
> > SingleSource/UnitTests/Vector/Altivec/casts
> > SingleSource/UnitTests/Vector/Altivec/test1
> >
> > SingleSource/UnitTests/ms_struct-bitfield
> > SingleSource/UnitTests/ms_struct-bitfield-1
> > SingleSource/UnitTests/ms_struct-bitfield-init
> > SingleSource/UnitTests/ms_struct-bitfield-init-1
> > SingleSource/UnitTests/ms_struct_pack_layout
> > SingleSource/UnitTests/ms_struct_pack_layout-1
>
>

--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory