Daniel Liew
2014-Jan-14 21:15 UTC
[LLVMdev] [llvm-lit] Mistakes in list of variables and substitutions documentation?
Hi All, I was taking a look at [1] because I was modifying an external tool to use llvm-lit instead of the deprecated DejaGNU testing framework and I was going to submit a patch to document the undocumented '%T' substitution when I realised the existing documentation doesn't quite make sense... * $test has a '$' prefix whereas most of the others don't. * I can't see any support for any of these '$' style substitutions (e.g. subdir, srcroot, objroot) in lit's python code or in the lit.site.cfg file. The only place I've seen the substitutions happening is parseIntegratedTestScript() inside TestRunner.py but they all use the '%' prefix. * Why the distinction between variables and substitutions? [1] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions Thanks, Dan Liew
Daniel Liew
2014-Jan-15 14:40 UTC
[LLVMdev] [llvm-lit] Mistakes in list of variables and substitutions documentation?
I thought I might get a better response if I provide something more
concrete.
Attached is the file subs_and_vars.c which can be placed in 'test/' in
the LLVM build directory which can then be executed by doing
$ cd test/
$ llvm-lit -v subs_and_vars.c
The output is shown in the attached file subs_and_vars_llvm-lit.txt.
As it can be seen most of the ('$' prefix) variables described in the
documentation do **not** work (expand to nothing) and quite a few
undocumented variables and substitutions ('%' prefix) do work.
I'm not saying that all of these should be documented but I don't think
the documentation should recommend the use of variables that do not
work. I also don't think the documentation should say the '%'
substitutions are deprecated when there are no '$' equivalents to
replace many '%' substitutions.
In my opinion the llvm-lit in built substitutions (e.g. %s, %T) should
be described in the llvm-lit manual [1] and the important lit.site.cfg
specific substitutions and variables should be described in [2]
[1] http://llvm.org/docs/CommandGuide/lit.html
[2] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions
I don't mind writing this documentation but I need some guidance on how
these variables and substitutions are being used currently by the LLVM
community.
Thanks,
Dan Liew.
On 14/01/14 21:15, Daniel Liew wrote:> Hi All,
>
> I was taking a look at [1] because I was modifying an external tool to
> use llvm-lit instead of the deprecated DejaGNU testing framework and I
> was going to submit a patch to document the undocumented '%T'
> substitution when I realised the existing documentation doesn't quite
> make sense...
>
> * $test has a '$' prefix whereas most of the others don't.
>
> * I can't see any support for any of these '$' style
substitutions
> (e.g. subdir, srcroot, objroot) in lit's python code or in the
> lit.site.cfg file. The only place I've seen the substitutions
> happening is
> parseIntegratedTestScript() inside TestRunner.py but they all use the
> '%' prefix.
>
> * Why the distinction between variables and substitutions?
>
>
> [1] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions
>
> Thanks,
> Dan Liew
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: subs_and_vars.c
Type: text/x-csrc
Size: 1473 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140115/37031804/attachment.c>
-------------- next part --------------
-- Testing: 1 tests, 1 threads --
FAIL: LLVM :: subs_and_vars.c (1 of 1)
******************** TEST 'LLVM :: subs_and_vars.c' FAILED
********************
Script:
--
echo === Documented and working ==echo perc_s =
/home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
echo perc_shlibext = .so
echo === Documented and NOT working ==echo dollar_test = $test
echo dollar_src = $srcdir
echo dollar_objdir = $objdir
echo dollar_subdir = $subdir
echo dollar_srcroot = $srcroot
echo dollar_objroot = $objroot
echo dollar_path = $path
echo dollar_tmp = $tmp
echo dollar_target_triplet = $target_triplet
echo perc_target_triplet =
/home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmparget_triplet
echo dollar_link = $link
echo perc_link = %link
echo dollar_shlibext = $shlibext
echo === NOT documented and working ==echo perc_t =
/home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
echo perc_T = /home/dsl11/dev/llvm-upstream/bin/test/Output
echo perc_p = /home/dsl11/dev/llvm-upstream/src/test
echo perc_S = /home/dsl11/dev/llvm-upstream/src/test
echo perc_{pathsep} = :
echo perc_/t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
echo perc_/T = /home/dsl11/dev/llvm-upstream/bin/test/Output
echo perc_/p = /home/dsl11/dev/llvm-upstream/src/test
echo perc_/S = /home/dsl11/dev/llvm-upstream/src/test
echo perc_/s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
echo dollar_PWD = $PWD
echo dollar_HOME = $HOME
echo dollar_LLVM_SRC_ROOT = $LLVM_SRC_ROOT
echo dollar_PYTHON_EXECUTABLE = $PYTHON_EXECUTABLE
echo perc_llvmshlibdir = /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib
echo perc_ocamlopt = -cc "" -cclib
-L/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib -I
/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib/ocaml
false
--
Exit Code: 1
Command Output (stdout):
--
=== Documented and working ==perc_s =
/home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
perc_shlibext = .so
=== Documented and NOT working ==dollar_test dollar_src dollar_objdir
dollar_subdir dollar_srcroot dollar_objroot dollar_path dollar_tmp
dollar_target_triplet perc_target_triplet =
/home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmparget_triplet
dollar_link perc_link = %link
dollar_shlibext === NOT documented and working ==perc_t =
/home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
perc_T = /home/dsl11/dev/llvm-upstream/bin/test/Output
perc_p = /home/dsl11/dev/llvm-upstream/src/test
perc_S = /home/dsl11/dev/llvm-upstream/src/test
perc_{pathsep} = :
perc_/t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
perc_/T = /home/dsl11/dev/llvm-upstream/bin/test/Output
perc_/p = /home/dsl11/dev/llvm-upstream/src/test
perc_/S = /home/dsl11/dev/llvm-upstream/src/test
perc_/s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
dollar_PWD = /home/dsl11/dev/llvm-upstream/bin/test
dollar_HOME = /home/dsl11
dollar_LLVM_SRC_ROOT = /home/dsl11/dev/llvm-upstream/src
dollar_PYTHON_EXECUTABLE = /usr/bin/python2.7
perc_llvmshlibdir = /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib
perc_ocamlopt = -cc -cclib
-L/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib -I
/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib/ocaml
--
********************
Testing Time: 0.01s
********************
Failing Tests (1):
LLVM :: subs_and_vars.c
Unexpected Failures: 1