Daniel Liew
2014-Jan-14 21:15 UTC
[LLVMdev] [llvm-lit] Mistakes in list of variables and substitutions documentation?
Hi All, I was taking a look at [1] because I was modifying an external tool to use llvm-lit instead of the deprecated DejaGNU testing framework and I was going to submit a patch to document the undocumented '%T' substitution when I realised the existing documentation doesn't quite make sense... * $test has a '$' prefix whereas most of the others don't. * I can't see any support for any of these '$' style substitutions (e.g. subdir, srcroot, objroot) in lit's python code or in the lit.site.cfg file. The only place I've seen the substitutions happening is parseIntegratedTestScript() inside TestRunner.py but they all use the '%' prefix. * Why the distinction between variables and substitutions? [1] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions Thanks, Dan Liew
Daniel Liew
2014-Jan-15 14:40 UTC
[LLVMdev] [llvm-lit] Mistakes in list of variables and substitutions documentation?
I thought I might get a better response if I provide something more concrete. Attached is the file subs_and_vars.c which can be placed in 'test/' in the LLVM build directory which can then be executed by doing $ cd test/ $ llvm-lit -v subs_and_vars.c The output is shown in the attached file subs_and_vars_llvm-lit.txt. As it can be seen most of the ('$' prefix) variables described in the documentation do **not** work (expand to nothing) and quite a few undocumented variables and substitutions ('%' prefix) do work. I'm not saying that all of these should be documented but I don't think the documentation should recommend the use of variables that do not work. I also don't think the documentation should say the '%' substitutions are deprecated when there are no '$' equivalents to replace many '%' substitutions. In my opinion the llvm-lit in built substitutions (e.g. %s, %T) should be described in the llvm-lit manual [1] and the important lit.site.cfg specific substitutions and variables should be described in [2] [1] http://llvm.org/docs/CommandGuide/lit.html [2] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions I don't mind writing this documentation but I need some guidance on how these variables and substitutions are being used currently by the LLVM community. Thanks, Dan Liew. On 14/01/14 21:15, Daniel Liew wrote:> Hi All, > > I was taking a look at [1] because I was modifying an external tool to > use llvm-lit instead of the deprecated DejaGNU testing framework and I > was going to submit a patch to document the undocumented '%T' > substitution when I realised the existing documentation doesn't quite > make sense... > > * $test has a '$' prefix whereas most of the others don't. > > * I can't see any support for any of these '$' style substitutions > (e.g. subdir, srcroot, objroot) in lit's python code or in the > lit.site.cfg file. The only place I've seen the substitutions > happening is > parseIntegratedTestScript() inside TestRunner.py but they all use the > '%' prefix. > > * Why the distinction between variables and substitutions? > > > [1] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions > > Thanks, > Dan Liew > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- A non-text attachment was scrubbed... Name: subs_and_vars.c Type: text/x-csrc Size: 1473 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140115/37031804/attachment.c> -------------- next part -------------- -- Testing: 1 tests, 1 threads -- FAIL: LLVM :: subs_and_vars.c (1 of 1) ******************** TEST 'LLVM :: subs_and_vars.c' FAILED ******************** Script: -- echo === Documented and working ==echo perc_s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c echo perc_shlibext = .so echo === Documented and NOT working ==echo dollar_test = $test echo dollar_src = $srcdir echo dollar_objdir = $objdir echo dollar_subdir = $subdir echo dollar_srcroot = $srcroot echo dollar_objroot = $objroot echo dollar_path = $path echo dollar_tmp = $tmp echo dollar_target_triplet = $target_triplet echo perc_target_triplet = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmparget_triplet echo dollar_link = $link echo perc_link = %link echo dollar_shlibext = $shlibext echo === NOT documented and working ==echo perc_t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp echo perc_T = /home/dsl11/dev/llvm-upstream/bin/test/Output echo perc_p = /home/dsl11/dev/llvm-upstream/src/test echo perc_S = /home/dsl11/dev/llvm-upstream/src/test echo perc_{pathsep} = : echo perc_/t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp echo perc_/T = /home/dsl11/dev/llvm-upstream/bin/test/Output echo perc_/p = /home/dsl11/dev/llvm-upstream/src/test echo perc_/S = /home/dsl11/dev/llvm-upstream/src/test echo perc_/s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c echo dollar_PWD = $PWD echo dollar_HOME = $HOME echo dollar_LLVM_SRC_ROOT = $LLVM_SRC_ROOT echo dollar_PYTHON_EXECUTABLE = $PYTHON_EXECUTABLE echo perc_llvmshlibdir = /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib echo perc_ocamlopt = -cc "" -cclib -L/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib -I /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib/ocaml false -- Exit Code: 1 Command Output (stdout): -- === Documented and working ==perc_s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c perc_shlibext = .so === Documented and NOT working ==dollar_test dollar_src dollar_objdir dollar_subdir dollar_srcroot dollar_objroot dollar_path dollar_tmp dollar_target_triplet perc_target_triplet = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmparget_triplet dollar_link perc_link = %link dollar_shlibext === NOT documented and working ==perc_t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp perc_T = /home/dsl11/dev/llvm-upstream/bin/test/Output perc_p = /home/dsl11/dev/llvm-upstream/src/test perc_S = /home/dsl11/dev/llvm-upstream/src/test perc_{pathsep} = : perc_/t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp perc_/T = /home/dsl11/dev/llvm-upstream/bin/test/Output perc_/p = /home/dsl11/dev/llvm-upstream/src/test perc_/S = /home/dsl11/dev/llvm-upstream/src/test perc_/s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c dollar_PWD = /home/dsl11/dev/llvm-upstream/bin/test dollar_HOME = /home/dsl11 dollar_LLVM_SRC_ROOT = /home/dsl11/dev/llvm-upstream/src dollar_PYTHON_EXECUTABLE = /usr/bin/python2.7 perc_llvmshlibdir = /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib perc_ocamlopt = -cc -cclib -L/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib -I /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib/ocaml -- ******************** Testing Time: 0.01s ******************** Failing Tests (1): LLVM :: subs_and_vars.c Unexpected Failures: 1