llvm dev - Jul 2015 - [LLVMdev] [cfe-dev] 3.6.2-rc1 has been tagged. Testers needed.

Hi Ben,

Thanks for taking the time to look at this.
>> It's not necessary for CMake to be installed when building with the
>> Autoconf/Makefile build system. If you look in the build directory
>> after a  build has been performed  you should see in cmake/modules
>>
>> ```
>> LLVMBuildExports.cmake  LLVMConfig.cmake  LLVMConfigVersion.cmake
>> LLVMExports.cmake  Makefile
>> ````
>
>
> I have these for the test-release build, but only in the *.obj directory,
> not the install directory.
Okay. So that means the files are being generated during the build but
not installed for some reason
>> The ``LLVMConfig.cmake`` and ``LLVMConfigExports.cmake`` file are
>> generated by the build in this directory and are later installed
>> (along with a bunch of other files). When I do a build of LLVM on Arch
>> Linux and install it in the install directory there is a
>> ``share/llvm/cmake/`` directory and it contains the following files
>> (this is LLVM trunk rather than 3.6.2 but it should be very similar).
>>
>> ```
>> AddLLVM.cmake             AddOCaml.cmake         ChooseMSVCCRT.cmake
>> FindOCaml.cmake   GetSVN.cmake             HandleLLVMStdlib.cmake
>> LLVMConfig.cmake         LLVMExports.cmake         TableGen.cmake
>> AddLLVMDefinitions.cmake  AddSphinxTarget.cmake  CrossCompile.cmake
>> FindSphinx.cmake  HandleLLVMOptions.cmake  LLVM-Config.cmake
>> LLVMConfigVersion.cmake  LLVMProcessSources.cmake
>> ```
>
> I have these in the install directory of my cmake based trunk build.
Yes those will definitely existing if you build LLVM using the CMake
build system but since LLVM 3.5 the Autoconf/Makefile build system
also generates and installs CMake files into the LLVM install tree.
Perhaps you could do a clean build of LLVM using the Autoconf/Makefile
build system outside of your chroot to see if there's something about
your chroot environment causing the problem?
>> You're going to have to do some debugging on your end because I
cannot
>> reproduce what ends up your tarball. I guess a first step would be to
>> try building LLVM in your chroot but outside of the test-release.sh
>> script. You could then try...
>>
>> 1. After doing a complete build have the
>> ``cmake/modules/LLVMConfig.cmake`` and
>> ``cmake/modules/LLVMExports.cmake`` files been generated in the build
>> directory?  If the CMake files are missing if you run ``make`` in the
>> ``cmake/modules/`` directory the files should be generated. Running
>> this manually should not be necessary though, it should happen
>> automatically.
>> SIDENOTE: After doing an initial configure the ``cmake/modules``
>> directory (and it's corresponding Makefile) will not exist if the
>> build is out of source. The directory and the makefile will exist
>> after doing a successful build.
>
> Yeah, I have those.
Okay and those existed after a build without you needing to run
``make`` inside the ``cmake/modules/`` directory in the build tree?
>> 2. Assuming the ``cmake/modules/LLVMConfig.cmake`` and
>> ``cmake/modules/LLVMExports.cmake`` files were generated in the
>> previous step, what happens when you run ``make install`` in the
>> ``cmake/modules`` directory? Do the files actually get installed?
>
> Now I have the share/llvm/cmake directory with 18 files.
Okay so I am right in understanding that after completing a build...

- When you ran ``make install`` from the root of the build tree the
CMake files are not installed into ``share/llvm/cmake``.
- After trying the above you ran ``make install`` in the
``cmake/modules`` directory in the build tree and the CMake files were
installed to ``share/llvm/cmake``

?
> In fact, I ran install from the root of the build directory and not a load
> of .a and a few .so files appeared in *.install/lib
I don't quite understand what you mean. Could you explain that again?
Are you saying that you ran ``make install`` again in the root of the
build tree and additional
libraries were installed?

If so that doesn't good. Sounds like a bug.

@Eric : Have you ever seen anything like this before?

Thanks,
Dan.

On Tuesday, July 07, 2015 11:35 AM, Dan Liew wrote:
> Hi Ben,
>
> Thanks for taking the time to look at this.
>
>>> It's not necessary for CMake to be installed when building with
the
>>> Autoconf/Makefile build system. If you look in the build directory
>>> after a  build has been performed  you should see in cmake/modules
>>>
>>> ```
>>> LLVMBuildExports.cmake  LLVMConfig.cmake  LLVMConfigVersion.cmake
>>> LLVMExports.cmake  Makefile
>>> ````
>>
>> I have these for the test-release build, but only in the *.obj
directory,
>> not the install directory.
> Okay. So that means the files are being generated during the build but
> not installed for some reason
>
>>> The ``LLVMConfig.cmake`` and ``LLVMConfigExports.cmake`` file are
>>> generated by the build in this directory and are later installed
>>> (along with a bunch of other files). When I do a build of LLVM on
Arch
>>> Linux and install it in the install directory there is a
>>> ``share/llvm/cmake/`` directory and it contains the following files
>>> (this is LLVM trunk rather than 3.6.2 but it should be very
similar).
>>>
>>> ```
>>> AddLLVM.cmake             AddOCaml.cmake        
ChooseMSVCCRT.cmake
>>> FindOCaml.cmake   GetSVN.cmake             HandleLLVMStdlib.cmake
>>> LLVMConfig.cmake         LLVMExports.cmake         TableGen.cmake
>>> AddLLVMDefinitions.cmake  AddSphinxTarget.cmake  CrossCompile.cmake
>>> FindSphinx.cmake  HandleLLVMOptions.cmake  LLVM-Config.cmake
>>> LLVMConfigVersion.cmake  LLVMProcessSources.cmake
>>> ```
>> I have these in the install directory of my cmake based trunk build.
> Yes those will definitely existing if you build LLVM using the CMake
> build system but since LLVM 3.5 the Autoconf/Makefile build system
> also generates and installs CMake files into the LLVM install tree.
> Perhaps you could do a clean build of LLVM using the Autoconf/Makefile
> build system outside of your chroot to see if there's something about
> your chroot environment causing the problem?
OK, I'll have a look, but it will have to be later.
>>> You're going to have to do some debugging on your end because I
cannot
>>> reproduce what ends up your tarball. I guess a first step would be
to
>>> try building LLVM in your chroot but outside of the test-release.sh
>>> script. You could then try...
>>>
>>> 1. After doing a complete build have the
>>> ``cmake/modules/LLVMConfig.cmake`` and
>>> ``cmake/modules/LLVMExports.cmake`` files been generated in the
build
>>> directory?  If the CMake files are missing if you run ``make`` in
the
>>> ``cmake/modules/`` directory the files should be generated. Running
>>> this manually should not be necessary though, it should happen
>>> automatically.
>>> SIDENOTE: After doing an initial configure the ``cmake/modules``
>>> directory (and it's corresponding Makefile) will not exist if
the
>>> build is out of source. The directory and the makefile will exist
>>> after doing a successful build.
>> Yeah, I have those.
> Okay and those existed after a build without you needing to run
> ``make`` inside the ``cmake/modules/`` directory in the build tree?
>
>>> 2. Assuming the ``cmake/modules/LLVMConfig.cmake`` and
>>> ``cmake/modules/LLVMExports.cmake`` files were generated in the
>>> previous step, what happens when you run ``make install`` in the
>>> ``cmake/modules`` directory? Do the files actually get installed?
>> Now I have the share/llvm/cmake directory with 18 files.
> Okay so I am right in understanding that after completing a build...
>
> - When you ran ``make install`` from the root of the build tree the
> CMake files are not installed into ``share/llvm/cmake``.
> - After trying the above you ran ``make install`` in the
> ``cmake/modules`` directory in the build tree and the CMake files were
> installed to ``share/llvm/cmake``
>
> ?
>
>> In fact, I ran install from the root of the build directory and not a
load
>> of .a and a few .so files appeared in *.install/lib
> I don't quite understand what you mean. Could you explain that again?
> Are you saying that you ran ``make install`` again in the root of the
> build tree and additional
> libraries were installed?
>
> If so that doesn't good. Sounds like a bug.
Sorry.  After I ran test-release, I didn't have the share/llvm/cmake 
directory.  When I ran make install from the root of the build 
directory, I got that directory and files.

It looks like it touched the .a and .so files, but I don't think new 
ones were created.

Ben

Hi,

@CC'ing Hans because this will likely be of interest to you.

Right I've started trying to build LLVM inside an Ubuntu chroot
(Ubuntu 14.04LTS Docker image) and I've already come across a pretty
bad bug in the ``test-release.sh`` script which potentially means that
builds and/or tests could potentially fail without anyone noticing
(unless someone carefully looks through the logs) because in certain
places if the build process fails the script will carry on executing
as if nothing bad happened!

The script contains ``set -e`` which is **supposed** to exit if any
command fails. However because of the way the script is implemented
this doesn't work. When commands are piped through the ``tee`` command
(which the script does for many parts of the build process) the exit
code is not exit code of the process of interest (e.g. the ``make`` or
``configure`` invocations).

Here's a simple example that illustrates the problem

```
$ false ; echo $?
1
 false | tee /dev/null ; echo $?
0
```

In addition to doing ``set -e`` what we also need to do ``set -o
pipefail`` in the ``test-release.sh`` script.

```
$ set -o pipefail
$ false ; echo $?
1
$ false | tee /dev/null ; echo $?
1
```

Is it okay for me to commit a fix for this to trunk?


@Ben: Could you add a line just after ``set -e`` in your copy of
``test-release.sh`` that runs ``set -o pipefail`` and try another
build using this patched script? This might not be the cause of the
packaging problems you're having but I suspect that it might be the
case that your build is failing in someway inside your chroot and that
the ``test-release.sh`` script is just carrying on regardless of this
failure resulting in a broken tarball being generated.


Thanks,
Dan.