llvm dev - Jul 2015 - [LLVMdev] [cfe-dev] 3.6.2-rc1 has been tagged. Testers needed.

Hi,

@CC'ing Hans because this will likely be of interest to you.

Right I've started trying to build LLVM inside an Ubuntu chroot
(Ubuntu 14.04LTS Docker image) and I've already come across a pretty
bad bug in the ``test-release.sh`` script which potentially means that
builds and/or tests could potentially fail without anyone noticing
(unless someone carefully looks through the logs) because in certain
places if the build process fails the script will carry on executing
as if nothing bad happened!

The script contains ``set -e`` which is **supposed** to exit if any
command fails. However because of the way the script is implemented
this doesn't work. When commands are piped through the ``tee`` command
(which the script does for many parts of the build process) the exit
code is not exit code of the process of interest (e.g. the ``make`` or
``configure`` invocations).

Here's a simple example that illustrates the problem

```
$ false ; echo $?
1
 false | tee /dev/null ; echo $?
0
```

In addition to doing ``set -e`` what we also need to do ``set -o
pipefail`` in the ``test-release.sh`` script.

```
$ set -o pipefail
$ false ; echo $?
1
$ false | tee /dev/null ; echo $?
1
```

Is it okay for me to commit a fix for this to trunk?


@Ben: Could you add a line just after ``set -e`` in your copy of
``test-release.sh`` that runs ``set -o pipefail`` and try another
build using this patched script? This might not be the cause of the
packaging problems you're having but I suspect that it might be the
case that your build is failing in someway inside your chroot and that
the ``test-release.sh`` script is just carrying on regardless of this
failure resulting in a broken tarball being generated.


Thanks,
Dan.

On 6 July 2015 at 22:26, Dan Liew <dan at su-root.co.uk>
wrote:
> Hi,
>
> @CC'ing Hans because this will likely be of interest to you.
>
> Right I've started trying to build LLVM inside an Ubuntu chroot
> (Ubuntu 14.04LTS Docker image) and I've already come across a pretty
> bad bug in the ``test-release.sh`` script which potentially means that
> builds and/or tests could potentially fail without anyone noticing
> (unless someone carefully looks through the logs) because in certain
> places if the build process fails the script will carry on executing
> as if nothing bad happened!
>
> The script contains ``set -e`` which is **supposed** to exit if any
> command fails. However because of the way the script is implemented
> this doesn't work. When commands are piped through the ``tee`` command
> (which the script does for many parts of the build process) the exit
> code is not exit code of the process of interest (e.g. the ``make`` or
> ``configure`` invocations).
>
> Here's a simple example that illustrates the problem
>
> ```
> $ false ; echo $?
> 1
>  false | tee /dev/null ; echo $?
> 0
> ```
>
> In addition to doing ``set -e`` what we also need to do ``set -o
> pipefail`` in the ``test-release.sh`` script.
>
> ```
> $ set -o pipefail
> $ false ; echo $?
> 1
> $ false | tee /dev/null ; echo $?
> 1
> ```
>
> Is it okay for me to commit a fix for this to trunk?
@dimitry: Could you tell me if

set -o pipefail

works under the FreeBSD? I seem to recall that FreeBSD does not use
bash by default so I'm not sure what the ``test-release.sh`` script
shebang actually invokes under FreeBSD. Is it real bash or is it some
bourne compatible shell?

On Tuesday, July 07, 2015 01:26 PM, Dan Liew wrote:
> Hi,
>
> @CC'ing Hans because this will likely be of interest to you.
>
> Right I've started trying to build LLVM inside an Ubuntu chroot
> (Ubuntu 14.04LTS Docker image) and I've already come across a pretty
> bad bug in the ``test-release.sh`` script which potentially means that
> builds and/or tests could potentially fail without anyone noticing
> (unless someone carefully looks through the logs) because in certain
> places if the build process fails the script will carry on executing
> as if nothing bad happened!
>
> The script contains ``set -e`` which is **supposed** to exit if any
> command fails. However because of the way the script is implemented
> this doesn't work. When commands are piped through the ``tee`` command
> (which the script does for many parts of the build process) the exit
> code is not exit code of the process of interest (e.g. the ``make`` or
> ``configure`` invocations).
>
> Here's a simple example that illustrates the problem
>
> ```
> $ false ; echo $?
> 1
>   false | tee /dev/null ; echo $?
> 0
> ```
>
> In addition to doing ``set -e`` what we also need to do ``set -o
> pipefail`` in the ``test-release.sh`` script.
>
> ```
> $ set -o pipefail
> $ false ; echo $?
> 1
> $ false | tee /dev/null ; echo $?
> 1
> ```
>
> Is it okay for me to commit a fix for this to trunk?
>
>
> @Ben: Could you add a line just after ``set -e`` in your copy of
> ``test-release.sh`` that runs ``set -o pipefail`` and try another
> build using this patched script? This might not be the cause of the
> packaging problems you're having but I suspect that it might be the
> case that your build is failing in someway inside your chroot and that
> the ``test-release.sh`` script is just carrying on regardless of this
> failure resulting in a broken tarball being generated.
>
OK, good.  The change made it obvious that I had groff missing and that 
seems to have been killing the install step mid-flow.

Dan, I think it works now, so I've uploaded the new build to the site I 
sent you before for your perusal.

I've also uploaded clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.tar.xz 
to the llvm ftp site.

Ben

On 07 Jul 2015, at 08:10, Dan Liew <dan at su-root.co.uk>
wrote:
> 
> On 6 July 2015 at 22:26, Dan Liew <dan at su-root.co.uk> wrote:
...
>> In addition to doing ``set -e`` what we also need to do ``set -o
>> pipefail`` in the ``test-release.sh`` script.
>> 
>> ```
>> $ set -o pipefail
>> $ false ; echo $?
>> 1
>> $ false | tee /dev/null ; echo $?
>> 1
>> ```
>> 
>> Is it okay for me to commit a fix for this to trunk?
> 
> @dimitry: Could you tell me if
> 
> set -o pipefail
> 
> works under the FreeBSD? I seem to recall that FreeBSD does not use
> bash by default so I'm not sure what the ``test-release.sh`` script
> shebang actually invokes under FreeBSD. Is it real bash or is it some
> bourne compatible shell?
On FreeBSD, /bin/sh is a POSIX shell, which does not support "set -o
pipefail".  But for test-release.sh, this will not matter, as it starts
with:

#!/usr/bin/env bash

So it will just run /usr/local/bin/bash instead.  Obviously, the user
will have to install the bash port, but you need to install a bunch of
ports to build llvm anyway.

-Dimitry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150707/3ef5de86/attachment.sig>

> Is it okay for me to commit a fix for this to trunk?
I've committed this to trunk in r241599 now that I know the fix is
okay for FreeBSD.

> OK, good.  The change made it obvious that I had groff missing and that
> seems to have been killing the install step mid-flow.
>
> Dan, I think it works now, so I've uploaded the new build to the site I
sent
> you before for your perusal.
Great. I've taken a quick look the CMake files are there and the sym
links are okay. The CMake files themselves are broken but it's
unfortunately too late to apply the necessary fix from trunk (r241080)
to the release branch.

Running `` diff -s -q --recursive
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04
clang+llvm-3.6.2-rc1-x86_64-fedora21 | grep ^Only`` returns nothing
now :)

I've noticed though that  there's a difference between some of the
uploaded packages

```
$ diff -s -q --recursive
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04
clang+llvm-3.6.2-rc1-aarch64-linux-gnu | grep ^Only
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/lib/clang/3.6.2: lib

$ diff -s -q --recursive
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04
clang+llvm-3.6.2-rc1-amd64-unknown-freebsd10 | grep ^Only
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/lib/clang/3.6.2: lib

diff -s -q --recursive clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04
clang+llvm-3.6.2-rc1-armv7a-linux-gnueabihf | grep ^Only
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/lib/clang/3.6.2: lib

$ diff -s -q --recursive
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04
clang+llvm-3.6.2-rc1-mips-linux-gnu | grep ^Only
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/lib/clang/3.6.2: lib

$ diff -s -q --recursive
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04
clang+llvm-3.6.2-rc1-mipsel-linux-gnu | grep ^Only
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/lib/clang/3.6.2: lib
```

The x86_64 linux builds (i.e. Fedora, OpenSUSE and Ubuntu) and x86_64
darwin build seem to have an extra folder in ``lib/clang/3.6.2/lib/``
that has compiler-rt stuff in it (I think).

Is this to be expected?

Thanks,
Dan.