llvm dev - Jul 2015 - [LLVMdev] [cfe-dev] 3.6.2-rc1 has been tagged. Testers needed.

Hi,

I've taken a look at the binary tarballs for clang+llvm-3.6.2-rc1-*
that have been uploaded and observed the following:

* All uploaded tarballs include the generated CMake files apart from
``clang+llvm-3.6.2-rc1-x86_64-linux-gnu-ubuntu-14.04.tar.xz``.
  These files (share/llvm/cmake/*.cmake) are missing in this tarball.

* Inside the Ubuntu tarball the directory is not consistently named
compared to the other tarballs. For example inside
``clang+llvm-3.6.2-rc1-x86_64-fedora21.tar.xz``
there is a directory named ``clang+llvm-3.6.2-rc1-x86_64-fedora21``.
Inside ``clang+llvm-3.6.2-rc1-x86_64-linux-gnu-ubuntu-14.04.tar.xz``
the directory is named
``clang+llvm-3.6.2-rc1-x86_64-linux-gnu``.

* In the tarballs that do include the CMake files they are broken
because they contain absolute paths. r241080 needs to be back ported
in order to fix this.

* All tarballs are packaged incorrectly because --prefix is being
misused by ``test-release.sh``. I have a patch for this [1] which is
awaiting review (look at V2, the original patch doesn't work). You can
observe
  this by taking a look at ``include/llvm/Config/llvm-config.h``. It
contains paths that are almost certainly not going to exist on a
user's system. The patch will replace the temporary install directory
in files like these with ``/usr/local`` but will still place files in
the temporary install directory for tarballing.


@Ben Pope: Your uploaded binary tarball is the odd one out. Any idea
as to why the CMake files are missing and why the root directory name
inside the tarball is not consistent with the uploaded tarballs for
other platforms?

[1]
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150629/285597.html

On 7 July 2015 at 01:12, Dan Liew <dan at su-root.co.uk> wrote:
>
> @Ben Pope: Your uploaded binary tarball is the odd one out. Any idea
> as to why the CMake files are missing and why the root directory name
> inside the tarball is not consistent with the uploaded tarballs for
> other platforms?
>
> [1]
>
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150629/285597.html
>
Well, I rename the tarball to include the distro, should I be attacking
that by passing an argument to the test-release script?

No idea why the cmake files aren't being packaged, I build in a pretty
basic chroot, should cmake be installed there?

Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150707/9a119ae6/attachment.html>

On 6 July 2015 at 10:34, Ben Pope <benpope81 at gmail.com>
wrote:
> On 7 July 2015 at 01:12, Dan Liew <dan at su-root.co.uk> wrote:
>>
>>
>> @Ben Pope: Your uploaded binary tarball is the odd one out. Any idea
>> as to why the CMake files are missing and why the root directory name
>> inside the tarball is not consistent with the uploaded tarballs for
>> other platforms?
>>
>> [1]
>>
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150629/285597.html
>
>
> Well, I rename the tarball to include the distro, should I be attacking
that
> by passing an argument to the test-release script?
AFAICT from reading ``test-release.sh`` the value passed to the
``-triple`` option controls the tarball name **ONLY**,
``-build-triple`` actually controls the target triple passed to the
configure script. I think that's kind of confusing, ``-triple``
probably ought to be called something like ``-tarballsuffix``.

So I guess when you run ``test-release.sh`` you should pass ``-triple
x86_64-linux-gnu-ubuntu-14.04`` instead of  ``-triple
x86_64-linux-gnu``.
> No idea why the cmake files aren't being packaged, I build in a pretty
basic
> chroot, should cmake be installed there?
It's not necessary for CMake to be installed when building with the
Autoconf/Makefile build system. If you look in the build directory
after a  build has been performed  you should see in cmake/modules

```
LLVMBuildExports.cmake  LLVMConfig.cmake  LLVMConfigVersion.cmake
LLVMExports.cmake  Makefile
````

The ``LLVMConfig.cmake`` and ``LLVMConfigExports.cmake`` file are
generated by the build in this directory and are later installed
(along with a bunch of other files). When I do a build of LLVM on Arch
Linux and install it in the install directory there is a
``share/llvm/cmake/`` directory and it contains the following files
(this is LLVM trunk rather than 3.6.2 but it should be very similar).

```
AddLLVM.cmake             AddOCaml.cmake         ChooseMSVCCRT.cmake
FindOCaml.cmake   GetSVN.cmake             HandleLLVMStdlib.cmake
LLVMConfig.cmake         LLVMExports.cmake         TableGen.cmake
AddLLVMDefinitions.cmake  AddSphinxTarget.cmake  CrossCompile.cmake
FindSphinx.cmake  HandleLLVMOptions.cmake  LLVM-Config.cmake
LLVMConfigVersion.cmake  LLVMProcessSources.cmake
```

You're going to have to do some debugging on your end because I cannot
reproduce what ends up your tarball. I guess a first step would be to
try building LLVM in your chroot but outside of the test-release.sh
script. You could then try...

1. After doing a complete build have the
``cmake/modules/LLVMConfig.cmake`` and
``cmake/modules/LLVMExports.cmake`` files been generated in the build
directory?  If the CMake files are missing if you run ``make`` in the
``cmake/modules/`` directory the files should be generated. Running
this manually should not be necessary though, it should happen
automatically.
SIDENOTE: After doing an initial configure the ``cmake/modules``
directory (and it's corresponding Makefile) will not exist if the
build is out of source. The directory and the makefile will exist
after doing a successful build.

2. Assuming the ``cmake/modules/LLVMConfig.cmake`` and
``cmake/modules/LLVMExports.cmake`` files were generated in the
previous step, what happens when you run ``make install`` in the
``cmake/modules`` directory? Do the files actually get installed?

Thanks,
Dan.

On 6 July 2015 at 20:29, Ben Pope <benpope81 at gmail.com>
wrote:
> I've built 3.6.2 final.  I changed the triple as suggested.
>
> I've uploaded two archives here:
<snip>
>
> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.tar.xz is the one produced
by
> the release script.
>
> To produce clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.installed.tar.xz
I
> basically did:
> cd Phase3/Release/llvmCore-3.6.2-final.obj/
> make install
> cd ../llvmCore-3.6.2-final.install/
> tar cavf
> ../../../clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.installed.tar.xz
> --transform 's,^,clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/,'
*
>
> I hope that helps.
Thanks.

I diffed the contents of your old tarball looking for files that only
exist in one of the tarballs.

```
$ diff --recursive -q clang+llvm-3.6.2-rc1-x86_64-linux-gnu
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04 | grep ^Only
diff: clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/clang++: No
such file or directory
diff: clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/llvm-ranlib:
No such file or directory
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/docs/llvm/html:
Dummy.html
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/docs/llvm: ps
Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/share: llvm
```

So I see the missing CMake files are present however there are new issues.

- The symlinks to clang++ and llvm-ranlib are now broken :(

```
$ ls -l clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/clang++
lrwxrwxrwx 1 dsl11 multicore 52 Jul  7 03:54
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/clang++ ->
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/clang

$ ls -l clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/clang++
lrwxrwxrwx 1 dsl11 multicore 5 Jun 24 02:48
clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/clang++ -> clang

$ ls -l clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/llvm-ranlib
lrwxrwxrwx 1 dsl11 multicore 54 Jul  7 03:53
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/llvm-ranlib ->
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/llvm-ar

$  ls -l clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/llvm-ranlib
lrwxrwxrwx 1 dsl11 multicore 7 Jun 24 02:48
clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/llvm-ranlib -> llvm-ar
```
I think the ``--transform`` line you are passing to tar command might
be doing this.

- There are additional files (Dummy.html and the ps directory). This is minor.

I don't understand why these weird install issues are coming up. Do
the logs ($LogDir/llvm.install-Phase$Phase-$Flavor.log) created by the
``test-release.sh`` script shed any light?

I'm going to try building LLVM in a clean Docker container when I get
home to see if I can reproduce what you're seeing.

On Tuesday, July 07, 2015 12:13 PM, Dan Liew wrote:
> On 6 July 2015 at 20:29, Ben Pope <benpope81 at gmail.com> wrote:
>> I've built 3.6.2 final.  I changed the triple as suggested.
>>
>> I've uploaded two archives here:
> <snip>
>
>> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.tar.xz is the one
produced by
>> the release script.
>>
>> To produce
clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.installed.tar.xz I
>> basically did:
>> cd Phase3/Release/llvmCore-3.6.2-final.obj/
>> make install
>> cd ../llvmCore-3.6.2-final.install/
>> tar cavf
>>
../../../clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04.installed.tar.xz
>> --transform
's,^,clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/,' *
>>
>> I hope that helps.
> Thanks.
>
> I diffed the contents of your old tarball looking for files that only
> exist in one of the tarballs.
>
> ```
> $ diff --recursive -q clang+llvm-3.6.2-rc1-x86_64-linux-gnu
> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04 | grep ^Only
> diff: clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/clang++: No
> such file or directory
> diff: clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/llvm-ranlib:
> No such file or directory
> Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/docs/llvm/html:
> Dummy.html
> Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/docs/llvm: ps
> Only in clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/share: llvm
> ```
>
> So I see the missing CMake files are present however there are new issues.
>
> - The symlinks to clang++ and llvm-ranlib are now broken :(
>
> ```
> $ ls -l clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/clang++
> lrwxrwxrwx 1 dsl11 multicore 52 Jul  7 03:54
> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/clang++ ->
> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/clang
>
> $ ls -l clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/clang++
> lrwxrwxrwx 1 dsl11 multicore 5 Jun 24 02:48
> clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/clang++ -> clang
>
> $ ls -l clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/llvm-ranlib
> lrwxrwxrwx 1 dsl11 multicore 54 Jul  7 03:53
> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/bin/llvm-ranlib ->
> clang+llvm-3.6.2-x86_64-linux-gnu-ubuntu-14.04/llvm-ar
>
> $  ls -l clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/llvm-ranlib
> lrwxrwxrwx 1 dsl11 multicore 7 Jun 24 02:48
> clang+llvm-3.6.2-rc1-x86_64-linux-gnu/bin/llvm-ranlib -> llvm-ar
> ```
> I think the ``--transform`` line you are passing to tar command might
> be doing this.
Perhaps, or maybe I need another command line option, it's tar, I'm sure
I forgot a handful.
> - There are additional files (Dummy.html and the ps directory). This is
minor.
>
> I don't understand why these weird install issues are coming up. Do
> the logs ($LogDir/llvm.install-Phase$Phase-$Flavor.log) created by the
> ``test-release.sh`` script shed any light?
I've put them in the same location for you.

Ben