llvm dev - Jul 2015 - [LLVMdev] [3.7 Release] RC1 has been tagged, Testing Phase I begins

On 19 Jul 2015, at 00:17, Dimitry Andric <dimitry at andric.com>
wrote:
> 
> On 17 Jul 2015, at 01:09, Hans Wennborg <hans at chromium.org> wrote:
>> 
>> On Thu, Jul 16, 2015 at 3:47 PM, Dimitry Andric <dimitry at
andric.com> wrote:
>>> On 17 Jul 2015, at 00:31, Hans Wennborg <hans at
chromium.org> wrote:
>>>> 
>>>> Dear testers,
>>>> 
>>>> 3.7.0-rc1 was just tagged; please start your testing engines
:-)
>>>> 
>>>> Upload binaries to the sftp and report your results to this
thread.
>>>> 
>>>> I'm sorry for the delay between branching and tagging. The
changes to
>>>> the release script took a little longer than I hoped.
>>>> 
>>>> Thanks for helping with the release, and do let me know of any
issues,
>>>> questions, etc.
>>>> 
>>>> The tracking bug for release blockers is PR24126.
>>> 
>>> Is it OK to do an autoconf build?  The CMake build tries to build
various components which do not yet work on FreeBSD, e.g. libcxxabi does not
compile at all, libcompiler-rt has a bunch of test failures, etc. 
Alternatively, can I disable these components in the CMake build locally?
>> 
>> Yes, go ahead and use the autoconf build.
>> 
>> Can you send a patch to test-release.sh that makes this default for
>> FreeBSD? It's already the default for Darwin.
> 
> Here it is.  While here, I replaced the multiple calls to uname -s with a
variable assignment.
> 
> It's currently building for FreeBSD 10.x i386 and amd64.
Hm, strangely enough, this version of the script does not go further than the
Phase 2 installation, and does not run any tests?  This used to work fine for
the release_36 branch.

I think it is because of the "set -o pipefail" which was introduced,
but I don't yet understand why this causes the Phase 2 installation to
appear to fail, as there is no visible error.  I will investigate, or work
around it by removing the pipefail option again.

-Dimitry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150719/6d30f466/attachment.sig>

On 19 Jul 2015, at 01:44, Dimitry Andric <dimitry at andric.com> wrote:
...
> Hm, strangely enough, this version of the script does not go further than
the Phase 2 installation, and does not run any tests?  This used to work fine
for the release_36 branch.
> 
> I think it is because of the "set -o pipefail" which was
introduced, but I don't yet understand why this causes the Phase 2
installation to appear to fail, as there is no visible error.  I will
investigate, or work around it by removing the pipefail option again.
It appears to be caused by the clean_RPATH() function, which has this line:

      rpath=`objdump -x $Candidate | grep 'RPATH' | sed -e's/^
*RPATH *//'`

If the objdump'd file does not have any RPATH, the whole statement will
fail, since set -o pipefail is in effect.  This terminates the script, since set
-e is also in effect.

The line can be replaced with this instead:

      rpath=`objdump -x $Candidate | sed -ne'/RPATH/{s/^ *RPATH
*//;p;}'`

-Dimitry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: use-autoconf-on-freebsd-too-2.diff
Type: application/octet-stream
Size: 2074 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150719/f94fbb2a/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150719/f94fbb2a/attachment.sig>

On 19 Jul 2015, at 17:32, Dimitry Andric <dimitry at andric.com>
wrote:
> 
> On 19 Jul 2015, at 01:44, Dimitry Andric <dimitry at andric.com>
wrote:
> ...
>> Hm, strangely enough, this version of the script does not go further
than the Phase 2 installation, and does not run any tests?  This used to work
fine for the release_36 branch.
>> 
>> I think it is because of the "set -o pipefail" which was
introduced, but I don't yet understand why this causes the Phase 2
installation to appear to fail, as there is no visible error.  I will
investigate, or work around it by removing the pipefail option again.
> 
> It appears to be caused by the clean_RPATH() function, which has this line:
> 
>      rpath=`objdump -x $Candidate | grep 'RPATH' | sed -e's/^
*RPATH *//'`
> 
> If the objdump'd file does not have any RPATH, the whole statement will
fail, since set -o pipefail is in effect.  This terminates the script, since set
-e is also in effect.
> 
> The line can be replaced with this instead:
> 
>      rpath=`objdump -x $Candidate | sed -ne'/RPATH/{s/^ *RPATH
*//;p;}'`
Another new problem with the test-release.sh script is that it can cause the
whole machine (!) to run out of memory, when comparing phase 2 with phase 3, due
to the following fragment:

   470          echo "# Comparing Phase 2 and Phase 3 files"
   471          for p2 in `find $llvmCore_phase2_objdir -name '*.o'` ;
do
   472              p3=`echo $p2 | sed -e 's,Phase2,Phase3,'`
   473              # Substitute 'Phase2' for 'Phase3' in the
Phase 2 object file in
   474              # case there are build paths in the debug info. On some
systems,
   475              # sed adds a newline to the output, so pass $p3 through sed
too.
   476              if ! cmp --ignore-initial=16 <(sed -e
's,Phase2,Phase3,g' $p2) \
   477                      <(sed -e '' $p3) > /dev/null
2>&1 ; then
   478                  echo "file `basename $p2` differs between phase 2
and phase 3"
   479              fi
   480          done

Because cmp(1) on FreeBSD does not support the --ignore-initial option, which is
a GNU extension, the command immediately fails.  Then, the <(...) constructs
on lines 476 and 477 spawn two new bash instances per iteration, and these never
get cleaned up, at least not on FreeBSD.  This may very well be a bash bug.

Alternatively, the 'skip' values can be specified as the third and
fourth argument on the cmp(1) command line, and this works on both Linux,
FreeBSD and OSX; e.g. this:

            if ! cmp -s <(sed -e 's,Phase2,Phase3,g' $p2) <(sed -e
'' $p3) \
                    16 16 ; then

I think this is the final adjustment I have to make to get the building and
testing to complete.  Hans, is it OK if I commit these changes to
test-release.sh in trunk?  Then I'll merge them to release_37 afterwards.

-Dimitry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: use-autoconf-on-freebsd-too-3.diff
Type: application/octet-stream
Size: 2477 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150719/c85baa5b/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150719/c85baa5b/attachment.sig>

On Sun, Jul 19, 2015 at 8:32 AM, Dimitry Andric <dimitry at andric.com>
wrote:
> On 19 Jul 2015, at 01:44, Dimitry Andric <dimitry at andric.com>
wrote:
> ...
>> Hm, strangely enough, this version of the script does not go further
than the Phase 2 installation, and does not run any tests?  This used to work
fine for the release_36 branch.
>>
>> I think it is because of the "set -o pipefail" which was
introduced, but I don't yet understand why this causes the Phase 2
installation to appear to fail, as there is no visible error.  I will
investigate, or work around it by removing the pipefail option again.
>
> It appears to be caused by the clean_RPATH() function, which has this line:
>
>       rpath=`objdump -x $Candidate | grep 'RPATH' | sed -e's/^
*RPATH *//'`
>
> If the objdump'd file does not have any RPATH, the whole statement will
fail, since set -o pipefail is in effect.  This terminates the script, since set
-e is also in effect.
Nice find!
>
> The line can be replaced with this instead:
>
>       rpath=`objdump -x $Candidate | sed -ne'/RPATH/{s/^ *RPATH
*//;p;}'`
How about just guarding it with an if statement? I'm worried about
adding non-obvious commands to the script, both for making it harder
to understand and for portability.

Thanks,
Hans