R devel - Jan 2015 - New version of Rtools for Windows

Very timely, as this is how I got into the problem I posted about
earlier; maybe some of the problems I ran into will mean more to the
you and the experts on this thread, Dr. Murdoch.For reference, I run
Windows 7 64bit, and I am trying to build a 64 bit version of R-3.1.2.

As we discussed offline, Dr. Murdoch, I've been trying to build R
using more recent tools than GCC4.6.3 prerelease. Ruben Von Boxen
(rubenvb) told me he is no longer developing his own builds of GCC,
but is focusing on MSYS2 and the mingw64 personal builds. So, similar
to what Jeroen said, I first installed MSYS2, whose initial
installation on windows is not so simple[1]. After the initial
install, the following packages need to be manually installed: make,
tar, zip, unzip, zlib, and rsync. I also installed base-devel, which
is way more than necessary, but there may be packages in there which
are necessary.

I originally installed the most up-to-date version of GCC (4.9.2)[2],
and I did pick the -seh version, as since I install (almost) all
packages from source (the one exception being nloptr for now), the
exception handling should be consistent and it is supposed to up to
~15% faster[3].

The initial build crashed with the following error:

gcc -std=gnu99 -m64 -I../../include -I. -DHAVE_CONFIG_H  -O3 -Wall
-pedantic -mtune=core2   -c xmalloc.c -o xmalloc.o
ar crs libtre.a regcomp.o regerror.o regexec.o tre-ast.o tre-compile.o
tre-match -approx.o tre-match-backtrack.o tre-match-parallel.o
tre-mem.o tre-parse.o tre-stack.o xmalloc.o
gcc -std=gnu99 -m64   -O3 -Wall -pedantic -mtune=core2   -c compat.c -o compat.o
compat.c:65:5: error: redefinition of 'snprintf'
 int snprintf(char *buffer, size_t max, const char *format, ...)
     ^
In file included from compat.c:3:0:
F:/MinGW64/x86_64-w64-mingw32/include/stdio.h:553:5: note: previous
definition of 'snprintf' was here
 int snprintf (char * __restrict__ __stream, size_t __n, const char *
__restrict__ __format, ...)
     ^
compat.c:75:5: error: redefinition of 'vsnprintf'
 int vsnprintf(char *buffer, size_t bufferSize, const char *format,
va_list args)
     ^
In file included from compat.c:3:0:
F:/MinGW64/x86_64-w64-mingw32/include/stdio.h:543:7: note: previous
definition of 'vsnprintf' was here
   int vsnprintf (char * __restrict__ __stream, size_t __n, const char
* __restrict__ __format, va_list __local_argv)
       ^
../../gnuwin32/MkRules:218: recipe for target 'compat.o' failed
make[4]: *** [compat.o] Error 1
Makefile:120: recipe for target 'rlibs' failed
make[3]: *** [rlibs] Error 1
Makefile:179: recipe for target '../../bin/x64/R.dll' failed
make[2]: *** [../../bin/x64/R.dll] Error 2
Makefile:104: recipe for target 'rbuild' failed
make[1]: *** [rbuild] Error 2
Makefile:14: recipe for target 'all' failed
make: *** [all] Error 2

After doing some checking (for example see [4]), I asked Duncan about
the problem, and he suggested moving the #ifndef _W64 in compat.c up
above the offending lines (65-75). That did not work, so, I figured
(it seems mistakenly from the other thread) that if those functions
are included from stdio already, I can just delete them from compat.c.
The specific lines are:

int snprintf(char *buffer, size_t max, const char *format, ...)
{
    int res;
    va_list(ap);
    va_start(ap, format);
    res = trio_vsnprintf(buffer, max, format, ap);
    va_end(ap);
    return res;
}

int vsnprintf(char *buffer, size_t bufferSize, const char *format, va_list args)
{
    return trio_vsnprintf(buffer, bufferSize, format, args);
}

Continuing the build using 4.9.2 crashed again at the following point:

gcc -std=gnu99 -m64 -I../include -I. -I../extra -DHAVE_CONFIG_H
-DR_DLL_BUILD  -O3 -Wall -pedantic -mtune=core2   -c malloc.c -o
malloc.o
windres -F pe-x86-64  -I../include -i dllversion.rc -o dllversion.o
gcc -std=gnu99 -m64 -shared -s -mwindows -o R.dll R.def console.o
dynload.o editor.o embeddedR.o extra.o opt.o pager.o preferences.o
psignal.o rhome.o rt_complete.o rui.o run.o shext.o sys-win32.o
system.o dos_wglob.o malloc.o ../main/libmain.a ../appl/libappl.a
../nmath/libnmath.a getline/gl.a ../extra/xdr/libxdr.a
../extra/pcre/libpcre.a ../extra/bzip2/libbz2.a
../extra/intl/libintl.a ../extra/trio/libtrio.a ../extra/tzone/libtz.a
../extra/tre/libtre.a ../extra/xz/liblzma.a dllversion.o -fopenmp -L.
-lgfortran -lRblas -L../../bin/x64 -lRzlib -lRgraphapp -lRiconv
-lcomctl32 -lversion
collect2.exe: error: ld returned 5 exit status
Makefile:150: recipe for target 'R.dll' failed
make[3]: *** [R.dll] Error 1
Makefile:179: recipe for target '../../bin/x64/R.dll' failed
make[2]: *** [../../bin/x64/R.dll] Error 2
Makefile:104: recipe for target 'rbuild' failed
make[1]: *** [rbuild] Error 2
Makefile:14: recipe for target 'all' failed
make: *** [all] Error 2

As all those files existed in their correct places, the only reason I
could think of that this would fail here is that GCC version 4.9 did
make some changes to enhance link-time optimization [5], and probably
something isn't compatible. I then downgraded to GCC 4.8.4 [6], and,
with the deletion of those 10 or so lines from compat.c, I can
complete the build straight through rinstaller. However, I get that
failure issue due to the extra 0 in scientific notation [7].

It does not matter if I do the entire process in the MSYS2
environment, or if I do in in Windows with msys\usr\bin in my path.

Na?vely, it seems that if there were some what for stdio to be
included in compat.c, yet the versions of snprintf and vsprintf in
that file to "override" the standard, perhaps this method would work.
Of course, running make check-all may uncover more issues. I intend to
run the equivalent checks (from the tools library) inside of R with
kill on failure turned off to see if anything else is problematic.

Hopefully, something in this description resonates with one of the
readers here. If anyone has any ideas as to how to circumvent the
issues with compat.c, I'd be very grateful.

Thank you,

Avi


[1] http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
[2]
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.9.2/threads-win32/seh/x86_64-4.9.2-release-win32-seh-rt_v3-rev1.7z/download
[3]
https://stackoverflow.com/questions/15670169/what-is-difference-between-sjlj-vs-dwarf-vs-seh
[4]
http://www.tt-forums.net/viewtopic.php?p=1034657&sid=613fa47a379ffaa0b9a9fb182a4180e3#p1034657
[5] https://gcc.gnu.org/gcc-4.9/changes.html
[6]
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.4/threads-win32/seh/x86_64-4.8.4-release-win32-seh-rt_v3-rev0.7z/download
[7] https://stat.ethz.ch/pipermail/r-devel/2015-January/070354.html

Date: Wed, 07 Jan 2015 20:31:07 -0500
>
> From: Duncan Murdoch <murdoch.duncan at gmail.com>
> To: Jeroen Ooms <jeroenooms at gmail.com>
> Cc: "R-devel at r-project.org" <r-devel at r-project.org>
> Subject: Re: [Rd] New version of Rtools for Windows
> Message-ID: <54ADDDDB.4020500 at gmail.com>
> Content-Type: text/plain; charset=utf-8
>
> On 07/01/2015 5:20 PM, Jeroen Ooms wrote:
> > On Wed, Jan 7, 2015 at 8:00 AM, Duncan Murdoch <murdoch.duncan at
gmail.com> wrote:
> >>
> >> This version includes only minor updates to the tools.  I
indicated last summer that I was hoping to update GCC from the current version
4.6.3 before the R 3.2.0 release, but this now looks unlikely, unless someone
else with experience building it can help.
> >
> > I have been looking into this a bit over the past few months, also
> > with mixed success. Nevertheless, below some experiences that might be
> > worth sharing.
> >
> > The guys from mingw-w64 recommended (quite strongly) to move away from
> > multilib. They explained that the standard approach is to create two
> > separate toolchains; one that targets win32 and the other one that
> > targets win64 (both tool chains can compiled for win32). Hence the
> > only difference for R would be that instead of passing
"-m64" and
> > "-m32", it would need to set the path to the proper
compiler.
> >
> > There are several initiatives that provide very complete suites of
> > precompiled mingw-w64 tools. I think the ideal scenario would be if we
> > could take advantage of an existing tool chain as we do on other
> > platforms, although perhaps I do not fully understand the R-specific
> > requirements on the windows compiler.
>
> I feel quite strongly that we need to be able to build the toolchain,
> rather than relying on binaries produced by others.  We may need to
> customize the toolchain, or we may need to rebuild it when a bug is
> identified.  Lots of binary builders abandon their builds and you can't
> count on them to solve problems at a later date.
>
> >
> > One project that looks very promising is msys2 [1,2]. It has a package
> > manager (port of pacman from arch linux) and comes with a pretty
> > complete set of msys [3] and other [4] packages that seems quite well
> > maintained.
>
> Do they post complete instructions for building?  That's what I'm
> looking for.  I don't want to develop a build script (I don't know
how),
> but I would like to have one.
>
> Duncan Murdoch
>
> >
> > The only issue I ran into with msys2 is that it uses a different c++
> > exception model (seh/dwarf) than the current Rtools (which uses sjlj).
> > See also [5]. Therefore, if a library uses exceptions, we cannot use
> > the current Rtools to link a static library that was created with
> > msys2  [6]. I am not sure if it also be a problem the other way
> > around, and if this is still the case for recent versions of
> > gcc/mingw.
> >
> > Finally, Ruby has build very similar to Rtools called DevKit-mingw64
> > [7] that we might be able to borrow from.
> >
> >
> > [1] https://msys2.github.io/
> > [2]
http://stackoverflow.com/questions/25019057/how-are-msys-msys2-and-msysgit-related-to-each-other
> > [3] https://github.com/Alexpux/MSYS2-packages
> > [4] https://github.com/Alexpux/MINGW-packages
> > [5]
http://stackoverflow.com/questions/15670169/what-is-difference-between-sjlj-vs-dwarf-vs-seh
> > [6]
http://stackoverflow.com/questions/7751640/undefined-reference-to-gxx-personality-sj0
> > [7] http://rubyinstaller.org/downloads/

On 2015-01-08 14:18, Avraham Adler wrote:
> Very timely, as this is how I got into the problem I posted about
> earlier; maybe some of the problems I ran into will mean more to the
> you and the experts on this thread, Dr. Murdoch.For reference, I run
> Windows 7 64bit, and I am trying to build a 64 bit version of R-3.1.2.
>
> As we discussed offline, Dr. Murdoch, I've been trying to build R
> using more recent tools than GCC4.6.3 prerelease. Ruben Von Boxen
> (rubenvb) told me he is no longer developing his own builds of GCC,
> but is focusing on MSYS2 and the mingw64 personal builds. So, similar
> to what Jeroen said, I first installed MSYS2, whose initial
> installation on windows is not so simple[1]. After the initial
> install, the following packages need to be manually installed: make,
> tar, zip, unzip, zlib, and rsync. I also installed base-devel, which
> is way more than necessary, but there may be packages in there which
> are necessary.
>
> I originally installed the most up-to-date version of GCC (4.9.2)[2],
> and I did pick the -seh version, as since I install (almost) all
> packages from source (the one exception being nloptr for now), the
> exception handling should be consistent and it is supposed to up to
> ~15% faster[3].
>
> The initial build crashed with the following error:
>
> gcc -std=gnu99 -m64 -I../../include -I. -DHAVE_CONFIG_H  -O3 -Wall
> -pedantic -mtune=core2   -c xmalloc.c -o xmalloc.o
> ar crs libtre.a regcomp.o regerror.o regexec.o tre-ast.o tre-compile.o
> tre-match -approx.o tre-match-backtrack.o tre-match-parallel.o
> tre-mem.o tre-parse.o tre-stack.o xmalloc.o
> gcc -std=gnu99 -m64   -O3 -Wall -pedantic -mtune=core2   -c compat.c -o
compat.o
> compat.c:65:5: error: redefinition of 'snprintf'
>   int snprintf(char *buffer, size_t max, const char *format, ...)
>       ^
> In file included from compat.c:3:0:
> F:/MinGW64/x86_64-w64-mingw32/include/stdio.h:553:5: note: previous
> definition of 'snprintf' was here
>   int snprintf (char * __restrict__ __stream, size_t __n, const char *
> __restrict__ __format, ...)
>       ^
> compat.c:75:5: error: redefinition of 'vsnprintf'
>   int vsnprintf(char *buffer, size_t bufferSize, const char *format,
> va_list args)
>       ^
> In file included from compat.c:3:0:
> F:/MinGW64/x86_64-w64-mingw32/include/stdio.h:543:7: note: previous
> definition of 'vsnprintf' was here
>     int vsnprintf (char * __restrict__ __stream, size_t __n, const char
> * __restrict__ __format, va_list __local_argv)
>         ^
> ../../gnuwin32/MkRules:218: recipe for target 'compat.o' failed
> make[4]: *** [compat.o] Error 1
> Makefile:120: recipe for target 'rlibs' failed
> make[3]: *** [rlibs] Error 1
> Makefile:179: recipe for target '../../bin/x64/R.dll' failed
> make[2]: *** [../../bin/x64/R.dll] Error 2
> Makefile:104: recipe for target 'rbuild' failed
> make[1]: *** [rbuild] Error 2
> Makefile:14: recipe for target 'all' failed
> make: *** [all] Error 2
>
> After doing some checking (for example see [4]), I asked Duncan about
> the problem, and he suggested moving the #ifndef _W64 in compat.c up
> above the offending lines (65-75). That did not work, so, I figured
> (it seems mistakenly from the other thread) that if those functions
> are included from stdio already, I can just delete them from compat.c.
> The specific lines are:
>
> int snprintf(char *buffer, size_t max, const char *format, ...)
> {
>      int res;
>      va_list(ap);
>      va_start(ap, format);
>      res = trio_vsnprintf(buffer, max, format, ap);
>      va_end(ap);
>      return res;
> }
>
> int vsnprintf(char *buffer, size_t bufferSize, const char *format, va_list
args)
> {
>      return trio_vsnprintf(buffer, bufferSize, format, args);
> }
>
> Continuing the build using 4.9.2 crashed again at the following point:
>
> gcc -std=gnu99 -m64 -I../include -I. -I../extra -DHAVE_CONFIG_H
> -DR_DLL_BUILD  -O3 -Wall -pedantic -mtune=core2   -c malloc.c -o
> malloc.o
> windres -F pe-x86-64  -I../include -i dllversion.rc -o dllversion.o
> gcc -std=gnu99 -m64 -shared -s -mwindows -o R.dll R.def console.o
> dynload.o editor.o embeddedR.o extra.o opt.o pager.o preferences.o
> psignal.o rhome.o rt_complete.o rui.o run.o shext.o sys-win32.o
> system.o dos_wglob.o malloc.o ../main/libmain.a ../appl/libappl.a
> ../nmath/libnmath.a getline/gl.a ../extra/xdr/libxdr.a
> ../extra/pcre/libpcre.a ../extra/bzip2/libbz2.a
> ../extra/intl/libintl.a ../extra/trio/libtrio.a ../extra/tzone/libtz.a
> ../extra/tre/libtre.a ../extra/xz/liblzma.a dllversion.o -fopenmp -L.
> -lgfortran -lRblas -L../../bin/x64 -lRzlib -lRgraphapp -lRiconv
> -lcomctl32 -lversion
> collect2.exe: error: ld returned 5 exit status
> Makefile:150: recipe for target 'R.dll' failed
> make[3]: *** [R.dll] Error 1
> Makefile:179: recipe for target '../../bin/x64/R.dll' failed
> make[2]: *** [../../bin/x64/R.dll] Error 2
> Makefile:104: recipe for target 'rbuild' failed
> make[1]: *** [rbuild] Error 2
> Makefile:14: recipe for target 'all' failed
> make: *** [all] Error 2
>
> As all those files existed in their correct places, the only reason I
> could think of that this would fail here is that GCC version 4.9 did
> make some changes to enhance link-time optimization [5], and probably
> something isn't compatible.
Right.  Just before Christmas, Hin-Tak Leung reported build failure with 
LTO:

https://stat.ethz.ch/pipermail/r-devel/2014-December/070286.html
https://stat.ethz.ch/pipermail/r-devel/2014-December/070319.html


Many thanks to you and others for looking into this,
Henric


> I then downgraded to GCC 4.8.4 [6], and,
> with the deletion of those 10 or so lines from compat.c, I can
> complete the build straight through rinstaller. However, I get that
> failure issue due to the extra 0 in scientific notation [7].
>
> It does not matter if I do the entire process in the MSYS2
> environment, or if I do in in Windows with msys\usr\bin in my path.
>
> Na?vely, it seems that if there were some what for stdio to be
> included in compat.c, yet the versions of snprintf and vsprintf in
> that file to "override" the standard, perhaps this method would
work.
> Of course, running make check-all may uncover more issues. I intend to
> run the equivalent checks (from the tools library) inside of R with
> kill on failure turned off to see if anything else is problematic.
>
> Hopefully, something in this description resonates with one of the
> readers here. If anyone has any ideas as to how to circumvent the
> issues with compat.c, I'd be very grateful.
>
> Thank you,
>
> Avi
>
>
> [1] http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
> [2]
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.9.2/threads-win32/seh/x86_64-4.9.2-release-win32-seh-rt_v3-rev1.7z/download
> [3]
https://stackoverflow.com/questions/15670169/what-is-difference-between-sjlj-vs-dwarf-vs-seh
> [4]
http://www.tt-forums.net/viewtopic.php?p=1034657&sid=613fa47a379ffaa0b9a9fb182a4180e3#p1034657
> [5] https://gcc.gnu.org/gcc-4.9/changes.html
> [6]
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.4/threads-win32/seh/x86_64-4.8.4-release-win32-seh-rt_v3-rev0.7z/download
> [7] https://stat.ethz.ch/pipermail/r-devel/2015-January/070354.html
>
> Date: Wed, 07 Jan 2015 20:31:07 -0500
>>
>> From: Duncan Murdoch <murdoch.duncan at gmail.com>
>> To: Jeroen Ooms <jeroenooms at gmail.com>
>> Cc: "R-devel at r-project.org" <r-devel at
r-project.org>
>> Subject: Re: [Rd] New version of Rtools for Windows
>> Message-ID: <54ADDDDB.4020500 at gmail.com>
>> Content-Type: text/plain; charset=utf-8
>>
>> On 07/01/2015 5:20 PM, Jeroen Ooms wrote:
>>> On Wed, Jan 7, 2015 at 8:00 AM, Duncan Murdoch <murdoch.duncan
at gmail.com> wrote:
>>>>
>>>> This version includes only minor updates to the tools.  I
indicated last summer that I was hoping to update GCC from the current version
4.6.3 before the R 3.2.0 release, but this now looks unlikely, unless someone
else with experience building it can help.
>>>
>>> I have been looking into this a bit over the past few months, also
>>> with mixed success. Nevertheless, below some experiences that might
be
>>> worth sharing.
>>>
>>> The guys from mingw-w64 recommended (quite strongly) to move away
from
>>> multilib. They explained that the standard approach is to create
two
>>> separate toolchains; one that targets win32 and the other one that
>>> targets win64 (both tool chains can compiled for win32). Hence the
>>> only difference for R would be that instead of passing
"-m64" and
>>> "-m32", it would need to set the path to the proper
compiler.
>>>
>>> There are several initiatives that provide very complete suites of
>>> precompiled mingw-w64 tools. I think the ideal scenario would be if
we
>>> could take advantage of an existing tool chain as we do on other
>>> platforms, although perhaps I do not fully understand the
R-specific
>>> requirements on the windows compiler.
>>
>> I feel quite strongly that we need to be able to build the toolchain,
>> rather than relying on binaries produced by others.  We may need to
>> customize the toolchain, or we may need to rebuild it when a bug is
>> identified.  Lots of binary builders abandon their builds and you
can't
>> count on them to solve problems at a later date.
>>
>>>
>>> One project that looks very promising is msys2 [1,2]. It has a
package
>>> manager (port of pacman from arch linux) and comes with a pretty
>>> complete set of msys [3] and other [4] packages that seems quite
well
>>> maintained.
>>
>> Do they post complete instructions for building?  That's what
I'm
>> looking for.  I don't want to develop a build script (I don't
know how),
>> but I would like to have one.
>>
>> Duncan Murdoch
>>
>>>
>>> The only issue I ran into with msys2 is that it uses a different
c++
>>> exception model (seh/dwarf) than the current Rtools (which uses
sjlj).
>>> See also [5]. Therefore, if a library uses exceptions, we cannot
use
>>> the current Rtools to link a static library that was created with
>>> msys2  [6]. I am not sure if it also be a problem the other way
>>> around, and if this is still the case for recent versions of
>>> gcc/mingw.
>>>
>>> Finally, Ruby has build very similar to Rtools called
DevKit-mingw64
>>> [7] that we might be able to borrow from.
>>>
>>>
>>> [1] https://msys2.github.io/
>>> [2]
http://stackoverflow.com/questions/25019057/how-are-msys-msys2-and-msysgit-related-to-each-other
>>> [3] https://github.com/Alexpux/MSYS2-packages
>>> [4] https://github.com/Alexpux/MINGW-packages
>>> [5]
http://stackoverflow.com/questions/15670169/what-is-difference-between-sjlj-vs-dwarf-vs-seh
>>> [6]
http://stackoverflow.com/questions/7751640/undefined-reference-to-gxx-personality-sj0
>>> [7] http://rubyinstaller.org/downloads/
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>