R devel - Mar 2015 - Notes on building a gcc toolchain for Rtools (but not multilib)

Hi,

[This is a follow-up to the "New version of Rtools for Windows" thread
in January, but I just subscribed and don't know how to reply to an
old thread -- my apologies.]

I was able to use the nuwen distro to build a gcc 4.9.2 toolchain and
use it to build the latest R-patched with it.

Below are some notes about what I did; I hope they will be useful for
keeping Rtools up-to-date.


Note:

- This is 64-bit only; I tried but was unable to create a multilib toolchain

- I did not run any tests on the resulting R binary, other than
starting R and running some basic commands

- I don't necessarily know what I'm doing!


Outline of steps:

- The (nicely done!) nuwen website by Stephan Lavavej has made
available a MinGW distro and the scripts used to create it.

  + However, the gcc toolchain there is built with --disable-lib32 (so
no multilib) and with --disable-gomp (the default)

  + Moreover, the pthreads library is not included in the distro

  + I believe building R requires GOMP and pthreads, hence I tried to
modify the scripts to add these

- Installing your own toolchain

  + Read the instructions in the section "How To Build Your Own
Distro"

  + You don't have to rebuild everything in components-12.2.7z: you
only need the original binutils-2.25.7z and your own build of
mingw-w64+gcc.7z to replace Rtools's gcc

  + You will need to run a modified version of Stephan's
mingw-w64+gcc.sh script.  Besides the gcc source code, you will also
need to download the pthreads-w32 source code from
https://www.sourceware.org/pthreads-win32/

  + Here are the changes I made to the mingw-w64+gcc.sh script:

--------------------------------------------------

diff --git a/mingw-w64+gcc.sh b/mingw-w64+gcc.sh
index 2402ffc..fd44e76 100644
--- a/mingw-w64+gcc.sh
+++ b/mingw-w64+gcc.sh
@@ -8,6 +8,7 @@ source 0_append_distro_path.sh
 7z x '-oC:\Temp\gcc' gmp-6.0.0a.tar > NUL || fail_with
gmp-6.0.0a.tar
- EPIC FAIL
 7z x '-oC:\Temp\gcc' mpfr-3.1.2.tar > NUL || fail_with
mpfr-3.1.2.tar
- EPIC FAIL
 7z x '-oC:\Temp\gcc' mpc-1.0.2.tar > NUL || fail_with mpc-1.0.2.tar
- EPIC FAIL
+7z x '-oC:\Temp\gcc' pthreads-w32-2-9-1-release.tar > NUL ||
fail_with pthreads-w32-2-9-1-release.tar - EPIC FAIL

 patch -Z -d /c/temp/gcc/mpfr-3.1.2 -p1 < mpfr.patch

@@ -25,6 +26,14 @@ make all install "CFLAGS=-s -O3" || fail_with
mingw-w64 make - EPIC FAIL
 cd /c/temp/gcc
 rm -rf build src

+# Build pthreads-w32.
+cd pthreads-w32-2-9-1-release
+make clean GC
+
+cp libpthreadGC2.a ../dest/x86_64-w64-mingw32/lib/libpthread.a
+cp pthread.h sched.h semaphore.h ../dest/x86_64-w64-mingw32/include
+cp pthreadGC2.dll $X_DISTRO_ROOT/bin/
+
 # Prepare to build gcc - set up the in-tree builds of gmp, mpfr, and mpc.
 mv gcc-4.9.2 src
 mv gmp-6.0.0 src/gmp
@@ -40,7 +49,7 @@ cp -r dest/x86_64-w64-mingw32/include
src/gcc/winsup/mingw/include
 # Configure.
 mkdir build
 cd build
-../src/configure --enable-languages=c,c++ --build=x86_64-w64-mingw32
--host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
--disable-multilib --prefix=/c/temp/gcc/dest
--with-sysroot=/c/temp/gcc/dest --disable-libstdcxx-pch --disable-lto
--disable-nls --disable-shared --disable-win32-registry
--enable-checking=release --with-tune=haswell || fail_with gcc
configure - EPIC FAIL
+../src/configure --enable-languages=c,c++,fortran
--build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
--target=x86_64-w64-mingw32 --disable-multilib
--prefix=/c/temp/gcc/dest --with-sysroot=/c/temp/gcc/dest
--disable-libstdcxx-pch --disable-lto --disable-nls --disable-shared
--disable-win32-registry --enable-libgomp --enable-checking=release
--with-tune=haswell || fail_with gcc configure - EPIC FAIL

 # --enable-languages=c,c++        : I want C and C++ only.
 # --build=x86_64-w64-mingw32      : I want a native compiler.

--------------------------------------------------

  + After running this script, you'll get your own mingw-w64+gcc.7z.

    - You'll need some patience -- the full 3-stage bootstrap of gcc
took 10 hours for me

    - You can replace the "make bootstrap" in the script by "make
bootstrap2" to omit the last stage.

  + Install your toolchain:

    - If you haven't done so, install Rtools.  We will still use the
command line utilities in C:\Rtools\bin

    - Make a new directory, e.g., C:\Rtools\nuwen and unpack
binutils-2.25.7z and mingw-w64+gcc.7z there

    - Put the above directory in your PATH, in front of any other
toolchain locations such as C:\Rtools\gcc-4.6.3\bin (it may be better
to remove the latter from your PATH)

- Now for the installation of R:

  + Get the latest R-patched sources (rather than 3.1.2, because
Duncan (thanks!) have added some useful changes)

  + Make sure you have the prerequisites for building R (see the R
Installation and Administration Manual)

    - In particular, the source files for the recommended packages,
the support files for Tcl, and the "extsoft" headers and libraries

  + In src/gnuwin32, copy MkRules.dist to MkRules.local and apply the
following patch:

--------------------------------------------------

--- MkRules.local 2015-03-02 13:57:38.601903500 -0800
+++ MkRules.local 2015-03-06 10:43:44.708581800 -0800
@@ -52,6 +52,7 @@
 # BINPREF  # prefix for 64-bit: path or x86_64-w64-mingw32-
 # BINPREF64 = x86_64-w64-mingw32-
+BINPREF64 
 # Others use a -m64 or -m32 option to select architectures
 # M_ARCH @@ -64,6 +65,7 @@

 # 32- or 64-bit Windows?
 # WIN = 32
+WIN = 64

 # The gcc 4.9.2 64 bit toolchain is set up for the 'medium code'
model and needs
 # to remove the .refptr entries from the exports list; this is the default
@@ -135,12 +137,12 @@
 # Full paths of extra DLLs that need to be shipped
 # e.g
 # DLLs32 = c:/R/bin/pthreadGC2-w32.dll
-# DLLs64 = c:/R/bin64/pthreadGC2-w64.dll
+DLLs64 = c:/MinGW/bin/pthreadGC2.dll
 # DLLs32  # DLLs64 
 # Define this to 1 if using the gcc 4.9.2 toolchain with dynamic linking
-# COPY_RUNTIME_DLLS +COPY_RUNTIME_DLLS = 1

 ## ====== configuration macros for building MSI installer ==========
--------------------------------------------------

  + (I don't know whether the DLLs64 and COPY_RUNTIME_DLLS changes
above are necessary or not.  C:/MinGW/bin is where I put the pthreads
DLL that was built earlier)

  + Run "make all recommended".  If this works, you should have a
working R, built using your new toolchain!



Additional notes:

- I tried using my new R to install Rcpp from source, but this failed
because the R build scripts was not able to determine the right set of
symbols to be exported in the Rcpp DLL.  To solve this, edit the file
etc/x64/Makeconf under your R source tree, replacing

NM_FILTER = | sed -e '/.refptr./d'

by

NM_FILTER = | sed -e '/.refptr./d; /\.weak\./d'


Hope this helps,

- Hsiu-Khuern

On Sun, Mar 8, 2015 at 10:02 PM, Hsiu-Khuern Tang <tangoh at gmail.com>
wrote:
> Hi,
>
> [This is a follow-up to the "New version of Rtools for Windows"
thread
> in January, but I just subscribed and don't know how to reply to an
> old thread -- my apologies.]
>
> I was able to use the nuwen distro to build a gcc 4.9.2 toolchain and
> use it to build the latest R-patched with it.
>
> Below are some notes about what I did; I hope they will be useful for
> keeping Rtools up-to-date.
>
>
> Note:
>
> - This is 64-bit only; I tried but was unable to create a multilib
toolchain
>
> - I did not run any tests on the resulting R binary, other than
> starting R and running some basic commands
>
> - I don't necessarily know what I'm doing!
>
Thank you for the  update, Hsiu-Khuern. Can you check how many digits
scientific notation shows? I was able to build a 64bit version of R
using the mingw64 4.8.4 distro, but ran into trouble with compat.c,
and the only work-around I found ended breaking R's defualt two digit
notation and used Windows's three-digit notation; it seems as if you
did not have this problem. Also, do you plan on running 'make check'
eventually on your build?

Avi

On Sun, Mar 8, 2015 at 8:33 PM, Avraham Adler <avraham.adler at gmail.com>
wrote:
>
> Thank you for the  update, Hsiu-Khuern. Can you check how many digits
> scientific notation shows? I was able to build a 64bit version of R
> using the mingw64 4.8.4 distro, but ran into trouble with compat.c,
> and the only work-around I found ended breaking R's defualt two digit
> notation and used Windows's three-digit notation; it seems as if you
> did not have this problem.
I get notation like "1e-09".  I think (but am not sure) that there
were some changes in mingw's stdio.h that got rid of the errors about
redefined *sprintf functions.
> Also, do you plan on running 'make check'
> eventually on your build?
I don't have any plans to do that, mainly for fear that I would find
some problems and be sucked down the rabbit hole of trying to solve
them!  Hoping someone else more knowledgeable will take the plunge :)
>
> Avi
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> Hi,
> 
> [This is a follow-up to the "New version of Rtools for Windows"
thread
> in January, but I just subscribed and don't know how to reply to an
> old thread -- my apologies.]
I am planning to put a new Rtools online today that uses a different
build of gcc 4.9.2.  I will be concentrating on getting it to work with
all the external libraries before the 3.2.0 release next month.  I'm not
planning to try to get it to work with R-patched, and I expect it won't:
 I needed to make a number of patches to R-devel for compatibility.

It is also not multilib.  It uses static linking of libraries, and SJLJ
exception handling in both 32 and 64 bit builds -- this is necessary for
compatibility with existing compiles of external libraries, but may not
be ideal in 64 bits.  However, I believe using SEH will mean static
linking is impossible, and that leads to problems distributing run-time
DLLs, and I'd rather not worry about those.

Testing of the new Rtools will be appreciated.  I expect the R-devel
builds will start using it tomorrow or the next day.

Duncan Murdoch
> 
> I was able to use the nuwen distro to build a gcc 4.9.2 toolchain and
> use it to build the latest R-patched with it.
> 
> Below are some notes about what I did; I hope they will be useful for
> keeping Rtools up-to-date.
> 
> 
> Note:
> 
> - This is 64-bit only; I tried but was unable to create a multilib
toolchain
> 
> - I did not run any tests on the resulting R binary, other than
> starting R and running some basic commands
> 
> - I don't necessarily know what I'm doing!
> 
> 
> Outline of steps:
> 
> - The (nicely done!) nuwen website by Stephan Lavavej has made
> available a MinGW distro and the scripts used to create it.
> 
>   + However, the gcc toolchain there is built with --disable-lib32 (so
> no multilib) and with --disable-gomp (the default)
> 
>   + Moreover, the pthreads library is not included in the distro
> 
>   + I believe building R requires GOMP and pthreads, hence I tried to
> modify the scripts to add these
> 
> - Installing your own toolchain
> 
>   + Read the instructions in the section "How To Build Your Own
Distro"
> 
>   + You don't have to rebuild everything in components-12.2.7z: you
> only need the original binutils-2.25.7z and your own build of
> mingw-w64+gcc.7z to replace Rtools's gcc
> 
>   + You will need to run a modified version of Stephan's
> mingw-w64+gcc.sh script.  Besides the gcc source code, you will also
> need to download the pthreads-w32 source code from
> https://www.sourceware.org/pthreads-win32/
> 
>   + Here are the changes I made to the mingw-w64+gcc.sh script:
> 
> --------------------------------------------------
> 
> diff --git a/mingw-w64+gcc.sh b/mingw-w64+gcc.sh
> index 2402ffc..fd44e76 100644
> --- a/mingw-w64+gcc.sh
> +++ b/mingw-w64+gcc.sh
> @@ -8,6 +8,7 @@ source 0_append_distro_path.sh
>  7z x '-oC:\Temp\gcc' gmp-6.0.0a.tar > NUL || fail_with
gmp-6.0.0a.tar
> - EPIC FAIL
>  7z x '-oC:\Temp\gcc' mpfr-3.1.2.tar > NUL || fail_with
mpfr-3.1.2.tar
> - EPIC FAIL
>  7z x '-oC:\Temp\gcc' mpc-1.0.2.tar > NUL || fail_with
mpc-1.0.2.tar - EPIC FAIL
> +7z x '-oC:\Temp\gcc' pthreads-w32-2-9-1-release.tar > NUL ||
> fail_with pthreads-w32-2-9-1-release.tar - EPIC FAIL
> 
>  patch -Z -d /c/temp/gcc/mpfr-3.1.2 -p1 < mpfr.patch
> 
> @@ -25,6 +26,14 @@ make all install "CFLAGS=-s -O3" || fail_with
> mingw-w64 make - EPIC FAIL
>  cd /c/temp/gcc
>  rm -rf build src
> 
> +# Build pthreads-w32.
> +cd pthreads-w32-2-9-1-release
> +make clean GC
> +
> +cp libpthreadGC2.a ../dest/x86_64-w64-mingw32/lib/libpthread.a
> +cp pthread.h sched.h semaphore.h ../dest/x86_64-w64-mingw32/include
> +cp pthreadGC2.dll $X_DISTRO_ROOT/bin/
> +
>  # Prepare to build gcc - set up the in-tree builds of gmp, mpfr, and mpc.
>  mv gcc-4.9.2 src
>  mv gmp-6.0.0 src/gmp
> @@ -40,7 +49,7 @@ cp -r dest/x86_64-w64-mingw32/include
> src/gcc/winsup/mingw/include
>  # Configure.
>  mkdir build
>  cd build
> -../src/configure --enable-languages=c,c++ --build=x86_64-w64-mingw32
> --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
> --disable-multilib --prefix=/c/temp/gcc/dest
> --with-sysroot=/c/temp/gcc/dest --disable-libstdcxx-pch --disable-lto
> --disable-nls --disable-shared --disable-win32-registry
> --enable-checking=release --with-tune=haswell || fail_with gcc
> configure - EPIC FAIL
> +../src/configure --enable-languages=c,c++,fortran
> --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
> --target=x86_64-w64-mingw32 --disable-multilib
> --prefix=/c/temp/gcc/dest --with-sysroot=/c/temp/gcc/dest
> --disable-libstdcxx-pch --disable-lto --disable-nls --disable-shared
> --disable-win32-registry --enable-libgomp --enable-checking=release
> --with-tune=haswell || fail_with gcc configure - EPIC FAIL
> 
>  # --enable-languages=c,c++        : I want C and C++ only.
>  # --build=x86_64-w64-mingw32      : I want a native compiler.
> 
> --------------------------------------------------
> 
>   + After running this script, you'll get your own mingw-w64+gcc.7z.
> 
>     - You'll need some patience -- the full 3-stage bootstrap of gcc
> took 10 hours for me
> 
>     - You can replace the "make bootstrap" in the script by
"make
> bootstrap2" to omit the last stage.
> 
>   + Install your toolchain:
> 
>     - If you haven't done so, install Rtools.  We will still use the
> command line utilities in C:\Rtools\bin
> 
>     - Make a new directory, e.g., C:\Rtools\nuwen and unpack
> binutils-2.25.7z and mingw-w64+gcc.7z there
> 
>     - Put the above directory in your PATH, in front of any other
> toolchain locations such as C:\Rtools\gcc-4.6.3\bin (it may be better
> to remove the latter from your PATH)
> 
> - Now for the installation of R:
> 
>   + Get the latest R-patched sources (rather than 3.1.2, because
> Duncan (thanks!) have added some useful changes)
> 
>   + Make sure you have the prerequisites for building R (see the R
> Installation and Administration Manual)
> 
>     - In particular, the source files for the recommended packages,
> the support files for Tcl, and the "extsoft" headers and
libraries
> 
>   + In src/gnuwin32, copy MkRules.dist to MkRules.local and apply the
> following patch:
> 
> --------------------------------------------------
> 
> --- MkRules.local 2015-03-02 13:57:38.601903500 -0800
> +++ MkRules.local 2015-03-06 10:43:44.708581800 -0800
> @@ -52,6 +52,7 @@
>  # BINPREF >  # prefix for 64-bit: path or x86_64-w64-mingw32-
>  # BINPREF64 = x86_64-w64-mingw32-
> +BINPREF64 > 
>  # Others use a -m64 or -m32 option to select architectures
>  # M_ARCH > @@ -64,6 +65,7 @@
> 
>  # 32- or 64-bit Windows?
>  # WIN = 32
> +WIN = 64
> 
>  # The gcc 4.9.2 64 bit toolchain is set up for the 'medium code'
> model and needs
>  # to remove the .refptr entries from the exports list; this is the default
> @@ -135,12 +137,12 @@
>  # Full paths of extra DLLs that need to be shipped
>  # e.g
>  # DLLs32 = c:/R/bin/pthreadGC2-w32.dll
> -# DLLs64 = c:/R/bin64/pthreadGC2-w64.dll
> +DLLs64 = c:/MinGW/bin/pthreadGC2.dll
>  # DLLs32 >  # DLLs64 > 
>  # Define this to 1 if using the gcc 4.9.2 toolchain with dynamic linking
> -# COPY_RUNTIME_DLLS > +COPY_RUNTIME_DLLS = 1
> 
>  ## ====== configuration macros for building MSI installer ==========> 
> --------------------------------------------------
> 
>   + (I don't know whether the DLLs64 and COPY_RUNTIME_DLLS changes
> above are necessary or not.  C:/MinGW/bin is where I put the pthreads
> DLL that was built earlier)
> 
>   + Run "make all recommended".  If this works, you should have a
> working R, built using your new toolchain!
> 
> 
> 
> Additional notes:
> 
> - I tried using my new R to install Rcpp from source, but this failed
> because the R build scripts was not able to determine the right set of
> symbols to be exported in the Rcpp DLL.  To solve this, edit the file
> etc/x64/Makeconf under your R source tree, replacing
> 
> NM_FILTER = | sed -e '/.refptr./d'
> 
> by
> 
> NM_FILTER = | sed -e '/.refptr./d; /\.weak\./d'
> 
> 
> Hope this helps,
> 
> - Hsiu-Khuern
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <murdoch.duncan at
gmail.com> wrote:
> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>> Hi,
>>
>> [This is a follow-up to the "New version of Rtools for
Windows" thread
>> in January, but I just subscribed and don't know how to reply to an
>> old thread -- my apologies.]
>
> I am planning to put a new Rtools online today that uses a different
> build of gcc 4.9.2.  I will be concentrating on getting it to work with
> all the external libraries before the 3.2.0 release next month.  I'm
not
> planning to try to get it to work with R-patched, and I expect it
won't:
>  I needed to make a number of patches to R-devel for compatibility.
I also worked off R-devel (I said wrongly that it was R-patched in my
original post) and benefited from your compatibility changes.

I look forward to the new Rtools and will test it by compiling some packages.

- Hsiu-Khuern

On Mon, Mar 9, 2015 at 6:50 AM, Duncan Murdoch <murdoch.duncan at
gmail.com> wrote:
> It is also not multilib.  It uses static linking of libraries, and SJLJ
> exception handling in both 32 and 64 bit builds -- this is necessary for
> compatibility with existing compiles of external libraries, but may not
> be ideal in 64 bits.  However, I believe using SEH will mean static
> linking is impossible, and that leads to problems distributing run-time
> DLLs, and I'd rather not worry about those.
That is great news, Dr. Murdoch.

I know that ICU, curl, and the files in goodies (lzma, jpeg ,etc.) are
now external libraries necessary for compiling R. Is that all, or are
there (many) others of which I am unaware? I understand that given the
release date of 3.2.0, the focus needs to be on having a functional
toolset and all that entails, and if that means sjlj is used for now,
that is perfectly reasonable.

However, if the number of such external libraries is manageable, would
it make sense to recompile /them/ using an seh toolchain to take
advantage of the increased performance?

Thank you,

Avi