R devel - Mar 2017 - Seeking advice regarding compilation of large libraries using RTools (Windows)

Hello,

I am working on the SimpleITK package for R. This is an enormous package
that is largely automatically generated via a set of swig/json/lua magic,
and is working well under linux and osx. However we're having a lot of
trouble with the Windows side. In fact, we are struggling to get the base
libraries to build using the RTools 3.4 toolchain, even before the worrying
about the R-specific parts.

I'm hoping someone has observed and solved this problem.

The current issue is very long time (possibly infinite) linking of dlls, or
test executables. I've tried using a FAT32 file system for the build, as
suggested by some old bug reports, but still have the issue.

The build system is CMake-based, and we have the same issues with either
unix makefiles or ninja. The current build host is windows 2012 server, but
we've had the same problems on windows 7 and 10.

We're using the -Wa -mbig-obj flags.

One of the problem commands looks like this:

cd D:/B/SimpleITK-build/Code/Explicit/src &&
C:/Rtools/mingw_64/bin/g++.exe
-Wno-long-long -Wno-unused-local-typedefs -Wno-strict-overflow -Wextra
-Wformat=2 -Wno-format-nonliteral -Wunused -Wpointer-arith -Winvalid-pch
-Wcast-align -Wdisabled-optimization -Woverloaded-virtual -Wshadow
-Wwrite-strings -Wstrict-null-sentinel -Wno-invalid-offsetof  -Wa,-mbig-obj
-Wa,-mbig-obj  -mthreads   -mthreads -O3 -DNDEBUG   -mthreads   -mthreads
-shared -o ../../../bin/libSimpleITKExplicit-1.0.dll
-Wl,--out-implib,../../../lib/libSimpleITKExplicit-1.0.dll.a
-Wl,--major-image-version,1,--minor-image-version,0 -Wl,--whole-archive
CMakeFiles/SimpleITKExplicit.dir/objects.a -Wl,--no-whole-archive
@CMakeFiles/SimpleITKExplicit.dir/linklibs.rsp


As a guide, the final shared library is 259M on linux. The windows build
doesn't appear to be memory limited - the ld process is using about 1.5G
and consuming 1 core, but doesn't complete given an entire day.

Any suggestions on where to turn next? Are cross compilers the next step?

	[[alternative HTML version deleted]]

On Mon, Mar 6, 2017 at 6:33 AM, Richard Beare <richard.beare at gmail.com>
wrote:
> I am working on the SimpleITK package for R. This is an enormous package
> that is largely automatically generated via a set of swig/json/lua magic,
> and is working well under linux and osx.
Is it available somewhere so we can try it?

> However we're having a lot of trouble with the Windows side. In fact,
we are struggling to get the base libraries to build using the RTools 3.4
toolchain, even before the worrying about the R-specific parts.
What build environment do you use? The version of gcc with Rtools
should be ok, but the Rtools build utilities in the "bin" folder (in
particular 'make') are old and a frequent source of problems. However
for building external libs you can use other tools, for example those
from msys2. Just make sure you use gcc/g++ from Rtools.

> The current issue is very long time (possibly infinite) linking of dlls, or
> test executables. I've tried using a FAT32 file system for the build,
as
> suggested by some old bug reports, but still have the issue.
On Windows you can avoid the run-time dll mess by building static libs
of external libraries. See rwinlib for examples:
https://github.com/rwinlib

> Any suggestions on where to turn next? Are cross compilers the next step?
Try building with msys2, but make sure to use gcc/g++ from Rtools by
setting the `CC` and `CXX` variables in the configure script. Cross
compiling will make things even more complicated because binaries
might not be compatible if your cross compiler has a different version
of gcc or has been configured for another exception model (seh/drawf).

Yep - simpleITK is available at github.com/SimpleITK/SimpleITK. There's
also github.com/SimpleITK/SimpleITKRInstaller - a devtools based installer
for mac and linux.

CMake has a range of build environments. I experimented with MSYS2 and
mingw makefiles, but had trouble with incompatibilities in the path
required by CMake and those options - from memory the sh in RTools/bin
caused problems. Although it sounds like you are saying it is necessary to
install the MSYS2 system as well.

The unix makefile option for CMake appears to work well until the linking
stage. Ninja has problems at a similar stage.

I'll steer clear of dll's, as you suggest. I'm checking those links
for
compiler/links flags to see if we're missing anything

Thanks



On Mon, Mar 6, 2017 at 9:51 PM, Jeroen Ooms <jeroenooms at gmail.com>
wrote:
> On Mon, Mar 6, 2017 at 6:33 AM, Richard Beare <richard.beare at
gmail.com>
> wrote:
> > I am working on the SimpleITK package for R. This is an enormous
package
> > that is largely automatically generated via a set of swig/json/lua
magic,
> > and is working well under linux and osx.
>
> Is it available somewhere so we can try it?
>
>
> > However we're having a lot of trouble with the Windows side. In
fact, we
> are struggling to get the base libraries to build using the RTools 3.4
> toolchain, even before the worrying about the R-specific parts.
>
> What build environment do you use? The version of gcc with Rtools
> should be ok, but the Rtools build utilities in the "bin" folder
(in
> particular 'make') are old and a frequent source of problems.
However
> for building external libs you can use other tools, for example those
> from msys2. Just make sure you use gcc/g++ from Rtools.
>
>
> > The current issue is very long time (possibly infinite) linking of
dlls,
> or
> > test executables. I've tried using a FAT32 file system for the
build, as
> > suggested by some old bug reports, but still have the issue.
>
> On Windows you can avoid the run-time dll mess by building static libs
> of external libraries. See rwinlib for examples:
> https://github.com/rwinlib
>
>
> > Any suggestions on where to turn next? Are cross compilers the next
step?
>
> Try building with msys2, but make sure to use gcc/g++ from Rtools by
> setting the `CC` and `CXX` variables in the configure script. Cross
> compiling will make things even more complicated because binaries
> might not be compatible if your cross compiler has a different version
> of gcc or has been configured for another exception model (seh/drawf).
>
	[[alternative HTML version deleted]]