R devel - Mar 2017 - Seeking advice regarding compilation of large libraries using RTools (Windows)

Yep - simpleITK is available at github.com/SimpleITK/SimpleITK. There's
also github.com/SimpleITK/SimpleITKRInstaller - a devtools based installer
for mac and linux.

CMake has a range of build environments. I experimented with MSYS2 and
mingw makefiles, but had trouble with incompatibilities in the path
required by CMake and those options - from memory the sh in RTools/bin
caused problems. Although it sounds like you are saying it is necessary to
install the MSYS2 system as well.

The unix makefile option for CMake appears to work well until the linking
stage. Ninja has problems at a similar stage.

I'll steer clear of dll's, as you suggest. I'm checking those links
for
compiler/links flags to see if we're missing anything

Thanks



On Mon, Mar 6, 2017 at 9:51 PM, Jeroen Ooms <jeroenooms at gmail.com>
wrote:
> On Mon, Mar 6, 2017 at 6:33 AM, Richard Beare <richard.beare at
gmail.com>
> wrote:
> > I am working on the SimpleITK package for R. This is an enormous
package
> > that is largely automatically generated via a set of swig/json/lua
magic,
> > and is working well under linux and osx.
>
> Is it available somewhere so we can try it?
>
>
> > However we're having a lot of trouble with the Windows side. In
fact, we
> are struggling to get the base libraries to build using the RTools 3.4
> toolchain, even before the worrying about the R-specific parts.
>
> What build environment do you use? The version of gcc with Rtools
> should be ok, but the Rtools build utilities in the "bin" folder
(in
> particular 'make') are old and a frequent source of problems.
However
> for building external libs you can use other tools, for example those
> from msys2. Just make sure you use gcc/g++ from Rtools.
>
>
> > The current issue is very long time (possibly infinite) linking of
dlls,
> or
> > test executables. I've tried using a FAT32 file system for the
build, as
> > suggested by some old bug reports, but still have the issue.
>
> On Windows you can avoid the run-time dll mess by building static libs
> of external libraries. See rwinlib for examples:
> https://github.com/rwinlib
>
>
> > Any suggestions on where to turn next? Are cross compilers the next
step?
>
> Try building with msys2, but make sure to use gcc/g++ from Rtools by
> setting the `CC` and `CXX` variables in the configure script. Cross
> compiling will make things even more complicated because binaries
> might not be compatible if your cross compiler has a different version
> of gcc or has been configured for another exception model (seh/drawf).
>
	[[alternative HTML version deleted]]

On Mon, Mar 6, 2017 at 8:21 PM, Richard Beare <richard.beare at gmail.com>
wrote:
> Yep - simpleITK is available at github.com/SimpleITK/SimpleITK. There's
> also github.com/SimpleITK/SimpleITKRInstaller - a devtools based installer
> for mac and linux.
>
> CMake has a range of build environments. I experimented with MSYS2 and
> mingw makefiles, but had trouble with incompatibilities in the path
> required by CMake and those options - from memory the sh in RTools/bin
> caused problems. Although it sounds like you are saying it is necessary to
> install the MSYS2 system as well.
>
MSYS2 has two variants (3 distinctly named packages) of CMake:
mingw-w64-{i686,x86_64}-cmake and cmake. Which of these (if any) did you
use? I cannot state without experimenting which should be used, but the
mingw-w64 prefixed ones are usually the ones you want to generate native
code.

>
> The unix makefile option for CMake appears to work well until the linking
> stage. Ninja has problems at a similar stage.
>
> I'll steer clear of dll's, as you suggest. I'm checking those
links for
> compiler/links flags to see if we're missing anything
>
I would be happy to see if I can get it to compile in the Anaconda
Distribution too. Here we prefer DLLs. The gnu ld linker on Windows is very
slow and doesn't seem to scale too well. I do not expect cross-compiling
would make it much faster. The clang linker (ldd) is nearly viable on
Windows and apparently much faster. I'm not sure if this near-viability is
in relation to using it in msvc mode, gcc mode or both (or even whether it
implements frontends for both).

> Thanks
>
>
>
> On Mon, Mar 6, 2017 at 9:51 PM, Jeroen Ooms <jeroenooms at gmail.com>
wrote:
>
> > On Mon, Mar 6, 2017 at 6:33 AM, Richard Beare <richard.beare at
gmail.com>
> > wrote:
> > > I am working on the SimpleITK package for R. This is an enormous
> package
> > > that is largely automatically generated via a set of
swig/json/lua
> magic,
> > > and is working well under linux and osx.
> >
> > Is it available somewhere so we can try it?
> >
> >
> > > However we're having a lot of trouble with the Windows side.
In fact,
> we
> > are struggling to get the base libraries to build using the RTools 3.4
> > toolchain, even before the worrying about the R-specific parts.
> >
> > What build environment do you use? The version of gcc with Rtools
> > should be ok, but the Rtools build utilities in the "bin"
folder (in
> > particular 'make') are old and a frequent source of problems.
However
> > for building external libs you can use other tools, for example those
> > from msys2. Just make sure you use gcc/g++ from Rtools.
>
>
> >
> > > The current issue is very long time (possibly infinite) linking
of
> dlls,
> > or
> > > test executables. I've tried using a FAT32 file system for
the build,
> as
> > > suggested by some old bug reports, but still have the issue.
> >
> > On Windows you can avoid the run-time dll mess by building static libs
> > of external libraries. See rwinlib for examples:
> > https://github.com/rwinlib
> >
> >
> > > Any suggestions on where to turn next? Are cross compilers the
next
> step?
> >
> > Try building with msys2, but make sure to use gcc/g++ from Rtools by
> > setting the `CC` and `CXX` variables in the configure script. Cross
> > compiling will make things even more complicated because binaries
> > might not be compatible if your cross compiler has a different version
> > of gcc or has been configured for another exception model (seh/drawf).
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]

So far I've been using the separate cmake package (direct from kitware),
rather than one available via msys2 (as I started without msys2). I'll have
a play with the msys2 versions.

The problem definitely seems to be with ld - it sits at high load for days
at a time, and I haven't seen it complete successfully on the test exe
files. They also hang around when I kill the make.

I'm having a play with building libraries but not test infrastructure,
although this isn't a long term solution.

Sounds like clang is something worth checking out - any comments on whether
it is close to working with R?

On Wed, Mar 8, 2017 at 10:35 AM, Ray Donnelly <rdonnelly at continuum.io>
wrote:
> On Mon, Mar 6, 2017 at 8:21 PM, Richard Beare <richard.beare at
gmail.com>
> wrote:
>
>> Yep - simpleITK is available at github.com/SimpleITK/SimpleITK.
There's
>> also github.com/SimpleITK/SimpleITKRInstaller - a devtools based
>> installer
>> for mac and linux.
>>
>> CMake has a range of build environments. I experimented with MSYS2 and
>> mingw makefiles, but had trouble with incompatibilities in the path
>> required by CMake and those options - from memory the sh in RTools/bin
>> caused problems. Although it sounds like you are saying it is necessary
to
>> install the MSYS2 system as well.
>>
>
> MSYS2 has two variants (3 distinctly named packages) of CMake:
> mingw-w64-{i686,x86_64}-cmake and cmake. Which of these (if any) did you
> use? I cannot state without experimenting which should be used, but the
> mingw-w64 prefixed ones are usually the ones you want to generate native
> code.
>
>
>>
>> The unix makefile option for CMake appears to work well until the
linking
>> stage. Ninja has problems at a similar stage.
>>
>> I'll steer clear of dll's, as you suggest. I'm checking
those links for
>> compiler/links flags to see if we're missing anything
>>
>
> I would be happy to see if I can get it to compile in the Anaconda
> Distribution too. Here we prefer DLLs. The gnu ld linker on Windows is very
> slow and doesn't seem to scale too well. I do not expect
cross-compiling
> would make it much faster. The clang linker (ldd) is nearly viable on
> Windows and apparently much faster. I'm not sure if this near-viability
is
> in relation to using it in msvc mode, gcc mode or both (or even whether it
> implements frontends for both).
>
>
>> Thanks
>>
>>
>>
>> On Mon, Mar 6, 2017 at 9:51 PM, Jeroen Ooms <jeroenooms at
gmail.com> wrote:
>>
>> > On Mon, Mar 6, 2017 at 6:33 AM, Richard Beare <richard.beare at
gmail.com>
>> > wrote:
>> > > I am working on the SimpleITK package for R. This is an
enormous
>> package
>> > > that is largely automatically generated via a set of
swig/json/lua
>> magic,
>> > > and is working well under linux and osx.
>> >
>> > Is it available somewhere so we can try it?
>> >
>> >
>> > > However we're having a lot of trouble with the Windows
side. In fact,
>> we
>> > are struggling to get the base libraries to build using the RTools
3.4
>> > toolchain, even before the worrying about the R-specific parts.
>> >
>> > What build environment do you use? The version of gcc with Rtools
>> > should be ok, but the Rtools build utilities in the
"bin" folder (in
>> > particular 'make') are old and a frequent source of
problems. However
>> > for building external libs you can use other tools, for example
those
>> > from msys2. Just make sure you use gcc/g++ from Rtools.
>>
> >
>> >
>> > > The current issue is very long time (possibly infinite)
linking of
>> dlls,
>> > or
>> > > test executables. I've tried using a FAT32 file system
for the build,
>> as
>> > > suggested by some old bug reports, but still have the issue.
>> >
>> > On Windows you can avoid the run-time dll mess by building static
libs
>> > of external libraries. See rwinlib for examples:
>> > https://github.com/rwinlib
>> >
>> >
>> > > Any suggestions on where to turn next? Are cross compilers
the next
>> step?
>> >
>> > Try building with msys2, but make sure to use gcc/g++ from Rtools
by
>> > setting the `CC` and `CXX` variables in the configure script.
Cross
>> > compiling will make things even more complicated because binaries
>> > might not be compatible if your cross compiler has a different
version
>> > of gcc or has been configured for another exception model
(seh/drawf).
>> >
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
>
	[[alternative HTML version deleted]]