R devel - Mar 2015 - Notes on building a gcc toolchain for Rtools (but not multilib)

----- Original Message -----
> From: "Duncan Murdoch" <murdoch.duncan at gmail.com>
> To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
> Cc: "Hsiu-Khuern Tang" <tangoh at gmail.com>, r-devel at
r-project.org
> Sent: Tuesday, March 10, 2015 11:37:12 AM
> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools (but not
multilib)
> 
> On 10/03/2015 12:54 PM, Dan Tenenbaum wrote:
> >
> > ----- Original Message -----
> > > From: "Duncan Murdoch" <murdoch.duncan at
gmail.com>
> > > To: "Hsiu-Khuern Tang" <tangoh at gmail.com>,
r-devel at r-project.org
> > > Sent: Monday, March 9, 2015 10:40:02 AM
> > > Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools
> > > (but not	multilib)
> > >
> > > On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> > > > On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
> > > > <murdoch.duncan at gmail.com> wrote:
> > > > > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> > > > >> Hi,
> > > > >>
> > > > >> [This is a follow-up to the "New version of
Rtools for
> > > > >> Windows"
> > > > >> thread
> > > > >> in January, but I just subscribed and don't
know how to
> > > > >> reply to
> > > > >> an
> > > > >> old thread -- my apologies.]
> > > > >
> > > > > I am planning to put a new Rtools online today that
uses a
> > > > > different
> > > > > build of gcc 4.9.2.  I will be concentrating on getting
it to
> > > > > work with
> > > > > all the external libraries before the 3.2.0 release
next
> > > > > month.
> > > > >  I'm not
> > > > > planning to try to get it to work with R-patched, and I
> > > > > expect it
> > > > > won't:
> > > > >  I needed to make a number of patches to R-devel for
> > > > >  compatibility.
> > > >
> > > > I also worked off R-devel (I said wrongly that it was
R-patched
> > > > in
> > > > my
> > > > original post) and benefited from your compatibility
changes.
> > > >
> > > > I look forward to the new Rtools and will test it by
compiling
> > > > some
> > > > packages.
> > >
> > > It's now on the main site at CRAN, and should propagate to
the
> > > mirrors
> > > reasonably quickly.  I'm hoping that tomorrow's R-devel
build
> > > will
> > > use
> > > it, but there may be some last minute problems.
> > >
> >
> > Thanks to you and everyone who worked on this. Is there a way to
> > tell which toolchain built a given R-devel binary?
> > If not, can you let us know when there is one on CRAN that was
> > built with the new Rtools?
> 
> If you look in etc/*/Makeconf, you'll see something like this:
> 
> BINPREF ?= $(RTOOLS)gcc492_64/bin/
> 
> hopefully from tomorrow onwards.  If today's build was with the new
> toolchain, you should see a hardcoded path to where I have Rtools
> installed on the build machine, which isn't so helpful.  The previous
> toolchain left BINPREF blank.
> 
> If you want to use your own toolchain, just edit those files.  If you
> want to install the standard Rtools somewhere else, set an
> environment
> variable like
> 
> RTOOLS = C:/Rtools/
> 
> (where the terminal / is required.)
> 
Thanks, that's very helpful. I also notice with the latest R-devel binary
(67969, which is built with the new Rtools according to what you say above) that
I need to do setInternet2(TRUE) before I can download from any URLs; I see some
mention of that earlier in this thread, is this intended? If so is there a way
to make this the default?

Thanks,
Dan

> 
>

On 10/03/2015 2:56 PM, Dan Tenenbaum wrote:
> 
> 
> ----- Original Message -----
>> From: "Duncan Murdoch" <murdoch.duncan at gmail.com>
>> To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
>> Cc: "Hsiu-Khuern Tang" <tangoh at gmail.com>, r-devel
at r-project.org
>> Sent: Tuesday, March 10, 2015 11:37:12 AM
>> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools (but not
multilib)
>>
>> On 10/03/2015 12:54 PM, Dan Tenenbaum wrote:
>>>
>>> ----- Original Message -----
>>>> From: "Duncan Murdoch" <murdoch.duncan at
gmail.com>
>>>> To: "Hsiu-Khuern Tang" <tangoh at gmail.com>,
r-devel at r-project.org
>>>> Sent: Monday, March 9, 2015 10:40:02 AM
>>>> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools
>>>> (but not	multilib)
>>>>
>>>> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
>>>>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
>>>>> <murdoch.duncan at gmail.com> wrote:
>>>>>> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> [This is a follow-up to the "New version of
Rtools for
>>>>>>> Windows"
>>>>>>> thread
>>>>>>> in January, but I just subscribed and don't
know how to
>>>>>>> reply to
>>>>>>> an
>>>>>>> old thread -- my apologies.]
>>>>>>
>>>>>> I am planning to put a new Rtools online today that
uses a
>>>>>> different
>>>>>> build of gcc 4.9.2.  I will be concentrating on getting
it to
>>>>>> work with
>>>>>> all the external libraries before the 3.2.0 release
next
>>>>>> month.
>>>>>>  I'm not
>>>>>> planning to try to get it to work with R-patched, and I
>>>>>> expect it
>>>>>> won't:
>>>>>>  I needed to make a number of patches to R-devel for
>>>>>>  compatibility.
>>>>>
>>>>> I also worked off R-devel (I said wrongly that it was
R-patched
>>>>> in
>>>>> my
>>>>> original post) and benefited from your compatibility
changes.
>>>>>
>>>>> I look forward to the new Rtools and will test it by
compiling
>>>>> some
>>>>> packages.
>>>>
>>>> It's now on the main site at CRAN, and should propagate to
the
>>>> mirrors
>>>> reasonably quickly.  I'm hoping that tomorrow's R-devel
build
>>>> will
>>>> use
>>>> it, but there may be some last minute problems.
>>>>
>>>
>>> Thanks to you and everyone who worked on this. Is there a way to
>>> tell which toolchain built a given R-devel binary?
>>> If not, can you let us know when there is one on CRAN that was
>>> built with the new Rtools?
>>
>> If you look in etc/*/Makeconf, you'll see something like this:
>>
>> BINPREF ?= $(RTOOLS)gcc492_64/bin/
>>
>> hopefully from tomorrow onwards.  If today's build was with the new
>> toolchain, you should see a hardcoded path to where I have Rtools
>> installed on the build machine, which isn't so helpful.  The
previous
>> toolchain left BINPREF blank.
>>
>> If you want to use your own toolchain, just edit those files.  If you
>> want to install the standard Rtools somewhere else, set an
>> environment
>> variable like
>>
>> RTOOLS = C:/Rtools/
>>
>> (where the terminal / is required.)
>>
> 
> Thanks, that's very helpful. I also notice with the latest R-devel
binary (67969, which is built with the new Rtools according to what you say
above) that I need to do setInternet2(TRUE) before I can download from any URLs;
I see some mention of that earlier in this thread, is this intended? If so is
there a way to make this the default?
That's a bug.  I haven't tracked down what's going wrong with our
regular code.  If I can't find that and fix it soon, I'll make Internet2
the default.  For now, you can do it yourself using the instructions on
?setInternet2.

Duncan Murdoch

On Tue, Mar 10, 2015 at 3:17 PM, Duncan Murdoch
<murdoch.duncan at gmail.com> wrote:
>
> That's a bug.  I haven't tracked down what's going wrong with
our
> regular code.  If I can't find that and fix it soon, I'll make
Internet2
> the default.  For now, you can do it yourself using the instructions on
> ?setInternet2.
>
> Duncan Murdoch
I successfully rebuilt R-devel_2015-03-09 with the most recent version
of Rtools tonight, building both ICU_531 and this time libcurl (7.39)
as well (and OpenBLAS, of course). The internet bug is still there,
but the rest of make-check all passed with flying colors, as did
building 'microbenchmark' from source (with all the other needed
packages, including Rcpp and Hsiu-Kheurn's change to NM was *not*
used). My non-BLAS test tonight ran faster than last night; maybe 1000
iterations aren't enough or I had something else eating up clock
cycles last night. Either way, outside the internet bug, it's looking
good for Windows 64bit (Win7 at least).

Thanks,

Avi

On 10/03/2015 3:17 PM, Duncan Murdoch wrote:
> On 10/03/2015 2:56 PM, Dan Tenenbaum wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Duncan Murdoch" <murdoch.duncan at
gmail.com>
> >> To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
> >> Cc: "Hsiu-Khuern Tang" <tangoh at gmail.com>,
r-devel at r-project.org
> >> Sent: Tuesday, March 10, 2015 11:37:12 AM
> >> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools
(but not multilib)
> >>
> >> On 10/03/2015 12:54 PM, Dan Tenenbaum wrote:
> >>>
> >>> ----- Original Message -----
> >>>> From: "Duncan Murdoch" <murdoch.duncan at
gmail.com>
> >>>> To: "Hsiu-Khuern Tang" <tangoh at
gmail.com>, r-devel at r-project.org
> >>>> Sent: Monday, March 9, 2015 10:40:02 AM
> >>>> Subject: Re: [Rd] Notes on building a gcc toolchain for
Rtools
> >>>> (but not	multilib)
> >>>>
> >>>> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> >>>>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
> >>>>> <murdoch.duncan at gmail.com> wrote:
> >>>>>> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> [This is a follow-up to the "New version
of Rtools for
> >>>>>>> Windows"
> >>>>>>> thread
> >>>>>>> in January, but I just subscribed and
don't know how to
> >>>>>>> reply to
> >>>>>>> an
> >>>>>>> old thread -- my apologies.]
> >>>>>>
> >>>>>> I am planning to put a new Rtools online today
that uses a
> >>>>>> different
> >>>>>> build of gcc 4.9.2.  I will be concentrating on
getting it to
> >>>>>> work with
> >>>>>> all the external libraries before the 3.2.0
release next
> >>>>>> month.
> >>>>>>  I'm not
> >>>>>> planning to try to get it to work with R-patched,
and I
> >>>>>> expect it
> >>>>>> won't:
> >>>>>>  I needed to make a number of patches to R-devel
for
> >>>>>>  compatibility.
> >>>>>
> >>>>> I also worked off R-devel (I said wrongly that it was
R-patched
> >>>>> in
> >>>>> my
> >>>>> original post) and benefited from your compatibility
changes.
> >>>>>
> >>>>> I look forward to the new Rtools and will test it by
compiling
> >>>>> some
> >>>>> packages.
> >>>>
> >>>> It's now on the main site at CRAN, and should
propagate to the
> >>>> mirrors
> >>>> reasonably quickly.  I'm hoping that tomorrow's
R-devel build
> >>>> will
> >>>> use
> >>>> it, but there may be some last minute problems.
> >>>>
> >>>
> >>> Thanks to you and everyone who worked on this. Is there a way
to
> >>> tell which toolchain built a given R-devel binary?
> >>> If not, can you let us know when there is one on CRAN that was
> >>> built with the new Rtools?
> >>
> >> If you look in etc/*/Makeconf, you'll see something like this:
> >>
> >> BINPREF ?= $(RTOOLS)gcc492_64/bin/
> >>
> >> hopefully from tomorrow onwards.  If today's build was with
the new
> >> toolchain, you should see a hardcoded path to where I have Rtools
> >> installed on the build machine, which isn't so helpful.  The
previous
> >> toolchain left BINPREF blank.
> >>
> >> If you want to use your own toolchain, just edit those files.  If
you
> >> want to install the standard Rtools somewhere else, set an
> >> environment
> >> variable like
> >>
> >> RTOOLS = C:/Rtools/
> >>
> >> (where the terminal / is required.)
> >>
> >
> > Thanks, that's very helpful. I also notice with the latest R-devel
binary (67969, which is built with the new Rtools according to what you say
above) that I need to do setInternet2(TRUE) before I can download from any URLs;
I see some mention of that earlier in this thread, is this intended? If so is
there a way to make this the default?
>
> That's a bug.  I haven't tracked down what's going wrong with
our
> regular code.  If I can't find that and fix it soon, I'll make
Internet2
> the default.  For now, you can do it yourself using the instructions on
> ?setInternet2.
I've found the problem now, and it will be easy to fix.  In case anyone 
else finds this thread, here's the problem:

The regular Windows internet code (not internet2) used the Winsock 
interface.  It returns different error codes than the Unix sockets 
interface.   Some error codes are ignorable (EWOULDBLOCK and 
EINPROGRESS); these are WSAEWOULDBLOCK and WSAEINPROGRESS in Winsock.  
We had code like

#ifndef EWOULDBLOCK
# define EWOULDBLOCK             WSAEWOULDBLOCK
#endif

so our code could work with the Unix macro names.  However, the new 
toolchain defines both EWOULDBLOCK
and WSAEWOULDBLOCK, so the remapping never happened, and we didn't 
ignore those errors.

Tomorrow's R-devel should have the fix in place, unless something else 
goes wrong, or I'm too slow.

Duncan Murdoch