R devel - Aug 2021 - [External] Re: Update on rtools4 and ucrt support

On Tue, Aug 24, 2021 at 12:47 AM Simon Urbanek <simon.urbanek at
r-project.org>
wrote:
> Avi,
>
> please see the announcement:
>
>
>
https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html
>
> the documentation is in
>
>
>
https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html
>
> Cheers,
> Simon
>
>
>
>
> On Aug 24, 2021, at 8:34 AM, Avraham Adler <avraham.adler at
gmail.com>
> wrote:
>
> On Mon, Aug 23, 2021 at 11:09 PM <luke-tierney at uiowa.edu> wrote:
>
> On Mon, 23 Aug 2021, Duncan Murdoch wrote:
>
> On 23/08/2021 8:15 a.m., jan Vitek via R-devel wrote:
>
> Hi Jeroen,
>
> I mostly lurk on this list, but I was struck by your combative tone.
>
> To pick on two random bits:
>
> ? a 6gb tarball with manually built things on his personal machine?
>
>
> ? a black-box system that is so opaque and complex that only one person
> knows how it works, and would make it much more difficult for
> students, universities, and other organisations to build R packages
> and libraries on Windows?
>
>
>
> Tomas? tool chain isn't a blackbox, it has copious documentation (see
>
> [1])
>
> and builds on any machine thanks to the provided docker container.
>
> This is not to criticise your work which has its unique strengths, but
>
> to
>
> state the obvious: these strengths are best discussed without passion
> based on factually accurate descriptions.
>
>
> I agree with Jan.  I'm not sure a discussion in this forum would be
>
> fruitful,
>
> but I really wish Jeroen and Tomas would get together, aiming to merge
>
> their
>
> toolchains, keeping the best aspects of both.
>
> I haven't been involved in the development of either one, but have been
>
> a
>
> "victim" of the two chain rivalry, because the rgl package is not
easy
>
> to
>
> build.  I get instructions from each of them on how to do the build, and
> those instructions for one toolchain generally break the build on the
>
> other
>
> one.  While it is probably possible to detect the toolchain and have the
> build adapt to whichever one is in use, it would be a lot easier for me
>
> (and
>
> I imagine every other maintainer of a package using external libs) if I
>
> just
>
> had to follow one set of instructions.
>
> Duncan Murdoch
>
>
> Here are just a few comments from my perspective (I am an R-core
> member, but am not part of the CRAN team and do only very limited work
> on Windows). Other R-core members may have different perspectives and
> insights.
>
> One bit of background: dealing with encoding issues on Windows has
> been taking an unsustainable amount of R-core resources for some time
> now. Tomas Kalibera has been taking the lead on trying to address
> these issues in the existing framework, but this means he has not had
> the time to make any of the many other valuable and important
> contributions he could make. The only viable way forward is to move to
> a Windows tool chain that supports UTF-8 as the C library current
> encoding via the Windows UCRT framework.
>
> Tomas Kalibera has, on behalf of all of R core and in
> coordination with CRAN, been looking for a way forward for some
> time and has reported on the progress in several blog posts at
> https://developer.r-project.org/Blog/public/. This has lead to
> the development of the MXE-based UCRT tool chain, which is now
> well tested and ready for deployment.  Checks using the UCRT tool
> chain have been part of the CRAN check process for a while. I
> believe CRAN plans to switch R-devel checks and builds to the
> UCRT tool chain during the upcoming CRAN downtime. I expect there
> will be some communication from CRAN on this soon, including on
> any issues in supporting binaries for both R-devel and R-patched.
>
> In putting together something as large as a tool chain there will
> always be many choices, each with advantages and disadvantages.  Some
> things may be advantages in some settings and not others. Taking just
> one case in point: Cross compilation. This is likely to be a better
> approach for CRAN in the future and is supported by the MXE framework
> on which the new tool chain is based.
>
> The much more recent changes in rtools4 to support UCRT are at this
> point not yet as well tested as the new tool chain. Once these changes
> to rtools4 mature, and if binary compatibility can be assured, then
> having a second tool chain may be useful in some cases.  But if there
> are incompatibilities then it will be up to rtools4 to keep up with
> the tool chain used by CRAN. On the other, contributing to improving
> the MXE-based tool chain may be a better investment of time.
>
> Best,
>
> luke
>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>    Actuarial Science
> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
> <luke-tierney at uiowa.edu>
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
>
>
> Thank you, Dr. Tierney. However, I am concerned about the not-insignificant
> number of us who for various reasons can only do our development on
> Windows. Rtools has been the official tool chain with which to build
> windows for the at least 20 years I have been using R (yes, a babe in the
> woods compared to most, but not a complete neophyte). Duncan and Jereoen
> have each done yeoman?s jobs in ensuring that R can be built from source on
> Windows and that packages can be developed for all OSS on Windows?even
> Solaris SPARC.
>
> I am much less aware of Thomas?s work, and I?ll gladly take the blame for
> it, but I haven?t seen an accessible tool chain built by him which would
> allow me, the Windows developer, to build R and all packages from source on
> a native Windows box. Have I just missed it? If not, is that planned? If
> R-core switches the official Windows toolchain, where does that leave us?
>
> Thank you,
>
> Avi
>
> --
> Sent from Gmail Mobile
>
>
Thank you, Simon. That was valuable. Skimming that quickly, I get a bit
concerned. I?ve been building from source and then using OpenBLAS in my R
source for many, many years now, and it looks like its support is tenuous
in the experimental chain. Similarly with packages like nloptr, where I
build NLOPT 2.6+ and have adjusted Jelmer?s code for it to work in R for
Windows. I maintain packages with Fortran/OpenMP and Rcpp(Parallel). I hope
that should the decision be made to switch, it will be done when the build
process is more streamlined, especially for some fundamental packages.

That being said, I must take the opportunity again to thank R-core, Tomas,
Jeroen, Duncan among many others who have built an infrastructure that
allows amateur programmers to contribute to the statistical infrastructure.

Thanks again,

Avi
-- 
Sent from Gmail Mobile

	[[alternative HTML version deleted]]

On 8/24/21 12:03 AM, Avraham Adler wrote:
> On Tue, Aug 24, 2021 at 12:47 AM Simon Urbanek <simon.urbanek at
r-project.org>
> wrote:
>
>> Avi,
>>
>> please see the announcement:
>>
>>
>>
https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html
>>
>> the documentation is in
>>
>>
>>
https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html
>>
>> Cheers,
>> Simon
>>
>>
>>
>>
>> On Aug 24, 2021, at 8:34 AM, Avraham Adler <avraham.adler at
gmail.com>
>> wrote:
>>
>> On Mon, Aug 23, 2021 at 11:09 PM <luke-tierney at uiowa.edu>
wrote:
>>
>> On Mon, 23 Aug 2021, Duncan Murdoch wrote:
>>
>> On 23/08/2021 8:15 a.m., jan Vitek via R-devel wrote:
>>
>> Hi Jeroen,
>>
>> I mostly lurk on this list, but I was struck by your combative tone.
>>
>> To pick on two random bits:
>>
>> ? a 6gb tarball with manually built things on his personal machine?
>>
>>
>> ? a black-box system that is so opaque and complex that only one person
>> knows how it works, and would make it much more difficult for
>> students, universities, and other organisations to build R packages
>> and libraries on Windows?
>>
>>
>>
>> Tomas? tool chain isn't a blackbox, it has copious documentation
(see
>>
>> [1])
>>
>> and builds on any machine thanks to the provided docker container.
>>
>> This is not to criticise your work which has its unique strengths, but
>>
>> to
>>
>> state the obvious: these strengths are best discussed without passion
>> based on factually accurate descriptions.
>>
>>
>> I agree with Jan.  I'm not sure a discussion in this forum would be
>>
>> fruitful,
>>
>> but I really wish Jeroen and Tomas would get together, aiming to merge
>>
>> their
>>
>> toolchains, keeping the best aspects of both.
>>
>> I haven't been involved in the development of either one, but have
been
>>
>> a
>>
>> "victim" of the two chain rivalry, because the rgl package is
not easy
>>
>> to
>>
>> build.  I get instructions from each of them on how to do the build,
and
>> those instructions for one toolchain generally break the build on the
>>
>> other
>>
>> one.  While it is probably possible to detect the toolchain and have
the
>> build adapt to whichever one is in use, it would be a lot easier for me
>>
>> (and
>>
>> I imagine every other maintainer of a package using external libs) if I
>>
>> just
>>
>> had to follow one set of instructions.
>>
>> Duncan Murdoch
>>
>>
>> Here are just a few comments from my perspective (I am an R-core
>> member, but am not part of the CRAN team and do only very limited work
>> on Windows). Other R-core members may have different perspectives and
>> insights.
>>
>> One bit of background: dealing with encoding issues on Windows has
>> been taking an unsustainable amount of R-core resources for some time
>> now. Tomas Kalibera has been taking the lead on trying to address
>> these issues in the existing framework, but this means he has not had
>> the time to make any of the many other valuable and important
>> contributions he could make. The only viable way forward is to move to
>> a Windows tool chain that supports UTF-8 as the C library current
>> encoding via the Windows UCRT framework.
>>
>> Tomas Kalibera has, on behalf of all of R core and in
>> coordination with CRAN, been looking for a way forward for some
>> time and has reported on the progress in several blog posts at
>> https://developer.r-project.org/Blog/public/. This has lead to
>> the development of the MXE-based UCRT tool chain, which is now
>> well tested and ready for deployment.  Checks using the UCRT tool
>> chain have been part of the CRAN check process for a while. I
>> believe CRAN plans to switch R-devel checks and builds to the
>> UCRT tool chain during the upcoming CRAN downtime. I expect there
>> will be some communication from CRAN on this soon, including on
>> any issues in supporting binaries for both R-devel and R-patched.
>>
>> In putting together something as large as a tool chain there will
>> always be many choices, each with advantages and disadvantages.  Some
>> things may be advantages in some settings and not others. Taking just
>> one case in point: Cross compilation. This is likely to be a better
>> approach for CRAN in the future and is supported by the MXE framework
>> on which the new tool chain is based.
>>
>> The much more recent changes in rtools4 to support UCRT are at this
>> point not yet as well tested as the new tool chain. Once these changes
>> to rtools4 mature, and if binary compatibility can be assured, then
>> having a second tool chain may be useful in some cases.  But if there
>> are incompatibilities then it will be up to rtools4 to keep up with
>> the tool chain used by CRAN. On the other, contributing to improving
>> the MXE-based tool chain may be a better investment of time.
>>
>> Best,
>>
>> luke
>>
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>>
>> --
>> Luke Tierney
>> Ralph E. Wareham Professor of Mathematical Sciences
>> University of Iowa                  Phone:             319-335-3386
>> Department of Statistics and        Fax:               319-335-3017
>>     Actuarial Science
>> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
>> <luke-tierney at uiowa.edu>
>> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
>>
>>
>> Thank you, Dr. Tierney. However, I am concerned about the
not-insignificant
>> number of us who for various reasons can only do our development on
>> Windows. Rtools has been the official tool chain with which to build
>> windows for the at least 20 years I have been using R (yes, a babe in
the
>> woods compared to most, but not a complete neophyte). Duncan and
Jereoen
>> have each done yeoman?s jobs in ensuring that R can be built from
source on
>> Windows and that packages can be developed for all OSS on Windows?even
>> Solaris SPARC.
>>
>> I am much less aware of Thomas?s work, and I?ll gladly take the blame
for
>> it, but I haven?t seen an accessible tool chain built by him which
would
>> allow me, the Windows developer, to build R and all packages from
source on
>> a native Windows box. Have I just missed it? If not, is that planned?
If
>> R-core switches the official Windows toolchain, where does that leave
us?
>>
>> Thank you,
>>
>> Avi
>>
>> --
>> Sent from Gmail Mobile
>>
>>
> Thank you, Simon. That was valuable. Skimming that quickly, I get a bit
> concerned. I?ve been building from source and then using OpenBLAS in my R
> source for many, many years now, and it looks like its support is tenuous
> in the experimental chain. Similarly with packages like nloptr, where I
> build NLOPT 2.6+ and have adjusted Jelmer?s code for it to work in R for
> Windows. I maintain packages with Fortran/OpenMP and Rcpp(Parallel). I hope
> that should the decision be made to switch, it will be done when the build
> process is more streamlined, especially for some fundamental packages.
Hi Avi,

if your R package includes source code (C/C++/Fortran), there is no 
fundamental difference, just the version of the compilers and/or 
external libraries may have changed.

nloptr checks are fine, see 
https://cran.r-project.org/web/checks/check_results_nloptr.html

The architecture is r-devel-windows-x86_64-gcc10-UCRT.

Best
Tomas
>
> That being said, I must take the opportunity again to thank R-core, Tomas,
> Jeroen, Duncan among many others who have built an infrastructure that
> allows amateur programmers to contribute to the statistical infrastructure.
>
> Thanks again,
>
> Avi

Avi,

thanks. Yes, the whole point of the developer blog posts by R-core are for uses
to provide feedback, so that's great - it's odd that it required a
somewhat orthogonal post to start the discussion several months later, but
I'm glad we got here.

Note that the point of the switch is to iron out all issues that may be
encountered and provide real CRAN testing - that's why both R-core and CRAN
is involved in all this. This only affects R-devel since we want to be ready for
the R 4.2.0 release, it won't affect the current release builds.

Anyway, now I'll leave it to the Windows experts to address the details and
work together.

Cheers,
Simon

> On Aug 24, 2021, at 10:03 AM, Avraham Adler <avraham.adler at
gmail.com> wrote:
> 
> 
> Thank you, Simon. That was valuable. Skimming that quickly, I get a bit
concerned. I?ve been building from source and then using OpenBLAS in my R source
for many, many years now, and it looks like its support is tenuous in the
experimental chain. Similarly with packages like nloptr, where I build NLOPT
2.6+ and have adjusted Jelmer?s code for it to work in R for Windows. I maintain
packages with Fortran/OpenMP and Rcpp(Parallel). I hope that should the decision
be made to switch, it will be done when the build process is more streamlined,
especially for some fundamental packages.
> 
> That being said, I must take the opportunity again to thank R-core, Tomas,
Jeroen, Duncan among many others who have built an infrastructure that allows
amateur programmers to contribute to the statistical infrastructure.
> 
> Thanks again,
> 
> Avi