Simon Urbanek
2021-Aug-23 21:41 UTC
[Rd] [External] Re: Update on rtools4 and ucrt support
Avi, please see the announcement: https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html <https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html> the documentation is in https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html <https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html> Cheers, Simon> On Aug 24, 2021, at 8:34 AM, Avraham Adler <avraham.adler at gmail.com> wrote: > > On Mon, Aug 23, 2021 at 11:09 PM <luke-tierney at uiowa.edu> wrote: > >> On Mon, 23 Aug 2021, Duncan Murdoch wrote: >> >>> On 23/08/2021 8:15 a.m., jan Vitek via R-devel wrote: >>>> Hi Jeroen, >>>> >>>> I mostly lurk on this list, but I was struck by your combative tone. >>>> >>>> To pick on two random bits: >>>> >>>>> ? a 6gb tarball with manually built things on his personal machine? >>>> >>>>> ? a black-box system that is so opaque and complex that only one person >>>>> knows how it works, and would make it much more difficult for >>>>> students, universities, and other organisations to build R packages >>>>> and libraries on Windows? >>>> >>>> >>>> Tomas? tool chain isn't a blackbox, it has copious documentation (see >> [1]) >>>> and builds on any machine thanks to the provided docker container. >>>> >>>> This is not to criticise your work which has its unique strengths, but >> to >>>> state the obvious: these strengths are best discussed without passion >>>> based on factually accurate descriptions. >>> >>> I agree with Jan. I'm not sure a discussion in this forum would be >> fruitful, >>> but I really wish Jeroen and Tomas would get together, aiming to merge >> their >>> toolchains, keeping the best aspects of both. >>> >>> I haven't been involved in the development of either one, but have been >> a >>> "victim" of the two chain rivalry, because the rgl package is not easy >> to >>> build. I get instructions from each of them on how to do the build, and >>> those instructions for one toolchain generally break the build on the >> other >>> one. While it is probably possible to detect the toolchain and have the >>> build adapt to whichever one is in use, it would be a lot easier for me >> (and >>> I imagine every other maintainer of a package using external libs) if I >> just >>> had to follow one set of instructions. >>> >>> Duncan Murdoch >> >> Here are just a few comments from my perspective (I am an R-core >> member, but am not part of the CRAN team and do only very limited work >> on Windows). Other R-core members may have different perspectives and >> insights. >> >> One bit of background: dealing with encoding issues on Windows has >> been taking an unsustainable amount of R-core resources for some time >> now. Tomas Kalibera has been taking the lead on trying to address >> these issues in the existing framework, but this means he has not had >> the time to make any of the many other valuable and important >> contributions he could make. The only viable way forward is to move to >> a Windows tool chain that supports UTF-8 as the C library current >> encoding via the Windows UCRT framework. >> >> Tomas Kalibera has, on behalf of all of R core and in >> coordination with CRAN, been looking for a way forward for some >> time and has reported on the progress in several blog posts at >> https://developer.r-project.org/Blog/public/. This has lead to >> the development of the MXE-based UCRT tool chain, which is now >> well tested and ready for deployment. Checks using the UCRT tool >> chain have been part of the CRAN check process for a while. I >> believe CRAN plans to switch R-devel checks and builds to the >> UCRT tool chain during the upcoming CRAN downtime. I expect there >> will be some communication from CRAN on this soon, including on >> any issues in supporting binaries for both R-devel and R-patched. >> >> In putting together something as large as a tool chain there will >> always be many choices, each with advantages and disadvantages. Some >> things may be advantages in some settings and not others. Taking just >> one case in point: Cross compilation. This is likely to be a better >> approach for CRAN in the future and is supported by the MXE framework >> on which the new tool chain is based. >> >> The much more recent changes in rtools4 to support UCRT are at this >> point not yet as well tested as the new tool chain. Once these changes >> to rtools4 mature, and if binary compatibility can be assured, then >> having a second tool chain may be useful in some cases. But if there >> are incompatibilities then it will be up to rtools4 to keep up with >> the tool chain used by CRAN. On the other, contributing to improving >> the MXE-based tool chain may be a better investment of time. >> >> Best, >> >> luke >> >>> >>> ______________________________________________ >>> R-devel at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >>> >> >> -- >> Luke Tierney >> Ralph E. Wareham Professor of Mathematical Sciences >> University of Iowa Phone: 319-335-3386 >> Department of Statistics and Fax: 319-335-3017 >> Actuarial Science >> 241 Schaeffer Hall email: luke-tierney at uiowa.edu >> Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu >> > > Thank you, Dr. Tierney. However, I am concerned about the not-insignificant > number of us who for various reasons can only do our development on > Windows. Rtools has been the official tool chain with which to build > windows for the at least 20 years I have been using R (yes, a babe in the > woods compared to most, but not a complete neophyte). Duncan and Jereoen > have each done yeoman?s jobs in ensuring that R can be built from source on > Windows and that packages can be developed for all OSS on Windows?even > Solaris SPARC. > > I am much less aware of Thomas?s work, and I?ll gladly take the blame for > it, but I haven?t seen an accessible tool chain built by him which would > allow me, the Windows developer, to build R and all packages from source on > a native Windows box. Have I just missed it? If not, is that planned? If > R-core switches the official Windows toolchain, where does that leave us? > > Thank you, > > Avi > > -- > Sent from Gmail Mobile > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >[[alternative HTML version deleted]]
Avraham Adler
2021-Aug-23 22:03 UTC
[Rd] [External] Re: Update on rtools4 and ucrt support
On Tue, Aug 24, 2021 at 12:47 AM Simon Urbanek <simon.urbanek at r-project.org> wrote:> Avi, > > please see the announcement: > > > https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html > > the documentation is in > > > https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html > > Cheers, > Simon > > > > > On Aug 24, 2021, at 8:34 AM, Avraham Adler <avraham.adler at gmail.com> > wrote: > > On Mon, Aug 23, 2021 at 11:09 PM <luke-tierney at uiowa.edu> wrote: > > On Mon, 23 Aug 2021, Duncan Murdoch wrote: > > On 23/08/2021 8:15 a.m., jan Vitek via R-devel wrote: > > Hi Jeroen, > > I mostly lurk on this list, but I was struck by your combative tone. > > To pick on two random bits: > > ? a 6gb tarball with manually built things on his personal machine? > > > ? a black-box system that is so opaque and complex that only one person > knows how it works, and would make it much more difficult for > students, universities, and other organisations to build R packages > and libraries on Windows? > > > > Tomas? tool chain isn't a blackbox, it has copious documentation (see > > [1]) > > and builds on any machine thanks to the provided docker container. > > This is not to criticise your work which has its unique strengths, but > > to > > state the obvious: these strengths are best discussed without passion > based on factually accurate descriptions. > > > I agree with Jan. I'm not sure a discussion in this forum would be > > fruitful, > > but I really wish Jeroen and Tomas would get together, aiming to merge > > their > > toolchains, keeping the best aspects of both. > > I haven't been involved in the development of either one, but have been > > a > > "victim" of the two chain rivalry, because the rgl package is not easy > > to > > build. I get instructions from each of them on how to do the build, and > those instructions for one toolchain generally break the build on the > > other > > one. While it is probably possible to detect the toolchain and have the > build adapt to whichever one is in use, it would be a lot easier for me > > (and > > I imagine every other maintainer of a package using external libs) if I > > just > > had to follow one set of instructions. > > Duncan Murdoch > > > Here are just a few comments from my perspective (I am an R-core > member, but am not part of the CRAN team and do only very limited work > on Windows). Other R-core members may have different perspectives and > insights. > > One bit of background: dealing with encoding issues on Windows has > been taking an unsustainable amount of R-core resources for some time > now. Tomas Kalibera has been taking the lead on trying to address > these issues in the existing framework, but this means he has not had > the time to make any of the many other valuable and important > contributions he could make. The only viable way forward is to move to > a Windows tool chain that supports UTF-8 as the C library current > encoding via the Windows UCRT framework. > > Tomas Kalibera has, on behalf of all of R core and in > coordination with CRAN, been looking for a way forward for some > time and has reported on the progress in several blog posts at > https://developer.r-project.org/Blog/public/. This has lead to > the development of the MXE-based UCRT tool chain, which is now > well tested and ready for deployment. Checks using the UCRT tool > chain have been part of the CRAN check process for a while. I > believe CRAN plans to switch R-devel checks and builds to the > UCRT tool chain during the upcoming CRAN downtime. I expect there > will be some communication from CRAN on this soon, including on > any issues in supporting binaries for both R-devel and R-patched. > > In putting together something as large as a tool chain there will > always be many choices, each with advantages and disadvantages. Some > things may be advantages in some settings and not others. Taking just > one case in point: Cross compilation. This is likely to be a better > approach for CRAN in the future and is supported by the MXE framework > on which the new tool chain is based. > > The much more recent changes in rtools4 to support UCRT are at this > point not yet as well tested as the new tool chain. Once these changes > to rtools4 mature, and if binary compatibility can be assured, then > having a second tool chain may be useful in some cases. But if there > are incompatibilities then it will be up to rtools4 to keep up with > the tool chain used by CRAN. On the other, contributing to improving > the MXE-based tool chain may be a better investment of time. > > Best, > > luke > > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > > > -- > Luke Tierney > Ralph E. Wareham Professor of Mathematical Sciences > University of Iowa Phone: 319-335-3386 > Department of Statistics and Fax: 319-335-3017 > Actuarial Science > 241 Schaeffer Hall email: luke-tierney at uiowa.edu > <luke-tierney at uiowa.edu> > Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu > > > Thank you, Dr. Tierney. However, I am concerned about the not-insignificant > number of us who for various reasons can only do our development on > Windows. Rtools has been the official tool chain with which to build > windows for the at least 20 years I have been using R (yes, a babe in the > woods compared to most, but not a complete neophyte). Duncan and Jereoen > have each done yeoman?s jobs in ensuring that R can be built from source on > Windows and that packages can be developed for all OSS on Windows?even > Solaris SPARC. > > I am much less aware of Thomas?s work, and I?ll gladly take the blame for > it, but I haven?t seen an accessible tool chain built by him which would > allow me, the Windows developer, to build R and all packages from source on > a native Windows box. Have I just missed it? If not, is that planned? If > R-core switches the official Windows toolchain, where does that leave us? > > Thank you, > > Avi > > -- > Sent from Gmail Mobile > >Thank you, Simon. That was valuable. Skimming that quickly, I get a bit concerned. I?ve been building from source and then using OpenBLAS in my R source for many, many years now, and it looks like its support is tenuous in the experimental chain. Similarly with packages like nloptr, where I build NLOPT 2.6+ and have adjusted Jelmer?s code for it to work in R for Windows. I maintain packages with Fortran/OpenMP and Rcpp(Parallel). I hope that should the decision be made to switch, it will be done when the build process is more streamlined, especially for some fundamental packages. That being said, I must take the opportunity again to thank R-core, Tomas, Jeroen, Duncan among many others who have built an infrastructure that allows amateur programmers to contribute to the statistical infrastructure. Thanks again, Avi -- Sent from Gmail Mobile [[alternative HTML version deleted]]