R devel - Mar 2020 - Rebuilding and re-checking of downstream dependencies on CRAN Mac build machines

I have two questions about the CRAN machines that build binary
packages for Mac. When a new version of a package is released,
  (A) Do the downstream dependencies get re-checked?
  (B) Do the downstream dependencies get re-built?

I have heard (but do not know for sure) that the answer to (A) is no,
the downstream dependencies do not get rechecked.
>From publicly available information on the CRAN web server, it looks
like the answer to (B) is also no, the downstream dependencies do not
get rebuilt. Looking at
https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/, I see
the following dates for these binary packages:

- Rcpp_1.0.4.tgz: 2020-03-18
- httpuv_1.5.2.tgz: 2019-09-12
- dplyr_0.8.5.tgz: 2020-03-08

Rcpp was released recently, and httpuv and dplyr (which are downstream
dependencies of Rcpp) have older dates, which indicates that these
binary packages were not rebuilt when Rcpp was released.

In my particular case, I'm interested in the httpuv package (which I
maintain). I and several others have not been able to get the CRAN
version of httpuv to compile using the CRAN version of Rcpp on Mac.
(It seems to compile fine on other platforms.) I have heard from
maintainers of other Rcpp-dependent packages that they also can't get
their packages to compile on Mac, using both the default Mac compiler
toolchain and the CRAN-recommended toolchain, which uses clang 7.

For more technical details about the cause of breakage, see:
https://github.com/RcppCore/Rcpp/issues/1060
https://github.com/rstudio/httpuv/issues/260

If the CRAN Mac build machine is indeed able to build httpuv against
the current version of Rcpp, it would be really helpful to have more
information about the system configuration. If it is not able to
rebuild httpuv and other packages against Rcpp, then this is a
problem. Among other things, it prevents people from building their
packages from source using CRAN versions of packages, and it also
means that none of these packages can release a new version, because
the binaries can't be built on Mac.

-Winston

Winston,

the Mac CRAN build builds a package only if either is true:
1) the package has not passed checks
2) there is a new version of the package since last successful build+check

The old build machine doesn't have the capacity to do full re-builds (it
would take over 24h - currently the nightly build of packages takes 16-22
hours), but we're currently building a new setup for R 4.0.0 on new hardware
and as a part of it we are planning to setup a "mac-builder" similar
to what is currently available for Windows.

That said, I have run httpuv by hand on the CRAN build machine (against Rcpp
1.0.4) and I saw no issues. I have seen the discussion on Rcpp, but so far no
one actually posted details of what is breaking (nor do your links include any
actual details on this). I'd love to help, but the lack fo a useful report
makes this impossible. If you have any actual leads, please post them. The CRAN
machine uses the tools that are available on CRAN:
https://cran.r-project.org/bin/macosx/tools/ (clang-7 and gfortran-6.1 for
3.6.x)

Cheers,
Simon

> On 27/03/2020, at 7:38 AM, Winston Chang <winstonchang1 at gmail.com>
wrote:
> 
> I have two questions about the CRAN machines that build binary
> packages for Mac. When a new version of a package is released,
>  (A) Do the downstream dependencies get re-checked?
>  (B) Do the downstream dependencies get re-built?
> 
> I have heard (but do not know for sure) that the answer to (A) is no,
> the downstream dependencies do not get rechecked.
> 
> From publicly available information on the CRAN web server, it looks
> like the answer to (B) is also no, the downstream dependencies do not
> get rebuilt. Looking at
> https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/, I see
> the following dates for these binary packages:
> 
> - Rcpp_1.0.4.tgz: 2020-03-18
> - httpuv_1.5.2.tgz: 2019-09-12
> - dplyr_0.8.5.tgz: 2020-03-08
> 
> Rcpp was released recently, and httpuv and dplyr (which are downstream
> dependencies of Rcpp) have older dates, which indicates that these
> binary packages were not rebuilt when Rcpp was released.
> 
> In my particular case, I'm interested in the httpuv package (which I
> maintain). I and several others have not been able to get the CRAN
> version of httpuv to compile using the CRAN version of Rcpp on Mac.
> (It seems to compile fine on other platforms.) I have heard from
> maintainers of other Rcpp-dependent packages that they also can't get
> their packages to compile on Mac, using both the default Mac compiler
> toolchain and the CRAN-recommended toolchain, which uses clang 7.
> 
> For more technical details about the cause of breakage, see:
> https://github.com/RcppCore/Rcpp/issues/1060
> https://github.com/rstudio/httpuv/issues/260
> 
> If the CRAN Mac build machine is indeed able to build httpuv against
> the current version of Rcpp, it would be really helpful to have more
> information about the system configuration. If it is not able to
> rebuild httpuv and other packages against Rcpp, then this is a
> problem. Among other things, it prevents people from building their
> packages from source using CRAN versions of packages, and it also
> means that none of these packages can release a new version, because
> the binaries can't be built on Mac.
> 
> -Winston
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

If I do install.packages("dplyr", type = "source"), I see:

Installing package into ?/Users/hadley/R?
(as ?lib? is unspecified)
trying URL 'https://cran.rstudio.com/src/contrib/dplyr_0.8.5.tar.gz'
Content type 'application/x-gzip' length 1378766 bytes (1.3 MB)
=================================================downloaded 1.3 MB

* installing *source* package ?dplyr? ...
** package ?dplyr? successfully unpacked and MD5 sums checked
** using staged installation
** libs
ccache clang++ -Qunused-arguments
 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG
-I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR
-DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT
-DBOOST_NO_AUTO_PTR  -I"/Users/hadley/R/BH/include"
-I"/Users/hadley/R/plogr/include"
-I"/Users/hadley/R/Rcpp/include"
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
-I/usr/local/include  -fPIC  -Wall -g -O2  -c RcppExports.cpp -o
RcppExports.o
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/dplyr.h:4:
In file included from ../inst/include/dplyr/main.h:6:
In file included from ../inst/include/dplyr/workarounds/static_assert.h:17:
In file included from /Users/hadley/R/BH/include/boost/config.hpp:57:
In file included from
/Users/hadley/R/BH/include/boost/config/platform/macos.hpp:28:
In file included from
/Users/hadley/R/BH/include/boost/config/detail/posix_features.hpp:18:
In file included from
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:655:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/gethostuuid.h:39:17:
error: unknown type name 'uuid_t'
int gethostuuid(uuid_t, const struct timespec *)
__OSX_AVAILABLE_STARTING(__MAC_10_5, __IPHONE_NA);
                ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/dplyr.h:4:
In file included from ../inst/include/dplyr/main.h:6:
In file included from ../inst/include/dplyr/workarounds/static_assert.h:17:
In file included from /Users/hadley/R/BH/include/boost/config.hpp:57:
In file included from
/Users/hadley/R/BH/include/boost/config/platform/macos.hpp:28:
In file included from
/Users/hadley/R/BH/include/boost/config/detail/posix_features.hpp:18:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:662:27:
error: unknown type name 'uuid_t'; did you mean 'uid_t'?
int      getsgroups_np(int *, uuid_t);
                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
note: 'uid_t' declared here
typedef __darwin_uid_t        uid_t;
                              ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/dplyr.h:4:
In file included from ../inst/include/dplyr/main.h:6:
In file included from ../inst/include/dplyr/workarounds/static_assert.h:17:
In file included from /Users/hadley/R/BH/include/boost/config.hpp:57:
In file included from
/Users/hadley/R/BH/include/boost/config/platform/macos.hpp:28:
In file included from
/Users/hadley/R/BH/include/boost/config/detail/posix_features.hpp:18:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:664:27:
error: unknown type name 'uuid_t'; did you mean 'uid_t'?
int      getwgroups_np(int *, uuid_t);
                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
note: 'uid_t' declared here
typedef __darwin_uid_t        uid_t;
                              ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/dplyr.h:4:
In file included from ../inst/include/dplyr/main.h:6:
In file included from ../inst/include/dplyr/workarounds/static_assert.h:17:
In file included from /Users/hadley/R/BH/include/boost/config.hpp:57:
In file included from
/Users/hadley/R/BH/include/boost/config/platform/macos.hpp:28:
In file included from
/Users/hadley/R/BH/include/boost/config/detail/posix_features.hpp:18:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:727:31:
error: unknown type name 'uuid_t'; did you mean 'uid_t'?
int      setsgroups_np(int, const uuid_t);
                                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
note: 'uid_t' declared here
typedef __darwin_uid_t        uid_t;
                              ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/dplyr.h:4:
In file included from ../inst/include/dplyr/main.h:6:
In file included from ../inst/include/dplyr/workarounds/static_assert.h:17:
In file included from /Users/hadley/R/BH/include/boost/config.hpp:57:
In file included from
/Users/hadley/R/BH/include/boost/config/platform/macos.hpp:28:
In file included from
/Users/hadley/R/BH/include/boost/config/detail/posix_features.hpp:18:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:729:31:
error: unknown type name 'uuid_t'; did you mean 'uid_t'?
int      setwgroups_np(int, const uuid_t);
                                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
note: 'uid_t' declared here
typedef __darwin_uid_t        uid_t;
                              ^
In file included from RcppExports.cpp:5:
In file included from ./../inst/include/dplyr_types.h:4:
In file included from ../inst/include/dplyr/data/GroupedDataFrame.h:8:
In file included from ../inst/include/tools/SymbolMap.h:4:
In file included from ../inst/include/tools/hash.h:10:
In file included from /Users/hadley/R/BH/include/boost/unordered_map.hpp:17:
In file included from
/Users/hadley/R/BH/include/boost/unordered/unordered_map.hpp:19:
In file included from /Users/hadley/R/BH/include/boost/move/move.hpp:30:
In file included from /Users/hadley/R/BH/include/boost/move/iterator.hpp:27:
/Users/hadley/R/BH/include/boost/move/detail/iterator_traits.hpp:29:1:
warning: inline namespaces are a C++11 feature [-Wc++11-inline-namespace]
BOOST_MOVE_STD_NS_BEG
^
/Users/hadley/R/BH/include/boost/move/detail/std_ns_begin.hpp:18:34: note:
expanded from macro 'BOOST_MOVE_STD_NS_BEG'
   #define BOOST_MOVE_STD_NS_BEG _LIBCPP_BEGIN_NAMESPACE_STD
                                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:866:53:
note: expanded from macro '_LIBCPP_BEGIN_NAMESPACE_STD'
#define _LIBCPP_BEGIN_NAMESPACE_STD namespace std { inline namespace
_LIBCPP_ABI_NAMESPACE {
                                                    ^
1 warning and 5 errors generated.
make: *** [RcppExports.o] Error 1
ERROR: compilation failed for package ?dplyr?
* removing ?/Users/hadley/R/dplyr?
* restoring previous ?/Users/hadley/R/dplyr?

Hadley


On Thu, Mar 26, 2020 at 4:05 PM Simon Urbanek <simon.urbanek at
r-project.org>
wrote:
> Winston,
>
> the Mac CRAN build builds a package only if either is true:
> 1) the package has not passed checks
> 2) there is a new version of the package since last successful build+check
>
> The old build machine doesn't have the capacity to do full re-builds
(it
> would take over 24h - currently the nightly build of packages takes 16-22
> hours), but we're currently building a new setup for R 4.0.0 on new
> hardware and as a part of it we are planning to setup a
"mac-builder"
> similar to what is currently available for Windows.
>
> That said, I have run httpuv by hand on the CRAN build machine (against
> Rcpp 1.0.4) and I saw no issues. I have seen the discussion on Rcpp, but so
> far no one actually posted details of what is breaking (nor do your links
> include any actual details on this). I'd love to help, but the lack fo
a
> useful report makes this impossible. If you have any actual leads, please
> post them. The CRAN machine uses the tools that are available on CRAN:
> https://cran.r-project.org/bin/macosx/tools/ (clang-7 and gfortran-6.1
> for 3.6.x)
>
> Cheers,
> Simon
>
>
> > On 27/03/2020, at 7:38 AM, Winston Chang <winstonchang1 at
gmail.com>
> wrote:
> >
> > I have two questions about the CRAN machines that build binary
> > packages for Mac. When a new version of a package is released,
> >  (A) Do the downstream dependencies get re-checked?
> >  (B) Do the downstream dependencies get re-built?
> >
> > I have heard (but do not know for sure) that the answer to (A) is no,
> > the downstream dependencies do not get rechecked.
> >
> > From publicly available information on the CRAN web server, it looks
> > like the answer to (B) is also no, the downstream dependencies do not
> > get rebuilt. Looking at
> > https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/, I see
> > the following dates for these binary packages:
> >
> > - Rcpp_1.0.4.tgz: 2020-03-18
> > - httpuv_1.5.2.tgz: 2019-09-12
> > - dplyr_0.8.5.tgz: 2020-03-08
> >
> > Rcpp was released recently, and httpuv and dplyr (which are downstream
> > dependencies of Rcpp) have older dates, which indicates that these
> > binary packages were not rebuilt when Rcpp was released.
> >
> > In my particular case, I'm interested in the httpuv package (which
I
> > maintain). I and several others have not been able to get the CRAN
> > version of httpuv to compile using the CRAN version of Rcpp on Mac.
> > (It seems to compile fine on other platforms.) I have heard from
> > maintainers of other Rcpp-dependent packages that they also can't
get
> > their packages to compile on Mac, using both the default Mac compiler
> > toolchain and the CRAN-recommended toolchain, which uses clang 7.
> >
> > For more technical details about the cause of breakage, see:
> > https://github.com/RcppCore/Rcpp/issues/1060
> > https://github.com/rstudio/httpuv/issues/260
> >
> > If the CRAN Mac build machine is indeed able to build httpuv against
> > the current version of Rcpp, it would be really helpful to have more
> > information about the system configuration. If it is not able to
> > rebuild httpuv and other packages against Rcpp, then this is a
> > problem. Among other things, it prevents people from building their
> > packages from source using CRAN versions of packages, and it also
> > means that none of these packages can release a new version, because
> > the binaries can't be built on Mac.
> >
> > -Winston
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
http://hadley.nz

	[[alternative HTML version deleted]]

Simon,

The link I provided to the httpuv issue has detailed information about
the error:
  https://github.com/rstudio/httpuv/issues/260

The error occurs when I run the following (although note, depending on
which mirror you use, the recent CRAN server issues may result in
problems downloading the packages):
  install.packages("Rcpp")
  install.packages("httpuv", type = "source")

Here's the full output:
  https://gist.github.com/wch/c70b438381c9d2a8b1f917b054e0ba7e

Here's an example of the errors:

In file included from staticpath.cpp:1:
In file included from ./staticpath.h:7:
In file included from /Users/winston/R/3.6/BH/include/boost/optional.hpp:15:
In file included from
/Users/winston/R/3.6/BH/include/boost/optional/optional.hpp:28:
In file included from
/Users/winston/R/3.6/BH/include/boost/core/addressof.hpp:17:
In file included from /Users/winston/R/3.6/BH/include/boost/config.hpp:57:
In file included from
/Users/winston/R/3.6/BH/include/boost/config/platform/macos.hpp:28:
In file included from
/Users/winston/R/3.6/BH/include/boost/config/detail/posix_features.hpp:18:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:664:27:
error: unknown type name 'uuid_t'; did you mean 'uid_t'?
int      getwgroups_np(int *, uuid_t);
                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
note: 'uid_t' declared here
typedef __darwin_uid_t        uid_t;
                              ^

This issue describes the same problem:
  https://github.com/RcppCore/Rcpp/issues/1046

And it has been fixed in the development version of Rcpp by:
   https://github.com/RcppCore/Rcpp/pull/1047

I know that Kevin Ushey has tried building with the CRAN-recommended
clang 7 toolchain and encountered the same errors. I haven't done that
yet myself, though.

-Winston

On Thu, Mar 26, 2020 at 4:00 PM Simon Urbanek
<simon.urbanek at r-project.org> wrote:
>
> Winston,
>
> the Mac CRAN build builds a package only if either is true:
> 1) the package has not passed checks
> 2) there is a new version of the package since last successful build+check
>
> The old build machine doesn't have the capacity to do full re-builds
(it would take over 24h - currently the nightly build of packages takes 16-22
hours), but we're currently building a new setup for R 4.0.0 on new hardware
and as a part of it we are planning to setup a "mac-builder" similar
to what is currently available for Windows.
>
> That said, I have run httpuv by hand on the CRAN build machine (against
Rcpp 1.0.4) and I saw no issues. I have seen the discussion on Rcpp, but so far
no one actually posted details of what is breaking (nor do your links include
any actual details on this). I'd love to help, but the lack fo a useful
report makes this impossible. If you have any actual leads, please post them.
The CRAN machine uses the tools that are available on CRAN:
https://cran.r-project.org/bin/macosx/tools/ (clang-7 and gfortran-6.1 for
3.6.x)
>
> Cheers,
> Simon
>
>
> > On 27/03/2020, at 7:38 AM, Winston Chang <winstonchang1 at
gmail.com> wrote:
> >
> > I have two questions about the CRAN machines that build binary
> > packages for Mac. When a new version of a package is released,
> >  (A) Do the downstream dependencies get re-checked?
> >  (B) Do the downstream dependencies get re-built?
> >
> > I have heard (but do not know for sure) that the answer to (A) is no,
> > the downstream dependencies do not get rechecked.
> >
> > From publicly available information on the CRAN web server, it looks
> > like the answer to (B) is also no, the downstream dependencies do not
> > get rebuilt. Looking at
> > https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/, I see
> > the following dates for these binary packages:
> >
> > - Rcpp_1.0.4.tgz: 2020-03-18
> > - httpuv_1.5.2.tgz: 2019-09-12
> > - dplyr_0.8.5.tgz: 2020-03-08
> >
> > Rcpp was released recently, and httpuv and dplyr (which are downstream
> > dependencies of Rcpp) have older dates, which indicates that these
> > binary packages were not rebuilt when Rcpp was released.
> >
> > In my particular case, I'm interested in the httpuv package (which
I
> > maintain). I and several others have not been able to get the CRAN
> > version of httpuv to compile using the CRAN version of Rcpp on Mac.
> > (It seems to compile fine on other platforms.) I have heard from
> > maintainers of other Rcpp-dependent packages that they also can't
get
> > their packages to compile on Mac, using both the default Mac compiler
> > toolchain and the CRAN-recommended toolchain, which uses clang 7.
> >
> > For more technical details about the cause of breakage, see:
> > https://github.com/RcppCore/Rcpp/issues/1060
> > https://github.com/rstudio/httpuv/issues/260
> >
> > If the CRAN Mac build machine is indeed able to build httpuv against
> > the current version of Rcpp, it would be really helpful to have more
> > information about the system configuration. If it is not able to
> > rebuild httpuv and other packages against Rcpp, then this is a
> > problem. Among other things, it prevents people from building their
> > packages from source using CRAN versions of packages, and it also
> > means that none of these packages can release a new version, because
> > the binaries can't be built on Mac.
> >
> > -Winston
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
>