R SIG Debian - Jun 2021 - LTO flags in 4.1.0 Ubuntu 21.04 Makeconf

Hello Dirk,

Thanks for the very helpful response!


Answering the points you made, not in order:
> | Would it be possible to remove the flags -flto=auto and
-ffat-lto-objects?
>
> Not likely for the package I ship. I tend to not second guess upstream, and
> have followed R Core pretty much without changes to the program for 20
years.
>
> If you think you need a locally modified version I may be able to help you
> build one.
Got it, and thank you for the offer! I can edit my Makeconf (everything 
works when I remove the LTO-related flags). I was hoping to make 
installation smoother for other Arrow and Ubuntu users, but maybe that's 
something users or the Arrow developers will need to tackle.


> but I would be interested to hear how others fared on Debian or Ubuntu
systems.
In my experience, installing Arrow has gotten a lot better over time 
(including the reinstall requirement; see 
https://cran.r-project.org/web/packages/arrow/vignettes/install.html). 
It's still the most finicky of the packages I install, but substantially 
improved.




I think I'm confused by this comment (because I don't know much about 
the packaging process).
> - the Debian and hence derived Ubuntu package of R 4.1.0 does*not*  enable
>    LTO yet, in other words I have not yet add --enable-lto to the configure
>    call and we use to default of 'nope'
I'm confused because the R 4.1.0 Debian versions have flags like:
CXX11FLAGS = -g -O2 
-ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.1.0=. 
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
-D_FORTIFY_SOURCE=2 -g $(LTO)

The R 4.1.0 Ubuntu 20.10 flags are:
CXX11FLAGS = -g -O2 
-fdebug-prefix-map=/build/r-base-2Aokdc/r-base-4.1.0=. 
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
-D_FORTIFY_SOURCE=2 -g $(LTO)

While the R 4.1.0 Ubuntu 21.04 flags are:
CXX11FLAGS = -g -O2 
-ffile-prefix-map=/build/r-base-aXXzqd/r-base-4.1.0=. -flto=auto 
-ffat-lto-objects -fstack-protector-strong -Wformat 
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g $(LTO)

Should the Ubuntu 21.04 LTO flags be different than the others?

> That is something you may need to discuss with either R Core upstream (for
> the Makeconf), or maybe with the Arrow team for how they build the
> package. And there could of course be a bug but I remind you that Arrow
> itself is on CRAN.
The GCC docs note that the -flto flag will enable LTO, but it can be 
disabled by passing -fno-lto in the link command 
(https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-flto). 
I'll see if the Arrow folks are able to set -fno-lto.



If it's at all helpful, here's a reproducible example using Rocker 
(thank you for making this easy to troubleshoot across R installations).

1. Download the Makeconf file with these LTO flags 
(https://issues.apache.org/jira/secure/attachment/13025950/Makeconf_local) 
or extract it from the Ubuntu 21.04 deb file 
(https://cloud.r-project.org/bin/linux/ubuntu/hirsute-cran40/r-base-core_4.1.0-1.2104.0_amd64.deb).
2. Start docker: docker run -e "LIBARROW_BINARY=false" -e 
"ARROW_R_DEV=true" -e "LIBARROW_MINIMAL=true" --rm -ti
rocker/r-base:4.1.0
3. In a different terminal, find the docker container ID: docker ps
4. Copy Makeconf: docker cp Makeconf_local <container ID>:/etc/R/Makeconf
5. In the docker R:
 ??? tools::Rcmd("config CXX11FLAGS") to see the new flags
 ??? install.packages("arrow") now fails

My challenges are with the Makeconf flags packaged with Ubuntu 21.04's R 
4.1.0, but the example above also fails when the same Makeconf file is 
copied into the rocker/r-base:4.0.5 image.


Thanks,
Karl



On 6/1/21 3:50 PM, Dirk Eddelbuettel wrote:
> Two more follow-ups if I may:
>
> - the Debian and hence derived Ubuntu package of R 4.1.0 does *not* enable
>    LTO yet, in other words I have not yet add --enable-lto to the configure
>    call and we use to default of 'nope'
> - my dev box is still 20.10; I only run 21.04 on one small machine where I
>    have not tried to build arrow; but under 20.10 + R 4.1.0 + arrow 4.0.?
the
>    two step dance that they force on us works
>
> Dirk
>

	[[alternative HTML version deleted]]

Hi Karl,

On 1 June 2021 at 19:12, Karl Dunkle Werner wrote:
| Thanks for the very helpful response!

Thanks for weaving the threads back together.

| Got it, and thank you for the offer! I can edit my Makeconf (everything 
| works when I remove the LTO-related flags). I was hoping to make 
| installation smoother for other Arrow and Ubuntu users, but maybe that's 
| something users or the Arrow developers will need to tackle.

Yes indeed. One difficulty is that package authors can generally not override
what is in R's makeconf. As e.g. with my Rcpp maintainer hat on I have found
that frustrating. But such is life with R: not everything can be altered on
the fly.
 
| I think I'm confused by this comment (because I don't know much about 
| the packaging process).
| > - the Debian and hence derived Ubuntu package of R 4.1.0 does*not* 
enable
| >    LTO yet, in other words I have not yet add --enable-lto to the
configure
| >    call and we use to default of 'nope'
| I'm confused because the R 4.1.0 Debian versions have flags like:
| CXX11FLAGS = -g -O2 
| -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.1.0=. 
| -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
| -D_FORTIFY_SOURCE=2 -g $(LTO)
| 
| The R 4.1.0 Ubuntu 20.10 flags are:
| CXX11FLAGS = -g -O2 
| -fdebug-prefix-map=/build/r-base-2Aokdc/r-base-4.1.0=. 
| -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
| -D_FORTIFY_SOURCE=2 -g $(LTO)
| 
| While the R 4.1.0 Ubuntu 21.04 flags are:
| CXX11FLAGS = -g -O2 
| -ffile-prefix-map=/build/r-base-aXXzqd/r-base-4.1.0=. -flto=auto 
| -ffat-lto-objects -fstack-protector-strong -Wformat 
| -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g $(LTO)
| 
| Should the Ubuntu 21.04 LTO flags be different than the others?

<shrug emoji here>

Bbut these come from two (or three) distinct sources
- default compiler settings plus distro settings (really two players)
- package changes

and here you only one of the three constant (R 4.1.0). I think the -flto=auto
(amd -ffat-lto-objects) may come from gcc itself.  I can assure you (and you
can test that by looking at the git log at salsa.debian.org under my
'edd'
handle for the r-base package) that the debian/rules files configuring the
build _has not changed_ recently and does not set either.  In particularly no
with/without lto settings.  It's all 'without lto by default' from
my end. So
back to ... distribution and compiler changes. And I have learned in the 20+
years here to trust my colleagues looking after gcc et al.

Also, and just to be plain, recall that $(LTO) is calling a variable which
may, or may not, bet set.  When I look at my most recent package build logs
I see the compilation (in the long log) commence with

make[4]: Entering directory '/build/r-base-4.1.0/src/main'
gcc -std=gnu99 -I../../src/extra  -I. -I../../src/include -I../../src/include  
-I../../src/nmath -DHAVE_CONFIG_H   -fopenmp -fpic  -g -O2
-ffile-prefix-map=/build/r-base-4.1.0=. -fstack-protector-strong -Wformat
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c CommandLineArgs.c
-o CommandLineArgs.o

No LTO or -flto here (or in the rest of the file). I may of course have
missed something.

| My challenges are with the Makeconf flags packaged with Ubuntu 21.04's R 
| 4.1.0, but the example above also fails when the same Makeconf file is 
| copied into the rocker/r-base:4.0.5 image.

I have not looked as deeply into this as you but I can assure that I myself
of the maintainer of 'r-base' have been deeply and profoundly frustrated
by
the inability of this one package to build properly.  If you can improve that
without heavy surgery, side effects to other packages, or other shenanigans I
would be all ears.

Cheers, Dirk

-- 
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org