R devel - Jun 2018 - Byte-compilation failure on different architectures / low-memory systems

As you may know, I look after the R package for Debian. My fellow Debianers
make me follow a specific protocol -- a so-called "transition" in
which all
dependent packages on an identified potential breakge are rebuilt under the
new (potentially breaking) change. We started adding an r-api-3.4 tag last
(major) release. Now it is r-api-3.5 and the transition got started with the
green light from the release team last Friday.

Now one build failure occurred for one of the (old, effectively litte
maintained) RMetrics packages: fBasics. It blows up at the byte-compilation
step on at least four (older, smaller) architectures: mips, mipsel, arm64
(ubuntu), ppc64el.  More at https://bugs.debian.org/900756 where we also
worked out the "solution" of suppressing byte-compilation at
installation.

Luke, Tomas, ...: Would it help you to get access to such hardware?

We may get you some sort of "guest pass" access to porter machines.
Else I
could try but I have my hands plenty full with the transition (having built
[and finally transferred to our Debian Gitlab instance] 20+ package during
day two of R/Finance...).

Cheers, Dirk

PS I CC'ed the bug report, if any follow-up keep the CC it gets logged
there.

PPS Happy to discuss / help off-list too, of course.

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

I believe a reproducible example is to simply have a very large object
defined literally (i.e. through structure() etc) in ./R/ . For
example, fBasics has a couple of files test-jbLM.R and test-jbTable.R
which are about 500 KB. For a given amount of RAM, I believe any
sufficiently large file will also break a package if byte compilation
is not turned off, regardless of architecture. A similar issue
occurred with package ddalpha, though the maintainer has since fixed
that I believe.

On 5 June 2018 at 01:14, Dirk Eddelbuettel <edd at debian.org>
wrote:
>
> As you may know, I look after the R package for Debian. My fellow Debianers
> make me follow a specific protocol -- a so-called "transition" in
which all
> dependent packages on an identified potential breakge are rebuilt under the
> new (potentially breaking) change. We started adding an r-api-3.4 tag last
> (major) release. Now it is r-api-3.5 and the transition got started with
the
> green light from the release team last Friday.
>
> Now one build failure occurred for one of the (old, effectively litte
> maintained) RMetrics packages: fBasics. It blows up at the byte-compilation
> step on at least four (older, smaller) architectures: mips, mipsel, arm64
> (ubuntu), ppc64el.  More at https://bugs.debian.org/900756 where we also
> worked out the "solution" of suppressing byte-compilation at
installation.
>
> Luke, Tomas, ...: Would it help you to get access to such hardware?
>
> We may get you some sort of "guest pass" access to porter
machines. Else I
> could try but I have my hands plenty full with the transition (having built
> [and finally transferred to our Debian Gitlab instance] 20+ package during
> day two of R/Finance...).
>
> Cheers, Dirk
>
> PS I CC'ed the bug report, if any follow-up keep the CC it gets logged
there.
>
> PPS Happy to discuss / help off-list too, of course.
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

Hi Dirk,

thanks for the report. Access to the test system is not necessary, the 
memory requirements of the byte-code compiler are usually 
platform-independent and specifically with this package I can reproduce 
they are very high. We'll have a look what we can do, certainly there 
should at least be a way to recover and use the uncompiled version when 
memory allocation fails, this is already done by the JIT but not when 
compiling during installation. Before R or the package is patched, the 
only workaround for memory constrained systems is probably to disable 
byte-compilation of this package, as I read you are doing already.

Best
Tomas

On 06/04/2018 05:14 PM, Dirk Eddelbuettel wrote:
> As you may know, I look after the R package for Debian. My fellow Debianers
> make me follow a specific protocol -- a so-called "transition" in
which all
> dependent packages on an identified potential breakge are rebuilt under the
> new (potentially breaking) change. We started adding an r-api-3.4 tag last
> (major) release. Now it is r-api-3.5 and the transition got started with
the
> green light from the release team last Friday.
>
> Now one build failure occurred for one of the (old, effectively litte
> maintained) RMetrics packages: fBasics. It blows up at the byte-compilation
> step on at least four (older, smaller) architectures: mips, mipsel, arm64
> (ubuntu), ppc64el.  More at https://bugs.debian.org/900756 where we also
> worked out the "solution" of suppressing byte-compilation at
installation.
>
> Luke, Tomas, ...: Would it help you to get access to such hardware?
>
> We may get you some sort of "guest pass" access to porter
machines. Else I
> could try but I have my hands plenty full with the transition (having built
> [and finally transferred to our Debian Gitlab instance] 20+ package during
> day two of R/Finance...).
>
> Cheers, Dirk
>
> PS I CC'ed the bug report, if any follow-up keep the CC it gets logged
there.
>
> PPS Happy to discuss / help off-list too, of course.
>

On 4 June 2018 at 20:06, Tomas Kalibera wrote:
| thanks for the report. Access to the test system is not necessary, the 
| memory requirements of the byte-code compiler are usually 
| platform-independent and specifically with this package I can reproduce 
| they are very high. We'll have a look what we can do, certainly there 
| should at least be a way to recover and use the uncompiled version when 
| memory allocation fails, this is already done by the JIT but not when 
| compiling during installation. Before R or the package is patched, the 
| only workaround for memory constrained systems is probably to disable 
| byte-compilation of this package, as I read you are doing already.

Yes. And as a shortcut, we just turned it off unconditionally, ie on all
build architectures.  Worked fine as per

   https://buildd.debian.org/status/package.php?p=fbasics

it has been built everywhere where we have R 3.5.0 (some 20 or so platforms).

The fix you suggest sounds ideal: if possible recover, and maybe WARN.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org