R devel - Mar 2022 - Making CRAN memory access checks more accessible?

Dear colleagues,

Two days after successfully submitting a package to CRAN I received a
message about "additional issues" with the package's C++ code
(clang-UBSAN to be precise) with a two-week deadline to resolve. While
attempting to somewhat blind-foldedly fix the problem I was wondering
whether it is sensible and feasible for base R to:

1. Implement/expose all these memory-related tests (c.f.
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access)
to package developers e.g. via options to R CMD check, much like
--use-gct or --use-valgrind are already? Or via a script etc.?

or

2. Expand the chapter
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access
with unequivocal and straightforward instructions how to setup and run
these tests locally on different platforms? I believe that the current
version of the manual is inaccessible to anybody but hardcore C/C++
developers while there is a broader spectrum of ppl able to write some
C without the deep understanding of the internals.

While I noticed that a similar problem has triggered some heat on
Twitter recently I independently decided to write to you all here to
ask the question above. I believe it might be rather difficult for
package contributors to adhere to tests which they are unable to
execute locally (or by a CI service). Alas, in the end it will end-up
with a developer playing package ping-pong with CRAN maintainers whose
time is a valuable resource.

Best wishes,
Michal

On 25/02/2022 11:31 a.m., Micha? Bojanowski wrote:
> Dear colleagues,
> 
> Two days after successfully submitting a package to CRAN I received a
> message about "additional issues" with the package's C++ code
> (clang-UBSAN to be precise) with a two-week deadline to resolve. While
> attempting to somewhat blind-foldedly fix the problem I was wondering
> whether it is sensible and feasible for base R to:
> 
> 1. Implement/expose all these memory-related tests (c.f.
>
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access)
> to package developers e.g. via options to R CMD check, much like
> --use-gct or --use-valgrind are already? Or via a script etc.?
Many users won't be able to run them.
> 
> or
> 
> 2. Expand the chapter
>
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access
> with unequivocal and straightforward instructions how to setup and run
> these tests locally on different platforms? I believe that the current
> version of the manual is inaccessible to anybody but hardcore C/C++
> developers while there is a broader spectrum of ppl able to write some
> C without the deep understanding of the internals.
I doubt if the instructions in WRE could be simplified and still work.

> 
> While I noticed that a similar problem has triggered some heat on
> Twitter recently I independently decided to write to you all here to
> ask the question above. I believe it might be rather difficult for
> package contributors to adhere to tests which they are unable to
> execute locally (or by a CI service). Alas, in the end it will end-up
> with a developer playing package ping-pong with CRAN maintainers whose
> time is a valuable resource.
I think R-hub offers UBSAN services.  Have you tried those?

Duncan Murdoch

valgrind will detect some of the memory issues that UBSAN does.  Here is
how you can use valgrind with the gdb debugger on Linux.  Use apt-get to
get valgrind and gdb if you have not yet installed them (If you have
Windows, install Microsoft's 'wsl2' and Ubuntu Linux and do this in
Ubuntu
windows.)

1. Configure your R build for valgrind as described in Writing R Extensions
section 4.3.2.
2. Run R with
    R --debugger=valgrind --debugger-args="--track-origins=yes --vgdb=full
--vgdb-error=0"
and any other R command line arguments you like (I often use --quiet and
--no-save).
You should see something like the following printed
  ==238== TO DEBUG THIS PROCESS USING GDB: start GDB like this
  ==238==   /path/to/gdb /home/bill/R-devel/R-build/bin/exec/R
  ==238== and then give GDB the following command
  ==238==   target remote |
/usr/lib/x86_64-linux-gnu/valgrind/../../bin/vgdb --pid=238
  ==238== --pid is optional if only one valgrind process is running
3.  In another window run gdb with that path to .../exec/R as its only
command line argument.
4.  On my copy of Ubuntu 20.04, vgdb is not in /usr/lib/... but is in
/usr/bin so
   target remote | vgdb
at the (gdb) prompt generally starts vgdb, valgrind's client for gdb.  Set
any break points you would like then issue the
   continue
command.

At this point R in the first window should start running.  It will break to
the debugger when valgrind detects a problem or when any of your
breakpoints are hit.  Control-C in the R window will also break to the
debugger.

The usual gdb commands will work.  There are some extra "monitor"
commands
supported
by vgdb.  E.g., at the (gdb) prompt
   monitor leak-check full
will describe all the memory leaks detected since the last time you asked
about them.
Look in

https://valgrind.org/docs/manual/mc-manual.html#mc-manual.monitor-commands
for other useful monitor commands.

-Bill

On Fri, Feb 25, 2022 at 8:31 AM Micha? Bojanowski <michal2992 at
gmail.com>
wrote:
> Dear colleagues,
>
> Two days after successfully submitting a package to CRAN I received a
> message about "additional issues" with the package's C++ code
> (clang-UBSAN to be precise) with a two-week deadline to resolve. While
> attempting to somewhat blind-foldedly fix the problem I was wondering
> whether it is sensible and feasible for base R to:
>
> 1. Implement/expose all these memory-related tests (c.f.
>
>
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access
> )
> to package developers e.g. via options to R CMD check, much like
> --use-gct or --use-valgrind are already? Or via a script etc.?
>
> or
>
> 2. Expand the chapter
>
>
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access
> with unequivocal and straightforward instructions how to setup and run
> these tests locally on different platforms? I believe that the current
> version of the manual is inaccessible to anybody but hardcore C/C++
> developers while there is a broader spectrum of ppl able to write some
> C without the deep understanding of the internals.
>
> While I noticed that a similar problem has triggered some heat on
> Twitter recently I independently decided to write to you all here to
> ask the question above. I believe it might be rather difficult for
> package contributors to adhere to tests which they are unable to
> execute locally (or by a CI service). Alas, in the end it will end-up
> with a developer playing package ping-pong with CRAN maintainers whose
> time is a valuable resource.
>
> Best wishes,
> Michal
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]

Thank you Bill! I'll test it out.
That's a kind of instruction I had in mind when I wrote about
extending the relevant part of "Writing R extensions"...
Best, Michal

On Mon, Feb 28, 2022 at 8:15 PM Bill Dunlap <williamwdunlap at gmail.com>
wrote:
>
> valgrind will detect some of the memory issues that UBSAN does.  Here is
how you can use valgrind with the gdb debugger on Linux.  Use apt-get to get
valgrind and gdb if you have not yet installed them (If you have Windows,
install Microsoft's 'wsl2' and Ubuntu Linux and do this in Ubuntu
windows.)
>
> 1. Configure your R build for valgrind as described in Writing R Extensions
section 4.3.2.
> 2. Run R with
>     R --debugger=valgrind --debugger-args="--track-origins=yes
--vgdb=full --vgdb-error=0"
> and any other R command line arguments you like (I often use --quiet and
--no-save).
> You should see something like the following printed
>   ==238== TO DEBUG THIS PROCESS USING GDB: start GDB like this
>   ==238==   /path/to/gdb /home/bill/R-devel/R-build/bin/exec/R
>   ==238== and then give GDB the following command
>   ==238==   target remote |
/usr/lib/x86_64-linux-gnu/valgrind/../../bin/vgdb --pid=238
>   ==238== --pid is optional if only one valgrind process is running
> 3.  In another window run gdb with that path to .../exec/R as its only
command line argument.
> 4.  On my copy of Ubuntu 20.04, vgdb is not in /usr/lib/... but is in
/usr/bin so
>    target remote | vgdb
> at the (gdb) prompt generally starts vgdb, valgrind's client for gdb. 
Set any break points you would like then issue the
>    continue
> command.
>
> At this point R in the first window should start running.  It will break to
the debugger when valgrind detects a problem or when any of your breakpoints are
hit.  Control-C in the R window will also break to the debugger.
>
> The usual gdb commands will work.  There are some extra "monitor"
commands supported
> by vgdb.  E.g., at the (gdb) prompt
>    monitor leak-check full
> will describe all the memory leaks detected since the last time you asked
about them.
> Look in
>   
https://valgrind.org/docs/manual/mc-manual.html#mc-manual.monitor-commands
> for other useful monitor commands.
>
> -Bill
>
> On Fri, Feb 25, 2022 at 8:31 AM Micha? Bojanowski <michal2992 at
gmail.com> wrote:
>>
>> Dear colleagues,
>>
>> Two days after successfully submitting a package to CRAN I received a
>> message about "additional issues" with the package's C++
code
>> (clang-UBSAN to be precise) with a two-week deadline to resolve. While
>> attempting to somewhat blind-foldedly fix the problem I was wondering
>> whether it is sensible and feasible for base R to:
>>
>> 1. Implement/expose all these memory-related tests (c.f.
>>
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access)
>> to package developers e.g. via options to R CMD check, much like
>> --use-gct or --use-valgrind are already? Or via a script etc.?
>>
>> or
>>
>> 2. Expand the chapter
>>
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-memory-access
>> with unequivocal and straightforward instructions how to setup and run
>> these tests locally on different platforms? I believe that the current
>> version of the manual is inaccessible to anybody but hardcore C/C++
>> developers while there is a broader spectrum of ppl able to write some
>> C without the deep understanding of the internals.
>>
>> While I noticed that a similar problem has triggered some heat on
>> Twitter recently I independently decided to write to you all here to
>> ask the question above. I believe it might be rather difficult for
>> package contributors to adhere to tests which they are unable to
>> execute locally (or by a CI service). Alas, in the end it will end-up
>> with a developer playing package ping-pong with CRAN maintainers whose
>> time is a valuable resource.
>>
>> Best wishes,
>> Michal
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel

On 28/02/2022 19:15, Bill Dunlap wrote:
> valgrind will detect some of the memory issues that UBSAN does.  
Only very few.  None of the current 43 CRAN packages with UBSAN issues 
have them detected by valgrind ... the valgrind overlap is more with 
ASAN issues (see 
https://github.com/google/sanitizers/wiki/AddressSanitizerComparisonOfMemoryTools).

The UB sanitizer is heavily tied to the compiler.  This has meant that 
new versions of gcc have found more and more issues and (most 
pertinently here) recent versions of clang find about twice as many 
issues as gcc 10/11.

UBSAN in theory runs on quite a range of platforms --- 
https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#supported-platforms 
-- which unfortunately does not say for which CPUs on those OS.  (Also, 
that is about LLVM clang and not Apple clang as normally used on macOS. 
  And gcc seems only to point to LLVM documentation for supported 
platforms, but its implementation clearly differs.)


-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford