On 3/12/21 7:43 PM, xiaoyan yu wrote:> I am writing C++ program based on R extensions and also try to test the > program with google address sanitizer. > > I thought if I don't protect the variable from the allocation API such as > Rf_allocVector, there will be a memory leak. However, the address sanitizer > didn't report it. Is my understanding correct? Or I will see the memory > leak only if I compile R source code with the address sanitizer.Yes, you should use special options for compilation and linking to use address sanitizer. See Writing R Extensions, section 4.3.3. If you allocate an R object using Rf_allocVector(), but don't protect it, it means this object is available for the garbage collector to reclaim. So it is not a memory leak. Memory leaks with a garbage collector are much less common than without, because if the program loses a pointer to some piece of memory, that piece will automatically be reclaimed (not leaked). Still, memory leaks are possible if the program forgets about a pointer to some piece of memory no longer needed, and keeps that pointer in say some global structure. Such memory leaks would not be found using address sanitizer. Address sanitizer/Undefined behavior sanitizer can sometimes find errors caused by that the program forgets to protect an R object, but this is relatively rare, as they don't understand R heap specifically, so you cannot assume that if you create such example, the error will always be found. Best Tomas> > Please help! > > Thanks, > Xiaoyan > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel
> Still, memory leaks are possible if the program forgets about a > pointer to some piece of memory no longer needed, and keeps that > pointer in say some global structure. Such memory leaks would not be > found using address sanitizer.We had a few cases of this in the past. Given the difficulty of tracing the leaking references, we wrote this package for taking snapshots of the R heap and finding dominators and shortest paths between nodes: Repo: https://github.com/r-lib/memtools Vignette: https://memtools.r-lib.org/articles/memtools.html One issue that complicates taking snapshots is that R doesn't expose the GC roots. In practice, only the precious list is needed I think. Would you consider a patch that allows retrieving the precious list for debugging purposes via a `.Internal()` call? Best, Lionel On 3/15/21, Tomas Kalibera <tomas.kalibera at gmail.com> wrote:> On 3/12/21 7:43 PM, xiaoyan yu wrote: >> I am writing C++ program based on R extensions and also try to test the >> program with google address sanitizer. >> >> I thought if I don't protect the variable from the allocation API such as >> Rf_allocVector, there will be a memory leak. However, the address >> sanitizer >> didn't report it. Is my understanding correct? Or I will see the memory >> leak only if I compile R source code with the address sanitizer. > > Yes, you should use special options for compilation and linking to use > address sanitizer. See Writing R Extensions, section 4.3.3. > > If you allocate an R object using Rf_allocVector(), but don't protect > it, it means this object is available for the garbage collector to > reclaim. So it is not a memory leak. > > Memory leaks with a garbage collector are much less common than without, > because if the program loses a pointer to some piece of memory, that > piece will automatically be reclaimed (not leaked). Still, memory leaks > are possible if the program forgets about a pointer to some piece of > memory no longer needed, and keeps that pointer in say some global > structure. Such memory leaks would not be found using address sanitizer. > > Address sanitizer/Undefined behavior sanitizer can sometimes find errors > caused by that the program forgets to protect an R object, but this is > relatively rare, as they don't understand R heap specifically, so you > cannot assume that if you create such example, the error will always be > found. > > Best > Tomas > >> >> Please help! >> >> Thanks, >> Xiaoyan >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >
Thank you all for your help. We embedded R in our program and found the memory in the process accumulated while our expectation is that the memory will go down after each R evaluation. I started to write a test program with only a few lines of R embedded codes and found the memory never went down even after R library is unloaded. Please find more details in the readme and test program at https://github.com/xiaoyanyuvt/RMemTest Thanks, Xiaoyan On Fri, Mar 19, 2021 at 2:21 PM Lionel Henry <lionel at rstudio.com> wrote:> > Still, memory leaks are possible if the program forgets about a > > pointer to some piece of memory no longer needed, and keeps that > > pointer in say some global structure. Such memory leaks would not be > > found using address sanitizer. > > We had a few cases of this in the past. Given the difficulty of > tracing the leaking references, we wrote this package for taking > snapshots of the R heap and finding dominators and shortest paths > between nodes: > > Repo: https://github.com/r-lib/memtools > Vignette: https://memtools.r-lib.org/articles/memtools.html > > One issue that complicates taking snapshots is that R doesn't expose > the GC roots. In practice, only the precious list is needed I think. > Would you consider a patch that allows retrieving the precious list > for debugging purposes via a `.Internal()` call? > > Best, > Lionel > > > On 3/15/21, Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > On 3/12/21 7:43 PM, xiaoyan yu wrote: > >> I am writing C++ program based on R extensions and also try to test the > >> program with google address sanitizer. > >> > >> I thought if I don't protect the variable from the allocation API such > as > >> Rf_allocVector, there will be a memory leak. However, the address > >> sanitizer > >> didn't report it. Is my understanding correct? Or I will see the memory > >> leak only if I compile R source code with the address sanitizer. > > > > Yes, you should use special options for compilation and linking to use > > address sanitizer. See Writing R Extensions, section 4.3.3. > > > > If you allocate an R object using Rf_allocVector(), but don't protect > > it, it means this object is available for the garbage collector to > > reclaim. So it is not a memory leak. > > > > Memory leaks with a garbage collector are much less common than without, > > because if the program loses a pointer to some piece of memory, that > > piece will automatically be reclaimed (not leaked). Still, memory leaks > > are possible if the program forgets about a pointer to some piece of > > memory no longer needed, and keeps that pointer in say some global > > structure. Such memory leaks would not be found using address sanitizer. > > > > Address sanitizer/Undefined behavior sanitizer can sometimes find errors > > caused by that the program forgets to protect an R object, but this is > > relatively rare, as they don't understand R heap specifically, so you > > cannot assume that if you create such example, the error will always be > > found. > > > > Best > > Tomas > > > >> > >> Please help! > >> > >> Thanks, > >> Xiaoyan > >> > >> [[alternative HTML version deleted]] > >> > >> ______________________________________________ > >> R-devel at r-project.org mailing list > >> https://stat.ethz.ch/mailman/listinfo/r-devel > > > > ______________________________________________ > > R-devel at r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > >[[alternative HTML version deleted]]