R help - Oct 2009 - Easy method to set user-mode virtual memory space in Windows Vista and 7

I thought I'd share this with the list since it appears to provide a
quick fix to some memory problems, and I haven't see it discussed in
relation to R.

To reallocate virtual memory from kernel-mode to user-mode in 32-bit
Vista or Windows 7 one can use the increaseuserva boot option value.
See
http://msdn.microsoft.com/en-us/library/aa906211.aspx

On my 4GB Vista machine, R is now able to use 3GB memory (previously
at the default value of 2GB).

Here's the method:

Open cmd.exe as administrator and type

BCDEdit /set IncreaseUserVA <value>

where <value> is between 2048 and 3072 (ie 2-3GB), then reboot.  Given
that you've set the --max-mem-size flag, or used the memory.limit
function in an R session to increase R's memory allocation, your R
processes should now be allowed to access up to 3GB of virtual memory.

I am not a Windows expert, so if anyone knows of disadvantages of
using this method *please* post a response.

best,

Kingsford Jones

On Wed, 28 Oct 2009, Kingsford Jones wrote:
> I thought I'd share this with the list since it appears to provide a
> quick fix to some memory problems, and I haven't see it discussed in
> relation to R.
Seems you didn't look in the places mentioned in the posting guide, fo 
this is an actual FAQ!

See the references in rw-FAQ Q2.9 (and this has been documented there 
for many years, and the R developer made R large-address-aware so 
users who looked at the rw-FAQ could benefit from this).

See http://msdn.microsoft.com/en-us/library/bb613473%28VS.85%29.aspx 
for a more rounded view covering all the relevant OSes.
> To reallocate virtual memory from kernel-mode to user-mode in 32-bit
> Vista or Windows 7 one can use the increaseuserva boot option value.
> See
> http://msdn.microsoft.com/en-us/library/aa906211.aspx
>
> On my 4GB Vista machine, R is now able to use 3GB memory (previously
> at the default value of 2GB).
>
> Here's the method:
>
> Open cmd.exe as administrator and type
>
> BCDEdit /set IncreaseUserVA <value>
>
> where <value> is between 2048 and 3072 (ie 2-3GB), then reboot. 
Given
> that you've set the --max-mem-size flag, or used the memory.limit
> function in an R session to increase R's memory allocation, your R
> processes should now be allowed to access up to 3GB of virtual memory.
See the rw-FAQ: it is 3GB of address space and 2.5GB of memory by 
default.
> I am not a Windows expert, so if anyone knows of disadvantages of
> using this method *please* post a response.
This is not advised in general by those who are experts (why do you 
think that it is not the default in Windows?).  As the URL above says

   'For applications that are memory-intensive, such as database
   management systems (DBMS), the use of a larger virtual address space
   can provide considerable performance and scalability benefits.
   However, the file cache, paged pool, and nonpaged pool are smaller,
   which can adversely affect applications with heavy networking or
   I/O.  Therefore, you might want to test your application under load,
   and examine the performance counters to determine whether your
   application benefits from the larger address space.'

Remember that you are setting this for all uses and users of your 
machine, not just the R process (so the last sentence is somewhat 
misleading).

The best advice seems to be that if you have 4GB or more of RAM is to 
run a 64-bit version of Windows, which gives a 32-bit process a 4GB 
user address space.  If you have a lot more memory, consider a 64-bit 
build of R (and an OS that handles memory more efficiently: one of my 
background tasks right now is to investigate why R under Windows is 
running a particular memory-intensive multiprocessor task so much more 
slowly than Linux on the same hardware).
> best,
>
> Kingsford Jones
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595