R help - Aug 2013 - Storing Big Matrix in R memory

I would like to store a big spatial weight matrix in R memory to do more
calculation. I know there are memory issue for 32 bit computer and I have tried
increasing the memory to maximum without success. I am using R version 2.15.2
and window vista. The data has about 15,000 observations and I would like to
create and store a distance based Spatial Weight Matrix in R memory. A
reproducible codes are as follows and for both codes produce out of memory
error. The codes works to up 5000 observations. 
 
Based on these codes, (for those with experiece in big data)is it possible to
use ffbase or bigmemory packages to achieve the objective (keep the matrix in
memory)? I will appreciate any help.
 
#Remove all objects
rm(list=ls()) 
library(spdep)
library(maptools)
library(GeoXp)
library(ffbase)
library(bigmemory) 
##########################################################################
#Testing using fake data
n = 15000
data = data.frame(n1=1:n)
data$SEQN  <- seq(1, n,1)
data$LAT   <- runif(n,  29.0000, 32.0000)
data$LON   <- runif(n, -100.0000, -89.0000)
######################################################################3
#Create distance based spatial weight matrix using dist function 
#using euclidean method
coords  <- cbind(data$LON, data$LAT)
SWM <- as.matrix(dist(coords, method = "euclidean", upper=TRUE))

#Other data management 
data1 <- cbind(data, SWM)

   
#Alternatively
#Create spatial weight matrix based on distance between Points
#using the spDistsN1 function and produce the same results as above
#Create coordinates
coordinates(data) <- ~LON+LAT
#Function to calculate the distance matrix
DistMatrix = function(obj, longlat = FALSE) {
  return(sapply(1:length(obj[[1]]), function(x) spDistsN1(coordinates(obj),
coordinates(obj[x,]), longlat)))
}
SWMD <- big.matrix(DistMatrix(data))

data2 <- cbind(data, SWMD)

With thanks

Peter Maclean
Department of Economics
UDSM
	[[alternative HTML version deleted]]

Hello,

If you can, you probably should upgrade for R version 3.0.1.

Regards,
Pascal



2013/8/27 Peter Maclean <pmaclean2011@yahoo.com>
> I would like to store a big spatial weight matrix in R memory to do more
> calculation. I know there are memory issue for 32 bit computer and I have
> tried increasing the memory to maximum without success. I am using R
> version 2.15.2 and window vista. The data has about 15,000 observations and
> I would like to create and store a distance based Spatial Weight Matrix in
> R memory. A reproducible codes are as follows and for both codes
> produce out of memory error. The codes works to up 5000 observations.
>
> Based on these codes, (for those with experiece in big data)is it possible
> to use ffbase or bigmemory packages to achieve the objective (keep the
> matrix in memory)? I will appreciate any help.
>
> #Remove all objects
> rm(list=ls())
> library(spdep)
> library(maptools)
> library(GeoXp)
> library(ffbase)
> library(bigmemory)
> ##########################################################################
> #Testing using fake data
> n = 15000
> data = data.frame(n1=1:n)
> data$SEQN  <- seq(1, n,1)
> data$LAT   <- runif(n,  29.0000, 32.0000)
> data$LON   <- runif(n, -100.0000, -89.0000)
> ######################################################################3
> #Create distance based spatial weight matrix using dist function
> #using euclidean method
> coords  <- cbind(data$LON, data$LAT)
> SWM <- as.matrix(dist(coords, method = "euclidean",
upper=TRUE))
>
> #Other data management
> data1 <- cbind(data, SWM)
>
>
> #Alternatively
> #Create spatial weight matrix based on distance between Points
> #using the spDistsN1 function and produce the same results as above
> #Create coordinates
> coordinates(data) <- ~LON+LAT
> #Function to calculate the distance matrix
> DistMatrix = function(obj, longlat = FALSE) {
>   return(sapply(1:length(obj[[1]]), function(x)
> spDistsN1(coordinates(obj), coordinates(obj[x,]), longlat)))
> }
> SWMD <- big.matrix(DistMatrix(data))
>
> data2 <- cbind(data, SWMD)
>
> With thanks
>
> Peter Maclean
> Department of Economics
> UDSM
>         [[alternative HTML version deleted]]
>
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
	[[alternative HTML version deleted]]