R help - Nov 2009 - R crashing

Hello,

This is what I am trying to do: I wrote a little function that takes
addresses (coordinates) as input, and returns the road distance between
every two points using Google Maps. Catch is, there are 2000 addresses, so I
have to get around 2x10^6 addresses. On my first go, this is what I did:

#########################################

getRoadDist = function(X,complete=F){    # X must be a matrix or data frame
of coordinates; lat and lon
    require(RCurl)
    Y = apply( X, 1, function(x){ paste(x[1], ",", x[2],
sep="") } )
    grid = expand.grid(Y,Y,KEEP.OUT.ATTRS=F)
    grid =
apply(grid,1,function(x){paste(x[1],"&daddr=",x[2],sep="")})
    grid = matrix(grid,ncol=length(Y),dimnames=list(names(Y),names(Y)))
    grid[upper.tri(grid,T)] = NA

        Distances = function(x){
            if (is.na(x)) {
                NA
            }
            else {
                URL =
getURL(paste("http://maps.google.com/maps?saddr",x,sep=""))
                y = strsplit(URL, "<div><b>")
                y = strsplit(y[[1]][2], "&#160;mi</b>"
)[[1]][1]
                as.numeric(y)
            }
        }

    dists = sapply(grid,Distances)
    dists = matrix(dists,ncol=ncol(grid),dimnames=dimnames(grid))
    if (complete) {
        diag(dists)=0
        dists[upper.tri(dists)]=dists[lower.tri(dists)]
        dists
    }
    else {
    dists
    }
}

#########################################

But R was crashing after 1 hour or so -- it either said "Reached total
allocation of 1535Mb" or, became unresponsive. Then, I tried to modify the
procedure to avoid big matrices at the. What I did was, I got the distances
and, one by one, appended them to a file in the hope that this would use
less memory:

##########################################

# X is the matrix of addresses, as before

    require(RCurl)
    Y = apply( X, 1, function(x){ paste(x[1], ",", x[2],
sep="") } )
    grid = expand.grid(Y,Y,KEEP.OUT.ATTRS=F)
    grid =
apply(grid,1,function(x){paste(x[1],"&daddr=",x[2],sep="")})
    grid = matrix(grid,ncol=length(Y),dimnames=list(names(Y),names(Y)))
    grid[upper.tri(grid,T)] = NA

        Distances = function(x){
            if (is.na(x)) {
                NA
            }
            else {
                URL =
getURL(paste("http://maps.google.com/maps?saddr",x,sep=""))
                y = strsplit(URL, "<div><b>")
                y = strsplit(y[[1]][2], "&#160;mi</b>"
)[[1]][1]
                as.numeric(y)
            }
        }


grid2=grid[!is.na(grid)]
n = length(grid2)
for (i in 1:n) {
temp = Distances(grid2[i])
write.table(temp,"distances.csv",col.names=F,row.names=F,append=T)
}

##########################################

But R still crashes after 2 hours (all I got was around 20.000 distances).
It doesn't really matter how long this will take me (I can always use more
than one machine), but I'd really like to get this done. Any thoughts?

Many many thanks,

Dimitri

	[[alternative HTML version deleted]]

On Sun, Nov 15, 2009 at 11:10 AM, Dimitri Szerman <dimitrijoe at
gmail.com> wrote:
> Hello,
>
> This is what I am trying to do: I wrote a little function that takes
> addresses (coordinates) as input, and returns the road distance between
> every two points using Google Maps. Catch is, there are 2000 addresses, so
I
> have to get around 2x10^6 addresses. On my first go, this is what I did:
 I hope on your first go you didn't run it with 2000 addresses. You
did test it with 13 addresses first didn't you?

 Another idea is to replace your Distance function with a function
that returns runif(1). This will either make your code fail much much
quicker or identify that the problem is in the Distance function (some
memory leak there).

 Also, you should check the return value from your google query - I've
seen google get a bit upset about repeated automated queries and
return a message saying "This looks like an automated query" and a
CAPTCHA test.

> grid2=grid[!is.na(grid)]
> n = length(grid2)
> for (i in 1:n) {
> temp = Distances(grid2[i])
>
write.table(temp,"distances.csv",col.names=F,row.names=F,append=T)
> }
This won't work - you're overwriting distances.csv with the new value
of 'temp' every time. Another good reason to test with 13 values
before waiting and failing after six hours, and then having to hammer
google's map server again.

I'd write this as a simple loop, and dump all the apply stuff. And
rewrite Distance to be a function of two lat-longs:

Distance=function(lat1,lon1,lat2,lon2){
....
return(distance)
}

Then (untested):

Dmat = matrix(NA,nrow(X),nrow(X))

for(i in 2:nrow(X)){
 for(j in 1:i){
  d = Distance(X[i,1],X[i,2],X[j,1],X[j,2])
  Dmat[i,j]=d
}
}

 I'm not sure apply wins much here.

Barry