thr3ads.net - R help - [R] comparison of large data set [Dec 2012]

If this information is useful, please help other people find it:
Share via:

Irucka Embry

2012-Dec-20 18:42 UTC

[R] comparison of large data set

Hi, how are you?

I have the following truncated R code:

fileobs <- "MaxFloodDepth_Observed.txt"
file1 <- "MaxFloodDepth_1.txt"
file2 <- "MaxFloodDepth_2.txt"

...

file54 <- "MaxFloodDepth_54.txt"


observeddepth = as.matrix(read.ascii.grid(fileobs)$data)
observeddepth[observeddepth!=0]<-1

modeldepth1 = as.matrix(read.ascii.grid(file1)$data)
modeldepth1[modeldepth1!=0]<-1

modeldepth2 = as.matrix(read.ascii.grid(file2)$data)
modeldepth2[modeldepth2!=0]<-1

...

modeldepth54 = as.matrix(read.ascii.grid(file54)$data)
modeldepth54[modeldepth54!=0]<-1

Each modeldepth* and observeddepth is 272x976

f2 <- function(obs, dat){
obs <- as.logical(obs)
dat <- as.logical(dat)
s1 <- sum(obs & dat)
s2 <- sum(obs & !dat)
s3 <- sum(!obs & dat)
s1/(s1 + s2 + s3)
}

f2all <- f2(observeddepth, modeldepth[1:54]) # I am including the
"[1:54]" in the code to indicate that I want to compare the
observeddepth data to each of the modeldepth data files

modeldepth <- as.matrix(c(modeldepth1, modeldepth2, ... modeldepth54))

modeldepth <- modeldepth_index[, f2 > 0.7] # I want to end with
modeldepth that only lists the specific 272x976 modeldepth data files
that had an f2 greater than 0.7

Is there any way to reduce the code? As you see, I am using the name for
all 54 files with both file and also modeldepth.

How can I create an f2 for each of the 54 comparisons?

Thank-you.

Irucka Embry 


<span id=m2wTl><p><font face="Arial, Helvetica,
sans-serif" size="2"
style="font-size:13.5px">_______________________________________________________________<BR>Get
the Free email that has everyone talking at <a href=http://www.mail2world.com
target=new>http://www.mail2world.com</a><br>  <font
color=#999999>Unlimited Email Storage &#150; POP3 &#150; Calendar
&#150; SMS &#150; Translator &#150; Much
More!</font></font></span>
	[[alternative HTML version deleted]]

Adams, Jean

2012-Dec-21 16:32 UTC

head link

[R] comparison of large data set

Irucka,

I did not test this code out on any data, but I think it will work.

Jean


# a function to read in the data as a matrix of logicals
myreadfun <- function(file) {
as.matrix(read.ascii.grid(file)$data)!=0
}

# names of the 54 modeled depth files
modfiles <- paste0("MaxFloodDepth_", 1:54, ".txt")

# read in the observed and modeled depth files
# observeddepth is a matrix
observeddepth <- myreadfun("MaxFloodDepth_Observed.txt")
# modeldepths is a list of matrices
modeldepths <- lapply(filenames, myreadfun)

# calculate the proportion of matrix elements that agree with the observed
file
# the results is a vector with one number for each modeled depth matrix
compare.all <- sapply(modeldepths, function(md) mean(md==observeddepth))

# select just those matrices that had a large proportion of agreements
# justbig is a list of matrices
justbig <- modeldepths[[compare.all > 0.7]]



On Thu, Dec 20, 2012 at 12:42 PM, Irucka Embry
<iruckaE@mail2world.com>wrote:
> Hi, how are you?
>
> I have the following truncated R code:
>
> fileobs <- "MaxFloodDepth_Observed.txt"
> file1 <- "MaxFloodDepth_1.txt"
> file2 <- "MaxFloodDepth_2.txt"
>
> ...
>
> file54 <- "MaxFloodDepth_54.txt"
>
>
> observeddepth = as.matrix(read.ascii.grid(fileobs)$data)
> observeddepth[observeddepth!=0]<-1
>
> modeldepth1 = as.matrix(read.ascii.grid(file1)$data)
> modeldepth1[modeldepth1!=0]<-1
>
> modeldepth2 = as.matrix(read.ascii.grid(file2)$data)
> modeldepth2[modeldepth2!=0]<-1
>
> ...
>
> modeldepth54 = as.matrix(read.ascii.grid(file54)$data)
> modeldepth54[modeldepth54!=0]<-1
>
> Each modeldepth* and observeddepth is 272x976
>
> f2 <- function(obs, dat){
> obs <- as.logical(obs)
> dat <- as.logical(dat)
> s1 <- sum(obs & dat)
> s2 <- sum(obs & !dat)
> s3 <- sum(!obs & dat)
> s1/(s1 + s2 + s3)
> }
>
> f2all <- f2(observeddepth, modeldepth[1:54]) # I am including the
> "[1:54]" in the code to indicate that I want to compare the
> observeddepth data to each of the modeldepth data files
>
> modeldepth <- as.matrix(c(modeldepth1, modeldepth2, ... modeldepth54))
>
> modeldepth <- modeldepth_index[, f2 > 0.7] # I want to end with
> modeldepth that only lists the specific 272x976 modeldepth data files
> that had an f2 greater than 0.7
>
> Is there any way to reduce the code? As you see, I am using the name for
> all 54 files with both file and also modeldepth.
>
> How can I create an f2 for each of the 54 comparisons?
>
> Thank-you.
>
> Irucka Embry
>
>
> <span id=m2wTl><p><font face="Arial, Helvetica,
sans-serif" size="2"
>
style="font-size:13.5px">_______________________________________________________________<BR>Get
> the Free email that has everyone talking at <a href>
http://www.mail2world.com
target=new>http://www.mail2world.com</a><br>
>  <font color=#999999>Unlimited Email Storage &#150; POP3
&#150; Calendar
> &#150; SMS &#150; Translator &#150; Much
More!</font></font></span>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]

Adams, Jean

2012-Dec-21 19:25 UTC

head link

[R] comparison of large data set

Irucka,

You should cc R-help on all correspondence so that other readers of the
list can follow the conversation.

(1)  You say that you need 0s and 1s rather than TRUE/FALSE.  Since a 0/1
matrix and TRUE/FALSE matrix behave exactly the same way in most
applications, I'm not sure why it would matter which one you had.

(2)  My mistake.  I had not noticed that you were eliminating the case
where both the observed and the modeled were FALSE.  I have modified my
code to give the same results as your f2 function.
compare.all <- sapply(modeldepth, function(md) mean((md==observeddepth)[md
| observeddepth]))

(3)  My mistake again.  There should be only single brackets, not double
brackets.
justbig <- modeldepth[compare.all > 0.7]

Jean



On Fri, Dec 21, 2012 at 11:27 AM, Irucka Embry
<iruckaE@mail2world.com>wrote:
>  Hi Jean Adams, how are you?
>
> I want to thank you for your response to my request for assistance.
>
> I received some assistance yesterday afternoon and I was able to update
> the code which is posted here:
>
http://r.789695.n4.nabble.com/variable-names-in-numeric-list-and-Bayesian-inference-td4653674.html.
> I posted the new code with some new questions that I have with regards to
> the code that I have written. Can you look over that post and suggest any
> code revisions for those aspects that do not work? Thank-you.
>
> The code that you suggested worked well overall, except for 3 aspects of
> it:
>
> Here I actually needed the binary 0s and 1s rather than a TRUE/FALSE
> logical matrix
>
> # a function to read in the data as a matrix of logicals
> myreadfun <- function(file) {
> as.matrix(read.ascii.grid(file)$data)!=0
> }
>
> Here I needed to calculate the f2 probability rather than the mean
>
> compare.all <- sapply(modeldepths, function(md) mean(md==observeddepth))
> > str(compare.all)
> num [1:54] 0.936 0.94 0.944 0.944 0.945 ...
>
> Here most of the entries are greater than 0.7, but it should just be 31 of
> the 54 that are greater than 0.7
>
> > justbig <- modeldepths[[compare.all > 0.7]]
> Error in modeldepths[[compare.all > 0.7]] :
> recursive indexing failed at level 2
>
> Once again, thank-you for your assistance.
>
> Irucka Embry
>
>
> <-----Original Message----->
> >From: Adams, Jean [jvadams@usgs.gov]
> >Sent: 12/21/2012 10:32:54 AM
> >To: iruckaE@mail2world.com
> >Cc: r-help@r-project.org
> >Subject: Re: [R] comparison of large data set
> >
> >Irucka,
> >
> >
> >I did not test this code out on any data, but I think it will work.
> >
> >
> >Jean
> >
> >
> >
> >
> ># a function to read in the data as a matrix of logicals
> >myreadfun <- function(file) {
> >as.matrix(read.ascii.grid(file)$data)!=0
> >}
> >
> >
> ># names of the 54 modeled depth files
> >modfiles <- paste0("MaxFloodDepth_", 1:54,
".txt")
> >
> >
> ># read in the observed and modeled depth files
> ># observeddepth is a matrix
> >observeddepth <- myreadfun("MaxFloodDepth_Observed.txt")
> >
> ># modeldepths is a list of matrices
> >modeldepths <- lapply(filenames, myreadfun)
> >
> >
> ># calculate the proportion of matrix elements that agree with the
> observed file
> ># the results is a vector with one number for each modeled depth matrix
> >compare.all <- sapply(modeldepths, function(md)
mean(md==observeddepth))
> >
> >
> ># select just those matrices that had a large proportion of agreements
> ># justbig is a list of matrices
> >justbig <- modeldepths[[compare.all > 0.7]]
> >
> >______________________________________________
> >R-help@r-project.org mailing list
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
>
> _______________________________________________________________
> Get the Free email that has everyone talking at http://www.mail2world.com
> Unlimited Email Storage – POP3 – Calendar – SMS – Translator – Much More!
>
	[[alternative HTML version deleted]]

Irucka Embry

2012-Dec-28 17:10 UTC

head link

[R] comparison of large data set

Hi Jean, thank-you.

It was my fault on not ccing R-help on the previous correspondence. With
regards to the FALSE/TRUE or 0/1, you are right that they are the same
logical identities, but it better suits our original numerical data to
display 0s and 1s. 

I want to thank you for correcting the 2 mistakes, the code works now.

Is it possible to label each of the 54 matrices so that "justbig"
lists
the 31 approved matrices along with their name [whatever the name is for
the 31 matrices out of the 54 total matrices]? For this code to be fully
applied to this project we have to know which site locations are
successful (> 0.7) and which ones are not (< 0.7).

Thank-you Jean.

Irucka



<-----Original Message-----> >From: Adams, Jean [jvadams@usgs.gov]
>Sent: 12/21/2012 1:25:24 PM
>To: iruckaE@mail2world.com
>Cc: r-help@r-project.org
>Subject: Re: [R] comparison of large data set
>
>Irucka,
>
>
>You should cc R-help on all correspondence so that other readers of the
list can>follow the conversation.
>
>
>(1) You say that you need 0s and 1s rather than TRUE/FALSE. Since a 0/1
matrix>and TRUE/FALSE matrix behave exactly the same way in most applications,
I'm not>sure why it would matter which one you had.
>
>
>(2) My mistake. I had not noticed that you were eliminating the case
where>both the observed and the modeled were FALSE.. I have modified my code
to give>the same results as your f2 function.
>compare.all <- sapply(modeldepth, function(md)
mean((md==observeddepth)[md |>observeddepth]))
>
>
>
>(3) My mistake again. There should be only single brackets, not double
brackets.>justbig <- modeldepth[compare.all > 0.7]
>
>
>Jean
>
>
>
>
>
>On Fri, Dec 21, 2012 at 11:27 AM, Irucka Embry
<iruckaE@mail2world.com>
wrote:>
>Hi Jean Adams, how are you?
>
>I want to thank you for your response to my request for assistance. 
>
>I received some assistance yesterday afternoon and I was able to update
the code>which is posted here:
>http://r.789695.n4.nabble.com/variable-names-in-numeric-list-and-Bayesi
an-inference-td4653674.html.>I posted the new code with some new questions that I have with regards
to the>code that I have written. Can you look over that post and suggest any
code>revisions for those aspects that do not work? Thank-you.
>
>The code that you suggested worked well overall, except for 3 aspects
of it:>
>Here I actually needed the binary 0s and 1s rather than a TRUE/FALSE
logical matrix>
># a function to read in the data as a matrix of logicals
>myreadfun <- function(file) {
>as.matrix(read.ascii.grid(file)$data)!=0
>}
>
>
>Here I needed to calculate the f2 probability rather than the mean
>
>compare.all <- sapply(modeldepths, function(md)
mean(md==observeddepth))>
>> str(compare.all)
>num [1:54] 0.936 0.94 0.944 0.944 0.945 ...
>
>Here most of the entries are greater than 0.7, but it should just be 31
of the>54 that are greater than 0.7
>
>> justbig <- modeldepths[[compare.all > 0.7]]
>
>Error in modeldepths[[compare.all > 0.7]] : 
>recursive indexing failed at level 2
>
>Once again, thank-you for your assistance.
>
>Irucka Embry
>
>
><-----Original Message-----> 
>>From: Adams, Jean [jvadams@usgs.gov]
>>Sent: 12/21/2012 10:32:54 AM
>>To: iruckaE@mail2world.com
>>Cc: r-help@r-project.org
>>Subject: Re: [R] comparison of large data set
>>
>>Irucka,
>>
>>
>>I did not test this code out on any data, but I think it will work.
>>
>>
>>Jean
>>
>>
>>
>>
>># a function to read in the data as a matrix of logicals
>>myreadfun <- function(file) {
>>as.matrix(read.ascii.grid(file)$data)!=0
>>}
>>
>>
>># names of the 54 modeled depth files
>>modfiles <- paste0("MaxFloodDepth_", 1:54,
".txt")
>>
>>
>># read in the observed and modeled depth files
>># observeddepth is a matrix
>>observeddepth <- myreadfun("MaxFloodDepth_Observed.txt")
>>
>># modeldepths is a list of matrices
>>modeldepths <- lapply(filenames, myreadfun)
>>
>>
>># calculate the proportion of matrix elements that agree with the
observed file>># the results is a vector with one number for each modeled depth
matrix>>compare.all <- sapply(modeldepths, function(md)
mean(md==observeddepth))>>
>>
>># select just those matrices that had a large proportion of agreements
>># justbig is a list of matrices
>>justbig <- modeldepths[[compare.all > 0.7]]
>>
>
>>______________________________________________
>>R-help@r-project.org mailing list
>>https://stat.ethz.ch/mailman/listinfo/r-help
>>PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html>>and provide commented, minimal, self-contained, reproducible code. 

<span id=m2wTl><p><font face="Arial, Helvetica,
sans-serif" size="2"
style="font-size:13.5px">_______________________________________________________________<BR>Get
the Free email that has everyone talking at <a href=http://www.mail2world.com
target=new>http://www.mail2world.com</a><br>  <font
color=#999999>Unlimited Email Storage &#150; POP3 &#150; Calendar
&#150; SMS &#150; Translator &#150; Much
More!</font></font></span>
	[[alternative HTML version deleted]]

Irucka Embry

2012-Dec-31 07:42 UTC

head link

[R] comparison of large data set

Hi Jean, thank-you again.

Both the assignment of names and the indices of the successful locations
worked perfectly.

Irucka Embry


<-----Original Message-----> >From: Adams, Jean [jvadams@usgs.gov]
>Sent: 12/28/2012 1:28:29 PM
>To: iruckaE@mail2world.com
>Cc: r-help@r-project.org
>Subject: Re: [R] comparison of large data set
>
>Irucka,
>
>
>You could assign names to the compare.all list .... for example ...
>
> names(compare.all) <- paste0("Obs", 1:54)
>
>Then, when you create the subset list, justbig, it will have the
appropriate names.>
>
>
>If you just want to see the indices of the successful locations, you
could print> (1:54)[compare.all > 0.7]
>
>
>Jean
>
>
>
>
>
>On Fri, Dec 28, 2012 at 11:10 AM, Irucka Embry
<iruckaE@mail2world.com>
wrote:>
>Hi Jean, thank-you.
>
>It was my fault on not ccing R-help on the previous correspondence.
With regards to the FALSE/TRUE or >0/1, you are right that they are the same logical identities, but it
better suits our original numerical data to >display 0s and 1s. 
>
>I want to thank you for correcting the 2 mistakes, the code works now.
>
>Is it possible to label each of the 54 matrices so that "justbig"
lists
the 31 approved matrices along with >their name [whatever the name is for the 31 matrices out of the 54
total matrices]? For this code to be >fully applied to this project we have to know which site locations are
successful (> 0.7) and which ones >are not (< 0.7).
>
>Thank-you Jean.
>
>Irucka 

<span id=m2wTl><p><font face="Arial, Helvetica,
sans-serif" size="2"
style="font-size:13.5px">_______________________________________________________________<BR>Get
the Free email that has everyone talking at <a href=http://www.mail2world.com
target=new>http://www.mail2world.com</a><br>  <font
color=#999999>Unlimited Email Storage &#150; POP3 &#150; Calendar
&#150; SMS &#150; Translator &#150; Much
More!</font></font></span>
	[[alternative HTML version deleted]]

Apparently Analagous Threads

Search for more seemingly similar threads

R help - Dec 2012 - comparison of large data set

[R] comparison of large data set

[R] comparison of large data set

[R] comparison of large data set

[R] comparison of large data set

[R] comparison of large data set

Apparently Analagous Threads