R help - May 2012 - R does not recognise columns and rows as they are supposed to be

Dear All,
The code given bellow is to extract  values of one region and write that to
a text file(there are 365 binary files in the directory).
The problem which I am facing is that all my files are binary with size of
360 rows and 720 columns.
I specified that in this line:    file2<-matrix(data=file,ncol=720,nrow=360)
but I got an error : Error in mean(file2[X, Y], na.rm = TRUE) : subscript
out of bounds.
and then I rewrote the above line as
:file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows =720
which is not right.But that worked and I didn't get any error.however,the
results were not correct.
Any help please

dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
listfile<-dir()
for (i in c(1:365)) {
      conne <- file(listfile[i], "rb")
      file<- readBin(conne, double(), size=4,  n=720*360, signed=T)
     file2<-matrix(data=file,ncol=720,nrow=360)  
	extract[i]<-mean(file2[X,Y],na.rm=TRUE)
      close(conne)
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\sam.txt")}


--
View this message in context:
http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217.html
Sent from the R help mailing list archive at Nabble.com.

On May 24, 2012, at 11:51 AM, Jonsson wrote:
> Dear All,
> The code given bellow is to extract  values of one region and write  
> that to
> a text file(there are 365 binary files in the directory).
> The problem which I am facing is that all my files are binary with  
> size of
> 360 rows and 720 columns.
> I specified that in this line:    file2<- 
> matrix(data=file,ncol=720,nrow=360)
> but I got an error : Error in mean(file2[X, Y], na.rm = TRUE) :  
> subscript
> out of bounds.
> and then I rewrote the above line as
> :file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows  
> =720
> which is not right.But that worked and I didn't get any  
> error.however,the
> results were not correct.
> Any help please
>
> dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
> (10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
> listfile<-dir()
> for (i in c(1:365)) {
>      conne <- file(listfile[i], "rb")
>      file<- readBin(conne, double(), size=4,  n=720*360, signed=T)
>     file2<-matrix(data=file,ncol=720,nrow=360)
> 	extract[i]<-mean(file2[X,Y],na.rm=TRUE)
>      close(conne)
> write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
> (10)\\sam.txt")}
Please do not cross post to SO and Rhelp. (See Posting Guide.)

(But if you do, rhelp generally expects reproducible examples which  
this is not.)

-- 
David Winsemius, MD
West Hartford, CT

Hello,

There are several things with your code, most of which are not errors.

1. X<-c(364:369) ; Y<-c(82:92 and later c(1:365)
Expressions of the form m:n are integer sequences, the c() is not needed.

2. extract<-vector()
First you create the vector, then keep extending it throughout the loop.
This is much slower.
If you know the final length (and type), create it like this:
extract <- numeric(365)   # or double(365)

3. 'dir1' is not used.

4. Now the readBin instruction:
4.1. 'file' is the name of a function, use, say, 'file1'
instead.
4.2. Option 'signed' defaults to TRUE but it's only valid for
integer types.

5. And, maybe this is the problem, in R, doubles are 8 bytes, not 4. Get rid
of option 'size', like the help page says,

"If size is specified and not the natural size of the object, each element
of the vector is coerced to an appropriate type before being written or as
it is read."

You are NOT reading doubles.

As for the rest, it seems ok to me. (Keep option 'n' in readBin.)

Hope this helps,

Rui Barradas

Jonsson wrote
> 
> Dear All,
> The code given bellow is to extract  values of one region and write that
> to a text file(there are 365 binary files in the directory).
> The problem which I am facing is that all my files are binary with size of
> 360 rows and 720 columns.
> I specified that in this line:   
> file2<-matrix(data=file,ncol=720,nrow=360) but I got an error : Error in
> mean(file2[X, Y], na.rm = TRUE) : subscript out of bounds.
> and then I rewrote the above line as
> :file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows =720
> which is not right.But that worked and I didn't get any
error.however,the
> results were not correct.
> Any help please
>              X<-c(364:369) ; Y<-c(82:92) #####   for sellected region
>                 extract<-vector()
> dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
> (10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
> listfile<-dir()
> for (i in c(1:365)) {
>       conne <- file(listfile[i], "rb")
>       file<- readBin(conne, double(), size=4,  n=720*360, signed=T)
>      file2<-matrix(data=file,ncol=720,nrow=360)  
> 	extract[i]<-mean(file2[X,Y],na.rm=TRUE)
>       close(conne)
> write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
> (10)\\sam.txt")}
> 

--
View this message in context:
http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631227.html
Sent from the R help mailing list archive at Nabble.com.

Yes this helped a lot .
I exactly followed what you suggested:
X<-(82:92) ; Y<-(364:369) #####   for sellected
region
> extract <- double(365) 
> setwd("C:\\Users\\aalyaari\\Desktop\\New folder (10)\\")
> listfile<-dir()
> for (i in 1:365) {
+       conne <- file(listfile[i], "rb")
+       file1<- readBin(conne, double(),  n=360*720)
+      file2<-matrix(data=file1,ncol=720,nrow=360)
+      extract[i]<-mean(file2[X,Y],na.rm=TRUE)
+       close(conne)
+ write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\samregion1.txt")}

But I wonder why  I got all values like

-3.75E+306
-1.30E+54
-1.22E+58
and the right ones should be like:
22.25
22.76
33.25



--
View this message in context:
http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631241.html
Sent from the R help mailing list archive at Nabble.com.

Is the file format documented?

If not you can search for a possible format if you know the values
at the start of the file.   For one of your files, show the results of the
following:
  file <- "your filename here"
  for(what in c("double", "integer")) {
    for(size in c(4, 8)) {
        for(endian in c("little", "big")) {
        cat(sep="", what, "/", size, "/", endian,
":\n   ");
        print(readBin(file, what=what, size=size, endian=endian, n=6))
      }
    }
  }
Do any of them look ok?  If not you may want to loop over possible
offsets in the file by opening a connection, reading from 1 to 7 one-byte
integers, and then reading the data of interest.


Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at
r-project.org] On Behalf
> Of Jonsson
> Sent: Thursday, May 24, 2012 11:41 PM
> To: r-help at r-project.org
> Subject: Re: [R] R does not recognise columns and rows as they are supposed
to be
> 
> Yes I exactly followed what you all suggested:
> X<-(82:92) ; Y<-(364:369) #####   for sellected region
> > extract <- double(365)
> > setwd("C:\\Users\\aalyaari\\Desktop\\New folder (10)\\")
> > listfile<-dir()
> > for (i in 1:365) {
> +       conne <- file(listfile[i], "rb")
> +       file1<- readBin(conne, double(),  n=360*720)
> +      file2<-matrix(data=file1,ncol=720,nrow=360)
> +      extract[i]<-mean(file2[X,Y],na.rm=TRUE)
> +       close(conne) }
>  write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
> (10)\\samregion1.txt")
> 
> But I  am still getting(negative values) all values like:
> 
>             -3.75E+306
>             -1.30E+54
>             -1.22E+58
> and the right ones should be like:
>               22.25
>               22.76
>               33.25
> 
> --
> View this message in context:
http://r.789695.n4.nabble.com/R-does-not-recognise-
> columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631279.html
> Sent from the R help mailing list archive at Nabble.com.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Yes I did so.
Yes the first values are the right ones: -9999 -9999.
so this meant that I should consider my data as:     double/4/little
Is it so?

file <-
"C:\\Users\\aalyaari\\Documents\\INRA\\WFD_reprocessed\\dialyswco\\2001\\SWdown_200101_01.img"
>  for(what in c("double", "integer")) {
+     for(size in c(4, 8)) {
+         for(endian in c("little", "big")) {
+         cat(sep="", what, "/", size, "/",
endian, ":\n   ");
+         print(readBin(file, what=what, size=size, endian=endian, n=6))
+       }
+     }
+   }
double/4/little:
   [1] -9999 -9999 -9999 -9999 -9999 -9999
double/4/big:
   [1] 5.520452e-39 5.520452e-39 5.520452e-39 5.520452e-39 5.520452e-39
5.520452e-39
double/8/little:
   [1] -5.592396e+29 -5.592396e+29 -5.592396e+29 -5.592396e+29 -5.592396e+29
-5.592396e+29
double/8/big:
   [1] 1.563804e-307 1.563804e-307 1.563804e-307 1.563804e-307 1.563804e-307
1.563804e-307
integer/4/little:
   [1] -971228160 -971228160 -971228160 -971228160 -971228160 -971228160
integer/4/big:
   [1] 3939526 3939526 3939526 3939526 3939526 3939526
integer/8/little:
   [1] -971228160 -971228160 -971228160 -971228160 -971228160 -971228160
integer/8/big:
   [1] 3939526 3939526 3939526 3939526 3939526 3939526
> 
--
View this message in context:
http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631344.html
Sent from the R help mailing list archive at Nabble.com.

The trick may be behind not reading the files properly is that  all my 365
files donot have the same name. for example:
files from number 1 to number9 are named for the first month: 
 
SWdown_200101_01.img   
SWdown_200101_09.img   

files from number 10 to number30 are named: 

 SWdown_200101_10.img
SWdown_200101_30.img

 And so on for the second month:
files from number 1 to number9 are named: 
 
SWdown_200102_01.img   
SWdown_200102_09.img   

files from number 10 to number30 are named: 

 SWdown_200102_10.img
SWdown_200102_30.img

and in my code I just set the dierctory:   dir1<-
list.files("C:\\Users\\aalyaari\\Desktop\\New folder (11)\\",
"*.img",
full.names = TRUE).

 assuming that R would read files in order.But I do not really know if this
right or I shall specify the names.


X<-(82:85) ; Y<-(364:367) #####   for sellected region 
 extract <- double() 
 dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(11)\\",
"*.bin", full.names = TRUE)
 for (i in 1:365) { 
      conne <- file(dir1[i], "rb") 
      file1<- readBin(conne, numeric(),size=4,  n=360*720,
endian="little")
    file2<-matrix(data=file1,ncol=720,nrow=360) 
   extract[i]<-mean(file2[X,Y],na.rm=TRUE) 
    close(conne) } 
 write.table(as.double(extract),"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\new6.txt") 



--
View this message in context:
http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631352.html
Sent from the R help mailing list archive at Nabble.com.