thr3ads.net - R help - [R] Time intervals is converted into seconds after converting list of dfs into a single Df. [Dec 2019]

If this information is useful, please help other people find it:
Share via:

Allaisone 1

2019-Dec-24 18:53 UTC

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

Hi dear Patrick ,

Thanks for your replay. Below is a reproducible example . First,  I generated
two  similar Dfs with one column contains the interval. Then, I put the 2 dfs in
a list. Now, converting this list into df provides different results depending
on the code. See below for more details.


 # dataframe 1?

id <- c(1,1)?

dates1 <- c("2010/2/4","2011/2/4")?

dates2 <- c("2010/9/4","2011/1/1")?

df1 <- data.frame(id,dates1,dates2)?

df1[,2] <- as.Date(df1[,2])?

df1[,3] <- as.Date(df1[,3])?

df1$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))?

  ?

  # Dataframe 2?

id <- c(2,2)?

dates1 <- c("2010/1/4","2011/2/4")?

dates2 <- c("2010/10/4","2011/1/16")?

df2 <- data.frame(id,dates1,dates2)?

df2[,2] <- as.Date(df1[,2])?

df2[,3] <- as.Date(df1[,3])?


df2$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))?



 # 2 datframes in a list :?

 ListOfDFs <- list(df1,df2)?

 # Convert list of Dfs into a single df :-?

 SingDF <- ldply( ListOfDFs,data.frame)?

       # The interval has been converted into numbers which is not what I want.?

       #?but trying this code :
 SingDF <- do.call(rbind,ListOfDFs)?

       # It works perfectly but only with this example as? we have only 2
datframes. Howver, in my actual data I have? around 8000 datframes. Applying
this code to it , make R code? freezes and I waited for many hours but it still
freezes with? no results generated.?

 Could anyone please suggest any alternative syntax or modifications to the
codes above?

Kind Regards
 ?



Sent from Outlook
________________________________
From: Patrick (Malone Quantitative) <malone at malonequantitative.com>
Sent: 24 December 2019 17:01:59
To: Allaisone 1 <Allaisone1 at hotmail.com>
Cc: r-help at r-project.org <r-help at r-project.org>
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.

You didn't provide a reproducible example for testing (or post in
plain text), but lubridate has an as.interval() function. You'll need
to be able to extract the start time, though, for use in the function.

On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 <Allaisone1 at hotmail.com>
wrote:>
>
> Hi dear group ,
>
> I have list of datframes with similar column names. I want to rebind all
dataframes so I have a single dataframe. One of the column's in each df is
of 'interval' time class which was generated from 'lubridate'
package.
>
> The problem is that when I convert the list of dfs into a single df using
any of the below codes :
>
> Library(plyr)
> MySingleDf <- ldply(MyListOfDfs, data.frame)
> Or
> MySingleDf <- ldply(MyListOfDfs, rbind)
> Or
> MySingleDf <- rebind. fill (MyListOfDfs)
>
> What heppens is that  time intervals which looks like : 2010-4-5
UTC--2011-7-9 UTC is converted into a single numeric value which seems to be the
difference between the 2 dates in seconds.
>
> When I use :
> MySingleDf <- do.call ("rbind",MyListOfDfs)
>
> The code is freezes and it shows like of the data are being analysed but no
result. I have used this code previously for the same purpose but with another
datse and it works perfectly.
>
> What I want to see is that time intervals are shown as they are but not
converted into seconds.
>
> Could you please suggest any alternative syntax or modifications to my
codes ?
>
> Thank you so much in advance
>
> Regards
>
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
	[[alternative HTML version deleted]]

Bert Gunter

2019-Dec-24 21:03 UTC

head link

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

1. "Similar" or "same" column names. The former is probably
not going to
work.

2. Manipulations with data frames can consume a lot of memory. rbinding
8000 data frames is likely to be very slow with lots of time swapping
memory around(???). Perhaps try taking smaller bites (say 1000 at a time)
and then combining them. Or have you already tried this? If you do wish to
do this, wait to give experts a chance to tell you that my suggestion is
completely useless before you attempt it.

3. I'll let someone else resolve your dates problem, as I have never used
lubridate.

Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Tue, Dec 24, 2019 at 12:38 PM Allaisone 1 <Allaisone1 at hotmail.com>
wrote:
> Hi dear Patrick ,
>
> Thanks for your replay. Below is a reproducible example . First,  I
> generated two  similar Dfs with one column contains the interval. Then, I
> put the 2 dfs in a list. Now, converting this list into df provides
> different results depending on the code. See below for more details.
>
>
>  # dataframe 1
>
> id <- c(1,1)
>
> dates1 <- c("2010/2/4","2011/2/4")
>
> dates2 <- c("2010/9/4","2011/1/1")
>
> df1 <- data.frame(id,dates1,dates2)
>
> df1[,2] <- as.Date(df1[,2])
>
> df1[,3] <- as.Date(df1[,3])
>
> df1$interaction <-
> intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
>
>
>
>   # Dataframe 2
>
> id <- c(2,2)
>
> dates1 <- c("2010/1/4","2011/2/4")
>
> dates2 <- c("2010/10/4","2011/1/16")
>
> df2 <- data.frame(id,dates1,dates2)
>
> df2[,2] <- as.Date(df1[,2])
>
> df2[,3] <- as.Date(df1[,3])
>
>
> df2$interaction <-
> intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
>
>
>
>  # 2 datframes in a list :
>
>  ListOfDFs <- list(df1,df2)
>
>  # Convert list of Dfs into a single df :-
>
>  SingDF <- ldply( ListOfDFs,data.frame)
>
>        # The interval has been converted into numbers which is not what I
> want.
>
>        #but trying this code :
>  SingDF <- do.call(rbind,ListOfDFs)
>
>        # It works perfectly but only with this example as we have only 2
> datframes. Howver, in my actual data I have around 8000 datframes. Applying
> this code to it , make R code freezes and I waited for many hours but it
> still freezes with no results generated.
>
>  Could anyone please suggest any alternative syntax or modifications to
> the codes above?
>
> Kind Regards
>
>
>
>
> Sent from Outlook
> ________________________________
> From: Patrick (Malone Quantitative) <malone at
malonequantitative.com>
> Sent: 24 December 2019 17:01:59
> To: Allaisone 1 <Allaisone1 at hotmail.com>
> Cc: r-help at r-project.org <r-help at r-project.org>
> Subject: Re: [R] Time intervals is converted into seconds after converting
> list of dfs into a single Df.
>
> You didn't provide a reproducible example for testing (or post in
> plain text), but lubridate has an as.interval() function. You'll need
> to be able to extract the start time, though, for use in the function.
>
> On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 <Allaisone1 at
hotmail.com>
> wrote:
> >
> >
> > Hi dear group ,
> >
> > I have list of datframes with similar column names. I want to rebind
all
> dataframes so I have a single dataframe. One of the column's in each df
is
> of 'interval' time class which was generated from
'lubridate' package.
> >
> > The problem is that when I convert the list of dfs into a single df
> using any of the below codes :
> >
> > Library(plyr)
> > MySingleDf <- ldply(MyListOfDfs, data.frame)
> > Or
> > MySingleDf <- ldply(MyListOfDfs, rbind)
> > Or
> > MySingleDf <- rebind. fill (MyListOfDfs)
> >
> > What heppens is that  time intervals which looks like : 2010-4-5
> UTC--2011-7-9 UTC is converted into a single numeric value which seems to
> be the difference between the 2 dates in seconds.
> >
> > When I use :
> > MySingleDf <- do.call ("rbind",MyListOfDfs)
> >
> > The code is freezes and it shows like of the data are being analysed
but
> no result. I have used this code previously for the same purpose but with
> another datse and it works perfectly.
> >
> > What I want to see is that time intervals are shown as they are but
not
> converted into seconds.
> >
> > Could you please suggest any alternative syntax or modifications to my
> codes ?
> >
> > Thank you so much in advance
> >
> > Regards
> >
> >
> >
> >         [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]

Allaisone 1

2019-Dec-24 22:49 UTC

head link

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

Many thanks Bert for being so cooperative.
Deviding the data into small bites would be
a good suggestion but I will wait first to see
If someone else may have another idea.

Many thanks
________________________________
From: Bert Gunter <bgunter.4567 at gmail.com>
Sent: 24 December 2019 21:03:56
To: Allaisone 1 <Allaisone1 at hotmail.com>
Cc: Patrick (Malone Quantitative) <malone at malonequantitative.com>;
r-help at r-project.org <r-help at r-project.org>
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.

1. "Similar" or "same" column names. The former is probably
not going to work.

2. Manipulations with data frames can consume a lot of memory. rbinding 8000
data frames is likely to be very slow with lots of time swapping memory
around(???). Perhaps try taking smaller bites (say 1000 at a time) and then
combining them. Or have you already tried this? If you do wish to do this, wait
to give experts a chance to tell you that my suggestion is completely useless
before you attempt it.

3. I'll let someone else resolve your dates problem, as I have never used
lubridate.

Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )

On Tue, Dec 24, 2019 at 12:38 PM Allaisone 1 <Allaisone1 at
hotmail.com<mailto:Allaisone1 at hotmail.com>> wrote:
Hi dear Patrick ,

Thanks for your replay. Below is a reproducible example . First,  I generated
two  similar Dfs with one column contains the interval. Then, I put the 2 dfs in
a list. Now, converting this list into df provides different results depending
on the code. See below for more details.

 # dataframe 1

id <- c(1,1)

dates1 <- c("2010/2/4","2011/2/4")

dates2 <- c("2010/9/4","2011/1/1")

df1 <- data.frame(id,dates1,dates2)

df1[,2] <- as.Date(df1[,2])

df1[,3] <- as.Date(df1[,3])

df1$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))

  # Dataframe 2

id <- c(2,2)

dates1 <- c("2010/1/4","2011/2/4")

dates2 <- c("2010/10/4","2011/1/16")

df2 <- data.frame(id,dates1,dates2)

df2[,2] <- as.Date(df1[,2])

df2[,3] <- as.Date(df1[,3])

df2$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))

 # 2 datframes in a list :

 ListOfDFs <- list(df1,df2)

 # Convert list of Dfs into a single df :-

 SingDF <- ldply( ListOfDFs,data.frame)

       # The interval has been converted into numbers which is not what I want.

       #but trying this code :
 SingDF <- do.call(rbind,ListOfDFs)

       # It works perfectly but only with this example as we have only 2
datframes. Howver, in my actual data I have around 8000 datframes. Applying this
code to it , make R code freezes and I waited for many hours but it still
freezes with no results generated.

 Could anyone please suggest any alternative syntax or modifications to the
codes above?

Kind Regards

Sent from Outlook
________________________________
From: Patrick (Malone Quantitative) <malone at
malonequantitative.com<mailto:malone at malonequantitative.com>>
Sent: 24 December 2019 17:01:59
To: Allaisone 1 <Allaisone1 at hotmail.com<mailto:Allaisone1 at
hotmail.com>>
Cc: r-help at r-project.org<mailto:r-help at r-project.org> <r-help at
r-project.org<mailto:r-help at r-project.org>>
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.

You didn't provide a reproducible example for testing (or post in
plain text), but lubridate has an as.interval() function. You'll need
to be able to extract the start time, though, for use in the function.

On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 <Allaisone1 at
hotmail.com<mailto:Allaisone1 at hotmail.com>>
wrote:>
>
> Hi dear group ,
>
> I have list of datframes with similar column names. I want to rebind all
dataframes so I have a single dataframe. One of the column's in each df is
of 'interval' time class which was generated from 'lubridate'
package.
>
> The problem is that when I convert the list of dfs into a single df using
any of the below codes :
>
> Library(plyr)
> MySingleDf <- ldply(MyListOfDfs, data.frame)
> Or
> MySingleDf <- ldply(MyListOfDfs, rbind)
> Or
> MySingleDf <- rebind. fill (MyListOfDfs)
>
> What heppens is that  time intervals which looks like : 2010-4-5
UTC--2011-7-9 UTC is converted into a single numeric value which seems to be the
difference between the 2 dates in seconds.
>
> When I use :
> MySingleDf <- do.call ("rbind",MyListOfDfs)
>
> The code is freezes and it shows like of the data are being analysed but no
result. I have used this code previously for the same purpose but with another
datse and it works perfectly.
>
> What I want to see is that time intervals are shown as they are but not
converted into seconds.
>
> Could you please suggest any alternative syntax or modifications to my
codes ?
>
> Thank you so much in advance
>
> Regards
>
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org<mailto:R-help at r-project.org> mailing list
-- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
        [[alternative HTML version deleted]]

______________________________________________
R-help at r-project.org<mailto:R-help at r-project.org> mailing list -- To
UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

	[[alternative HTML version deleted]]

David Winsemius

2019-Dec-24 23:45 UTC

head link

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

Perhaps some modification of

masterList <- list()

for( dfnum in seq_along(ListOfDFs)){

 ???????? masterList <- rbind(masterList, ListOfDFs[[dfnum]])

 ?????????????????????????????????????????????????????????????? }

 ?masterList

#---------------

 ?? id??? ? ?? dates1??? ? ? ? dates2 ? ? ? ?? ?? interaction
1? 1 2010-02-04 2010-09-04 2010-09-04 UTC--2011-01-01 UTC
2? 1 2011-02-04 2011-01-01 2010-09-04 UTC--2011-01-01 UTC
3? 2 2010-02-04 2010-09-04 2010-09-04 UTC--2011-01-01 UTC

4? 2 2011-02-04 2011-01-01 2010-09-04 UTC--2011-01-01


You could add features to the for-loop such as printing a message to the 
console every 100 dfs or perhaps garbage collection although that should 
be handled automagically. Messages would probably reassure you that the 
process was not "hanging". (My suspicion is that the process was 
continuing but you were just too impatient.)

Note; you are posting in HTML and including non-printing characters in 
you code.


-- 

David


On 12/24/19 10:53 AM, Allaisone 1 wrote:> Hi dear Patrick ,
>
> Thanks for your replay. Below is a reproducible example . First,  I
generated two  similar Dfs with one column contains the interval. Then, I put
the 2 dfs in a list. Now, converting this list into df provides different
results depending on the code. See below for more details.
>
>
>   # dataframe 1?
>
> id <- c(1,1)?
>
> dates1 <- c("2010/2/4","2011/2/4")?
>
> dates2 <- c("2010/9/4","2011/1/1")?
>
> df1 <- data.frame(id,dates1,dates2)?
>
> df1[,2] <- as.Date(df1[,2])?
>
> df1[,3] <- as.Date(df1[,3])?
>
> df1$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))?
>
>    ?
>
>    # Dataframe 2?
>
> id <- c(2,2)?
>
> dates1 <- c("2010/1/4","2011/2/4")?
>
> dates2 <- c("2010/10/4","2011/1/16")?
>
> df2 <- data.frame(id,dates1,dates2)?
>
> df2[,2] <- as.Date(df1[,2])?
>
> df2[,3] <- as.Date(df1[,3])?
>
>
> df2$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))?
>
>
>
>   # 2 datframes in a list :?
>
>   ListOfDFs <- list(df1,df2)?
>
>   # Convert list of Dfs into a single df :-?
>
>   SingDF <- ldply( ListOfDFs,data.frame)?
>
>         # The interval has been converted into numbers which is not what I
want.?
>
>         #?but trying this code :
>   SingDF <- do.call(rbind,ListOfDFs)?
>
>         # It works perfectly but only with this example as? we have only 2
datframes. Howver, in my actual data I have? around 8000 datframes. Applying
this code to it , make R code? freezes and I waited for many hours but it still
freezes with? no results generated.?
>
>   Could anyone please suggest any alternative syntax or modifications to
the codes above?
>
> Kind Regards
>   ?
>
>
>
> Sent from Outlook
> ________________________________
> From: Patrick (Malone Quantitative) <malone at
malonequantitative.com>
> Sent: 24 December 2019 17:01:59
> To: Allaisone 1 <Allaisone1 at hotmail.com>
> Cc: r-help at r-project.org <r-help at r-project.org>
> Subject: Re: [R] Time intervals is converted into seconds after converting
list of dfs into a single Df.
>
> You didn't provide a reproducible example for testing (or post in
> plain text), but lubridate has an as.interval() function. You'll need
> to be able to extract the start time, though, for use in the function.
>
> On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 <Allaisone1 at
hotmail.com> wrote:
>>
>> Hi dear group ,
>>
>> I have list of datframes with similar column names. I want to rebind
all dataframes so I have a single dataframe. One of the column's in each df
is of 'interval' time class which was generated from 'lubridate'
package.
>>
>> The problem is that when I convert the list of dfs into a single df
using any of the below codes :
>>
>> Library(plyr)
>> MySingleDf <- ldply(MyListOfDfs, data.frame)
>> Or
>> MySingleDf <- ldply(MyListOfDfs, rbind)
>> Or
>> MySingleDf <- rebind. fill (MyListOfDfs)
>>
>> What heppens is that  time intervals which looks like : 2010-4-5
UTC--2011-7-9 UTC is converted into a single numeric value which seems to be the
difference between the 2 dates in seconds.
>>
>> When I use :
>> MySingleDf <- do.call ("rbind",MyListOfDfs)
>>
>> The code is freezes and it shows like of the data are being analysed
but no result. I have used this code previously for the same purpose but with
another datse and it works perfectly.
>>
>> What I want to see is that time intervals are shown as they are but not
converted into seconds.
>>
>> Could you please suggest any alternative syntax or modifications to my
codes ?
>>
>> Thank you so much in advance
>>
>> Regards
>>
>>
>>
>>          [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

R help - Dec 2019 - Time intervals is converted into seconds after converting list of dfs into a single Df.

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.

[R] Time intervals is converted into seconds after converting list of dfs into a single Df.