Ista Zahn
2018-Nov-15 14:53 UTC
[R] Corrupting files while copying (was Re: saveRDS() and readRDS() Why? [solved, pretty much anyway])
Hi Patrick, I think it would help to start from the beginning and give complete (but concise!) replication instructions, including telling us what host and gest operating systems you are using (including the versions), the version of virtualbox you used, and exactly what steps are needed to reproduce the surprising behavior. Best, Ista On Wed, Nov 14, 2018 at 2:36 AM Patrick Connolly <p_connolly at slingshot.co.nz> wrote:> > Thanks William, > > I've used Dolphin for years and never encountered that phenomenon. > Even so, that description doesn't fit what's going on here. 1.7 > kilobytes is hardly a 'large directory'. > > The problem seems to be with the way VirtualBox mounts directories > which isn't an R issue, nor is the fact that copying from Linux to > Windows isn't affected. But the fact that it happens only with rds > files that use the name of the R object as part of their own names > must be an R issue (that surfaces only when other conditions are > present). > > Theories short of divine intervention appreciated. > > > > On Tue, 13-Nov-2018 at 02:22PM -0800, William Dunlap wrote: > > |> Perhaps you got bitten by Dolphin's non-modal dialogs, as described in > |> https://userbase.kde.org/Dolphin/File_Management: > |> > |> Non Modal Dialogs > |> > |> When Moving, Copying or Deleting files/directories the dialog disappears > |> even when the operation has not yet completed. A progress bar then appears > |> in the bottom right of the screen, this then disappears also, if you want > |> see the progress you need to click a small (i) information icon in the > |> system tray. > |> > |> > |> Warning > |> New users who are not used to this way of working (and even experienced > |> users) can get caught out by this, if you are Moving, Copying or Deleting > |> large directories then you need to use the icon to monitor the progress of > |> your operation. If you don't then any subsequent actions you do, may well > |> use an incomplete file structure resulting in corrupted files. You have > |> been warned! > |> > |> Bill Dunlap > |> TIBCO Software > |> wdunlap tibco.com > |> > |> On Tue, Nov 13, 2018 at 2:10 PM, p_connolly <p_connolly at slingshot.co.nz> > |> wrote: > |> > |> > This is getting more strange. > |> > > |> > I normally copy from the shared folder to the appropriate directory using > |> > Dolphin, the KDE file manager. If instead I use the standard bash cp > |> > command, no corruption happens -- at least with the limited testing I have > |> > done. There also seems to be no problem copying from Linux to Windows. I > |> > installed R-3.5.1 for Windows just to eliminate that possible issue. > |> > > |> > However, R has *something* to do with it because it was used to make the > |> > .rds file. Just how the relationship between the name of the R object and > |> > the name of the .rds file comes into it, I can't imagine. > |> > > |> > Thanks for the suggestion William. > |> > > |> > > |> > On 2018-11-14 06:26, William Dunlap wrote: > |> > > |> >> It seems like copying the files corrupted them. How did you copy them > |> >> (with R > |> >> or cp or copy or ftp, etc.)? I don't see how this has anything to do > |> >> with R. > |> >> > |> >> Bill Dunlap > |> >> TIBCO Software > |> >> wdunlap tibco.com [1] > |> >> On Mon, Nov 12, 2018 at 7:10 PM, p_connolly > |> >> <p_connolly at slingshot.co.nz> wrote: > |> >> > |> > [...] > |> > > |> > > > -- > ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. > ___ Patrick Connolly > {~._.~} Great minds discuss ideas > _( Y )_ Average minds discuss events > (:_~*~_:) Small minds discuss people > (_)-(_) ..... Eleanor Roosevelt > > ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
Patrick Connolly
2018-Nov-18 08:44 UTC
[R] Corrupting files while copying (was Re: saveRDS() and readRDS() Why? [solved, pretty much anyway])
Sequence of steps: Using R-3.5.1 and Windows 7 with the latest Rstudio, shared directory as working directory on Virtual Box host machine:> x <- airquality > saveRDS(x, file = "x.rds") > saveRDS(x, file = "y.rds")On guest machine Mint Linux 17.3, KDE desktop, copy x.rds & y.rds to working directory PWD using file manager Dolphin. (Don't have the precise version of VirtualBox right now.)> x <- readRDS(file = "x.rds")Error in readRDS(file = "x.rds") : error reading from connection> x <- readRDS(file = "y.rds")> tools::md5sum(c("x.rds", "y.rds"))x.rds y.rds "5fef054848f39b4be02b7c54f1c71a20" "978a64d1dd342d16a381c9ca728d3665" Yet, if instead of using Dolphin, use bash commands from the shared directory $ cp *.rds ~/PWD/ no error reading from the connection or other differences between x.rds and y.rds. head(x)> head(datasets::airquality)Ozone Solar.R Wind Temp Month Day 1 41 190 7.4 67 5 1 2 36 118 8.0 72 5 2 3 12 149 12.6 74 5 3 4 18 313 11.5 62 5 4 5 NA NA 14.3 56 5 5 6 28 NA 14.9 66 5 6>On Thu, 15-Nov-2018 at 09:53AM -0500, Ista Zahn wrote: |> Hi Patrick, |> |> I think it would help to start from the beginning and give complete |> (but concise!) replication instructions, including telling us what |> host and gest operating systems you are using (including the |> versions), the version |> of virtualbox you used, and exactly what steps are needed to |> reproduce the surprising behavior. |> |> Best, |> Ista |> On Wed, Nov 14, 2018 at 2:36 AM Patrick Connolly |> <p_connolly at slingshot.co.nz> wrote: |> > |> > Thanks William, |> > |> > I've used Dolphin for years and never encountered that phenomenon. |> > Even so, that description doesn't fit what's going on here. 1.7 |> > kilobytes is hardly a 'large directory'. |> > |> > The problem seems to be with the way VirtualBox mounts directories |> > which isn't an R issue, nor is the fact that copying from Linux to |> > Windows isn't affected. But the fact that it happens only with rds |> > files that use the name of the R object as part of their own names |> > must be an R issue (that surfaces only when other conditions are |> > present). |> > |> > Theories short of divine intervention appreciated. |> > |> > |> > |> > On Tue, 13-Nov-2018 at 02:22PM -0800, William Dunlap wrote: |> > |> > |> Perhaps you got bitten by Dolphin's non-modal dialogs, as described in |> > |> https://userbase.kde.org/Dolphin/File_Management: |> > |> |> > |> Non Modal Dialogs |> > |> |> > |> When Moving, Copying or Deleting files/directories the dialog disappears |> > |> even when the operation has not yet completed. A progress bar then appears |> > |> in the bottom right of the screen, this then disappears also, if you want |> > |> see the progress you need to click a small (i) information icon in the |> > |> system tray. |> > |> |> > |> |> > |> Warning |> > |> New users who are not used to this way of working (and even experienced |> > |> users) can get caught out by this, if you are Moving, Copying or Deleting |> > |> large directories then you need to use the icon to monitor the progress of |> > |> your operation. If you don't then any subsequent actions you do, may well |> > |> use an incomplete file structure resulting in corrupted files. You have |> > |> been warned! |> > |> |> > |> Bill Dunlap |> > |> TIBCO Software |> > |> wdunlap tibco.com |> > |> |> > |> On Tue, Nov 13, 2018 at 2:10 PM, p_connolly <p_connolly at slingshot.co.nz> |> > |> wrote: |> > |> |> > |> > This is getting more strange. |> > |> > |> > |> > I normally copy from the shared folder to the appropriate directory using |> > |> > Dolphin, the KDE file manager. If instead I use the standard bash cp |> > |> > command, no corruption happens -- at least with the limited testing I have |> > |> > done. There also seems to be no problem copying from Linux to Windows. I |> > |> > installed R-3.5.1 for Windows just to eliminate that possible issue. |> > |> > |> > |> > However, R has *something* to do with it because it was used to make the |> > |> > .rds file. Just how the relationship between the name of the R object and |> > |> > the name of the .rds file comes into it, I can't imagine. |> > |> > |> > |> > Thanks for the suggestion William. |> > |> > |> > |> > |> > |> > On 2018-11-14 06:26, William Dunlap wrote: |> > |> > |> > |> >> It seems like copying the files corrupted them. How did you copy them |> > |> >> (with R |> > |> >> or cp or copy or ftp, etc.)? I don't see how this has anything to do |> > |> >> with R. |> > |> >> |> > |> >> Bill Dunlap |> > |> >> TIBCO Software |> > |> >> wdunlap tibco.com [1] |> > |> >> On Mon, Nov 12, 2018 at 7:10 PM, p_connolly |> > |> >> <p_connolly at slingshot.co.nz> wrote: |> > |> >> |> > |> > [...] |> > |> > |> > |> > |> > |> > -- |> > ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. |> > ___ Patrick Connolly |> > {~._.~} Great minds discuss ideas |> > _( Y )_ Average minds discuss events |> > (:_~*~_:) Small minds discuss people |> > (_)-(_) ..... Eleanor Roosevelt |> > |> > ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. |> > |> > ______________________________________________ |> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see |> > https://stat.ethz.ch/mailman/listinfo/r-help |> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html |> > and provide commented, minimal, self-contained, reproducible code. -- ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. ___ Patrick Connolly {~._.~} Great minds discuss ideas _( Y )_ Average minds discuss events (:_~*~_:) Small minds discuss people (_)-(_) ..... Eleanor Roosevelt ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
Patrick Connolly
2018-Dec-01 06:38 UTC
[R] Corrupting files while copying (was Re: saveRDS() and readRDS() Why? [solved, pretty much anyway])
I'd be interested to know if anyone can replicate what I experienced.
VBox version: 5.1.22 r115126 (Qt5.6.2) Versions of R and OSes below:
On Sun, 18-Nov-2018 at 09:44PM +1300, Patrick Connolly wrote:
|> Sequence of steps:
|>
|> Using R-3.5.1 and Windows 7 with the latest Rstudio, shared directory
|> as working directory on Virtual Box host machine:
|>
|> > x <- airquality
|> > saveRDS(x, file = "x.rds")
|> > saveRDS(x, file = "y.rds")
|>
|> On guest machine Mint Linux 17.3, KDE desktop, copy x.rds & y.rds to
|> working directory PWD using file manager Dolphin.
|>
|> (Don't have the precise version of VirtualBox right now.)
Update:
VBox version: 5.1.22 r115126 (Qt5.6.2)
|>
|> > x <- readRDS(file = "x.rds")
|> Error in readRDS(file = "x.rds") : error reading from connection
|> > x <- readRDS(file = "y.rds")
|>
|> > tools::md5sum(c("x.rds", "y.rds"))
|> x.rds y.rds
|> "5fef054848f39b4be02b7c54f1c71a20"
"978a64d1dd342d16a381c9ca728d3665"
|>
|> Yet, if instead of using Dolphin, use bash commands from the shared
|> directory
|>
|> $ cp *.rds ~/PWD/
|>
|> no error reading from the connection or other differences between
|> x.rds and y.rds.
|>
|> head(x)
|>
|> > head(datasets::airquality)
|> Ozone Solar.R Wind Temp Month Day
|> 1 41 190 7.4 67 5 1
|> 2 36 118 8.0 72 5 2
|> 3 12 149 12.6 74 5 3
|> 4 18 313 11.5 62 5 4
|> 5 NA NA 14.3 56 5 5
|> 6 28 NA 14.9 66 5 6
|> >
|>
|> On Thu, 15-Nov-2018 at 09:53AM -0500, Ista Zahn wrote:
|>
|> |> Hi Patrick,
|> |>
|> |> I think it would help to start from the beginning and give complete
|> |> (but concise!) replication instructions, including telling us what
|> |> host and gest operating systems you are using (including the
|> |> versions), the version
|> |> of virtualbox you used, and exactly what steps are needed to
|> |> reproduce the surprising behavior.
|> |>
|> |> Best,
|> |> Ista
|> |> On Wed, Nov 14, 2018 at 2:36 AM Patrick Connolly
|> |> <p_connolly at slingshot.co.nz> wrote:
|> |> >
|> |> > Thanks William,
|> |> >
|> |> > I've used Dolphin for years and never encountered that
phenomenon.
|> |> > Even so, that description doesn't fit what's going on
here. 1.7
|> |> > kilobytes is hardly a 'large directory'.
|> |> >
|> |> > The problem seems to be with the way VirtualBox mounts
directories
|> |> > which isn't an R issue, nor is the fact that copying from
Linux to
|> |> > Windows isn't affected. But the fact that it happens only
with rds
|> |> > files that use the name of the R object as part of their own
names
|> |> > must be an R issue (that surfaces only when other conditions
are
|> |> > present).
|> |> >
|> |> > Theories short of divine intervention appreciated.
|> |> >
|> |> >
|> |> >
|> |> > On Tue, 13-Nov-2018 at 02:22PM -0800, William Dunlap wrote:
|> |> >
|> |> > |> Perhaps you got bitten by Dolphin's non-modal
dialogs, as described in
|> |> > |> https://userbase.kde.org/Dolphin/File_Management:
|> |> > |>
|> |> > |> Non Modal Dialogs
|> |> > |>
|> |> > |> When Moving, Copying or Deleting files/directories the
dialog disappears
|> |> > |> even when the operation has not yet completed. A progress
bar then appears
|> |> > |> in the bottom right of the screen, this then disappears
also, if you want
|> |> > |> see the progress you need to click a small (i)
information icon in the
|> |> > |> system tray.
|> |> > |>
|> |> > |>
|> |> > |> Warning
|> |> > |> New users who are not used to this way of working (and
even experienced
|> |> > |> users) can get caught out by this, if you are Moving,
Copying or Deleting
|> |> > |> large directories then you need to use the icon to
monitor the progress of
|> |> > |> your operation. If you don't then any subsequent
actions you do, may well
|> |> > |> use an incomplete file structure resulting in corrupted
files. You have
|> |> > |> been warned!
|> |> > |>
|> |> > |> Bill Dunlap
|> |> > |> TIBCO Software
|> |> > |> wdunlap tibco.com
|> |> > |>
|> |> > |> On Tue, Nov 13, 2018 at 2:10 PM, p_connolly
<p_connolly at slingshot.co.nz>
|> |> > |> wrote:
|> |> > |>
|> |> > |> > This is getting more strange.
|> |> > |> >
|> |> > |> > I normally copy from the shared folder to the
appropriate directory using
|> |> > |> > Dolphin, the KDE file manager. If instead I use the
standard bash cp
|> |> > |> > command, no corruption happens -- at least with the
limited testing I have
|> |> > |> > done. There also seems to be no problem copying
from Linux to Windows. I
|> |> > |> > installed R-3.5.1 for Windows just to eliminate that
possible issue.
|> |> > |> >
|> |> > |> > However, R has *something* to do with it because it
was used to make the
|> |> > |> > .rds file. Just how the relationship between the
name of the R object and
|> |> > |> > the name of the .rds file comes into it, I can't
imagine.
|> |> > |> >
|> |> > |> > Thanks for the suggestion William.
|> |> > |> >
|> |> > |> >
|> |> > |> > On 2018-11-14 06:26, William Dunlap wrote:
|> |> > |> >
|> |> > |> >> It seems like copying the files corrupted them.
How did you copy them
|> |> > |> >> (with R
|> |> > |> >> or cp or copy or ftp, etc.)? I don't see
how this has anything to do
|> |> > |> >> with R.
|> |> > |> >>
|> |> > |> >> Bill Dunlap
|> |> > |> >> TIBCO Software
|> |> > |> >> wdunlap tibco.com [1]
|> |> > |> >> On Mon, Nov 12, 2018 at 7:10 PM, p_connolly
|> |> > |> >> <p_connolly at slingshot.co.nz> wrote:
|> |> > |> >>
|> |> > |> > [...]
|> |> > |> >
|> |> > |> >
|> |> >
|> |> > --
|> |> >
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
|> |> > ___ Patrick Connolly
|> |> > {~._.~} Great minds discuss ideas
|> |> > _( Y )_ Average minds discuss events
|> |> > (:_~*~_:) Small minds discuss people
|> |> > (_)-(_) ..... Eleanor Roosevelt
|> |> >
|> |> >
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
|> |> >
|> |> > ______________________________________________
|> |> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and
more, see
|> |> > https://stat.ethz.ch/mailman/listinfo/r-help
|> |> > PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
|> |> > and provide commented, minimal, self-contained, reproducible
code.
|>
|> --
|> ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
|> ___ Patrick Connolly
|> {~._.~} Great minds discuss ideas
|> _( Y )_ Average minds discuss events
|> (:_~*~_:) Small minds discuss people
|> (_)-(_) ..... Eleanor Roosevelt
|>
|> ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
|>
|> ______________________________________________
|> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
|> https://stat.ethz.ch/mailman/listinfo/r-help
|> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
|> and provide commented, minimal, self-contained, reproducible code.
--
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
___ Patrick Connolly
{~._.~} Great minds discuss ideas
_( Y )_ Average minds discuss events
(:_~*~_:) Small minds discuss people
(_)-(_) ..... Eleanor Roosevelt
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.