rsync - Jan 2002 - Moving files revisited

I'd like to revisit the topic of moving files from system to system
using rsync.  I've just updated my patch from its 2.5.0 version to
2.5.1, and I'm curious what people think about getting it integrated
into rsync.

The patch comes in two parts.  The first eliminates a potential hang
condition that can happen if the data channel from the receiver to the
generator gets clogged up.  Since my move-files patch is using this
channel to communicate when a file gets successfully written to disk
(from the receiver to the sender via the generator), it needs to ensure
that this hang cannot happen.  The fix is rather complicated (because
the generator is doing a lot of reading and writing of other data), but
I've been using this patch in production conditions for quite a few
months now and haven't encountered any problems yet.  Here's the nohang
patch:

    http://www.clari.net/~wayne/rsync-nohang.patch

The second part of the equation actually adds the --move-files option,
the communication of the receiver back to the sender of which file was
successfully finished, and the actual unlinking of the source file:

    http://www.clari.net/~wayne/rsync-move-files.patch

Comments?

..wayne..

On Wed, 23 Jan 2002, Wayne Davison wrote:
> I'd like to revisit the topic of moving files from system to system
> using rsync.
I'm sad that nobody wanted to talk about --move-files yet, but maybe
this will help things along.  I've adapted the patch files to be based
on the latest CVS source:

    http://www.clari.net/~wayne/rsync-nohang.patch
    http://www.clari.net/~wayne/rsync-move-files.patch

The version for 2.5.1 was renamed:

    http://www.clari.net/~wayne/rsync-2.5.1-nohang.patch
    http://www.clari.net/~wayne/rsync-2.5.1-move-files.patch

If anyone has any questions, let me know.

..wayne..

On Thu, 31 Jan 2002, Dave Dykstra wrote:
> Ouch, is that another byte for every file?  Are there no bits free in
> the "flags" field already in file_struct?
Yes, it is an extra byte per file.  An earlier patch of mine did use
bits in the existing flag word in the current per-file structure,
but since that structure is created before the receiver forks off the
generator, I was thinking that any bit-twiddling of the existing flags
would cause a lot of shared memory between the two processes to cease
being shared (on systems that support copy-on-write forks, such as
Linux).  Thus, I think it would be more memory intensive to use the
existing data structure's flags (but I haven't verified this with actual
memory-size testing).

One solution to this would be to use actual shared memory for the file
structure shared by the receiver & generator.

As for the move-files option, I was thinking that I could write a perl
script that would parse the output of rsync -v and delete files that
were successfully transferred by rsync when they show up in the verbose
output.  If can make that work, the need for my nohang patch isn't as
great, and I could probably come up with a simpler way to keep the redo
channel from filling up (perhaps using a buffer in the receiver process
or looking into how to do some portable shared memory).  Hmm, something
to consider.

..wayne..

Hi all,

I'm having a little trouble using rsync from my windows machine to a Linux
server in a different time zone.  I did some hunting through the
documentation and through the code, but it's not clear to me what rsync
does in this case.

I gather that in principle it uses UTC at both ends.  Will that happen for
a windows client (ssh to a linux box), courtesy of cygwin?

Thanks for any advice.
Rob.


--
Robert Scholten
Eindhoven University of Technology
Physics Department, building N-laag room g2.02
P.O. Box 513, 5600 MB Eindhoven
The Netherlands

Tel:    +31 40 247 4242
Mobile: +31 611 430 467
Fax:    +31 40 245 6050

email: r.scholten@physics.unimelb.edu.au
http://www.ph.unimelb.edu.au/~scholten

>Hi,
I am using rsync to back up some files from a WinXP laptop to a Linux 
server.  The two machines are in different time zones (8 hour 
separation).  It seems that rsync wants to do a full checksum on every file 
because it thinks their time stamps are different.

Example:
         GMT is 9am, Local time (Netherlands) 10am, remote time (Australia) 8pm

In this case, the file was created and copied to both machines with the 
same timestamp (e.g. 8pm) when both machines were in the same timezone 
(Australia).  Then I changed countries with my laptop, and ran 
rsync.  After rsync, the remote (Linux) file has a new timestamp which 8 
hours earlier (e.g. 10am).

I guess that in some sense, rsync "thinks" they were created at
different
universal times, and after rsyncing, they are matched to the same UTC.

This is OK after I have done it once, but would it be possible to tell 
rsync that if the timestamp difference is the same as the current time 
difference, it should ignore?  Or just change the timestamp rather than 
doing a full checksum?  I could write a script to run on the Linux box, to 
change the timestamps by the 8-hour time diff, and revert when I return to 
Australia, but surely this happens regularly to other people with laptops?

Or am I totally confused?

Any help will be appreciated.
Thanks,
Rob.



--
Robert Scholten
Eindhoven University of Technology
Physics Department, building N-laag room g2.02
P.O. Box 513, 5600 MB Eindhoven
The Netherlands

Tel:    +31 40 247 4242
Mobile: +31 611 430 467
Fax:    +31 40 245 6050

email: r.scholten@physics.unimelb.edu.au
http://www.ph.unimelb.edu.au/~scholten

On  4 Apr 2002, Robert Scholten <r.scholten@physics.unimelb.edu.au>
wrote:
> 
> >Hi,
> 
> I am using rsync to back up some files from a WinXP laptop to a Linux 
> server.
Linux stores file times in UTC, and rsync transfers them in UTC.  I
thought that NT and XP did too, but perhaps not, or perhaps there is a
problem with Cygwin.  Changing timezone ought not to have any effect
on rsync.

What does 

  TZ=UTC ls -l 

show on the two boxes before and after changing timezone?

-- 
Martin