rsync - Nov 2007 - Bug or undesired behavior when quota exceeded

Hi,

I have just noticed an undesired behavior when using rsync when disk 
quota is exceeded. I do not know how to avoid it (or maybe it can be 
considered as a bug).

The following is a scenario that leads to the undesired behavior:

1) I normally execute two following commands to synchronize folders 
between a local host and a remote host:
  rsync -aCPvu --existing <dir1> user@host:<dir2>
  rsync -aCPvu --existing user@host:<dir2> <dir1>

2) Assume that files are in-sync on the hosts. Then we change a file on 
the local host. When I execute the first rsync command, the file shall 
be copied to the remote host. When the second command is executed, no 
copying shall occur as the files are in sync after the first command.

3) Unfortunately, it may happen that the disk quota during the first 
command execution is exceeded. I get the following in such a case:
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe 
(32)
rsync: write failed on "<dir2>/<file>": Disk quota
exceeded (122)
rsync error: error in file IO (code 11) at receiver.c(259) 
[receiver=2.6.9]
rsync: connection unexpectedly closed (219584 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) 
[generator=2.6.9]
rsync: connection unexpectedly closed (25270 bytes received so far) 
[sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) 
[sender=2.6.9]

4) The above error is normal and expected.

5) When the second command is invoked, rsync starts copying the file 
from the remote host to the local one. This is an undesired behavior, as 
the file changed on the local host gets overwritten. This is because the 
time of the file on the remote host is left in an incorrect state during 
the first command execution (stopped by the quota limit).

Any idea on how to avoid such a situation? Or maybe we can correct rsync 
code to set file time to the correct one in case of disk quota error?

Regards,
JJ

----------------------------------------------------
Wojna z terrorem wkracza w decyduj?c? faz?:
Robert Redford, Meryl Streep i Tom Cruise w filmie
UKRYTA STRATEGIA - w kinach od 9 listopada!
http://klik.wp.pl/?adr=http%3A%2F%2Fcorto.www.wp.pl%2Fas%2Fstrategia.html&sid=90

On Thu, 2007-11-08 at 12:30 +0100, Janusz Jurski wrote:
> I have just noticed an undesired behavior when using rsync when disk 
> quota is exceeded.
> 1) I normally execute two following commands to synchronize folders 
> between a local host and a remote host:
>   rsync -aCPvu --existing <dir1> user@host:<dir2>
>   rsync -aCPvu --existing user@host:<dir2> <dir1>
> 
> 2) Assume that files are in-sync on the hosts. Then we change a file on 
> the local host. When I execute the first rsync command, the file shall 
> be copied to the remote host. When the second command is executed, no 
> copying shall occur as the files are in sync after the first command.
> 
> 3) Unfortunately, it may happen that the disk quota during the first 
> command execution is exceeded.
> 5) When the second command is invoked, rsync starts copying the file 
> from the remote host to the local one. This is an undesired behavior, as 
> the file changed on the local host gets overwritten. This is because the 
> time of the file on the remote host is left in an incorrect state during 
> the first command execution (stopped by the quota limit).
> 
> Any idea on how to avoid such a situation?
The trouble is that you are passing --partial (-P = --partial
--progress), which makes the receiver in the first command move the
partially received copy of the new source file over the old destination
file.  Remove --partial (by replacing -P with --progress) and you should
be fine: the receiver will discard the partially received file, leaving
the old destination file as is, and the second command will skip
transferring it back to the first host because of --update.  For a more
robust solution, you might look into a program that is designed for
two-way synchronization, such as Unison
( http://www.cis.upenn.edu/~bcpierce/unison/ ).
> Or maybe we can correct rsync 
> code to set file time to the correct one in case of disk quota error?
No.  For one thing, this won't solve the problem because rsync checks
size as well as mtime and the sizes will be different.  Also, we want
the source and destination files to appear different so that if the user
gets the disk quota on the second host raised and attempts the first
command again, the new source file will be retransferred.

Matt

On 2007-11-08 at 15:05, Matt McCutchen wrote:
> On Thu, 2007-11-08 at 12:30 +0100, Janusz Jurski wrote:
> > Any idea on how to avoid such a situation?
> 
> The trouble is that you are passing --partial (-P = --partial
> --progress), which makes the receiver in the first command move the
> partially received copy of the new source file over the old destination
> file.  Remove --partial (by replacing -P with --progress) and you should
> be fine: the receiver will discard the partially received file, leaving
> the old destination file as is,
Thank you for the explanation.

But it seems to me that this is against the description of the
"partial"
option in rsync manual. According to the manual: "By  default, rsync 
will delete any partially transferred file if the transfer is 
interrupted. In some circumstances  it  is  more desirable  to keep 
partially transferred files. Using the --partial option tells rsync to 
keep the partial  file  which  should make  a subsequent transfer of the 
rest of the file much faster." This specifically describes the situation 
when the transfer is interrupted. In my opinion, the partially 
transferred file shall be kept for future reuse and shall not overwrite 
the destination file.

Note that the "partial" option makes no sense otherwise because it 
actually equals to the "in-place" option - "partial" becomes
"in-place"
when the transfer is interrupted and any assumption that the transfer is 
not interrupted is not reasonable in my opinion. Also, the current 
behavior is inconsistent with the "--partial-dir" option (the manual 
says for "--partial-dir" that such partially copied files are kept in 
the specified directory to speed up next transfers).

These are just my thoughts. What is your opinion?
Regards,
JJ

PS: I use "--partial" because the files I transfer are very large and 
the connection speed is not high... I will try using --partial-dir now.

----------------------------------------------------
Mistrzowie stepowania ju? w Polsce! 
Wybierz si? na niecodzienne widowisko z Austrii. 
Zobacz wi?cej:
http://klik.wp.pl/?adr=http%3A%2F%2Fcorto.www.wp.pl%2Fas%2FTap-Dogs.html&sid=89