Hi, All,
I am implementing real-time Rsync on Windows 2008 system. I set up Rsync
server and Rsync client on two machines. An windows service is watching all
the Windows file events with FileSystemWatcher. However, the service
cannot tell the exactly what happened to folders such as create, delete, or
modified. So, I ignored folder event, and only catch file changing events.
After I catch all those files changed, then call Rsync client periodically
to transfer changes to the server.
Now the problems arises. Because Rsync can only create one level folder but
recursively creating missing folders, sometime, my Rsync client failed
because it need to create more than 1 level of folders on destination to
hold the new file.
Example "rsync -av source/l1/l2/l3/abc.txt
dest::Worksheets/l1/l2/l3/"
The above command is generated by my windows file watching system to make
sure source and destination have the same folder structure. If the
destination only misses l3 folder, it will creates automatically. However,
it misses l2 also, Rsync server will return back an error.
Example:
"C:\Program Files (x86)\cwRsync\bin\rsync" -av
"/cygdrive/e/fileserver/Worksheets/scotia/stalk/download/work/ctrl.2972"
"lba023::Rsync/scotia/stalk/download/work/"
sending incremental file list
rsync: mkdir "scotia/stalk/download/work" (in Rsync) failed: No such
file or
dir
ectory (2)
rsync error: error in file IO (code 11) at main.c(577) [Receiver=3.0.7]
rsync: connection unexpectedly closed (5 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(610)
[sender=3.0.7]
So, I changed main.c to create folder recursively. The following are the
changes.
if (mkdir_defmode(dest_path) != 0)
{
/*create folder recursively*/
char parentFolder[200];
char *pFound = dest_path + 1;//ignore the first /
STRUCT_STAT st1;
while( (pFound = strchr(pFound, '/')) != NULL)
{
memcpy(parentFolder, dest_path, pFound - dest_path);
parentFolder[pFound - dest_path] = '\0';
if ( do_stat(parentFolder, &st1) == 0) {
/* If the destination is a dir, enter it and use
mode 1. */
if (S_ISDIR(st1.st_mode)) {
pFound = pFound + 1;
continue;
}
}
if( mkdir_defmode( parentFolder ) != 0 )
{
rsyserr(FERROR, errno, "mkdir recursive parent
folder %s for child folder %s failed",
full_fname(parentFolder),
full_fname(dest_path));
exit_cleanup(RERR_FILEIO);
}
pFound = pFound + 1;
}
if( mkdir_defmode( dest_path ) != 0 )
{
rsyserr(FERROR, errno, "mkdir %s failed",
full_fname(dest_path));
exit_cleanup(RERR_FILEIO);
}
}
The original code is:
if (mkdir_defmode(dest_path) != 0) {
rsyserr(FERROR, errno, "mkdir %s failed",
full_fname(dest_path));
exit_cleanup(RERR_FILEIO);
}
The problems related to this change is that the folder time stamp of the
folder could not be same as source.
If you have comments, please let me know.
Thanks,
Steven Xu
Wayne Davison
2010-Nov-09 21:48 UTC
Changes made to main.c on implementing real time Rsync
On Tue, Nov 9, 2010 at 11:04 AM, Steven Xu <steven.xu at lba.ca> wrote:> Example "rsync -av source/l1/l2/l3/abc.txt dest::Worksheets/l1/l2/l3/" >Do this instead: rsync -avR source/./l1/l2/l3/abc.txt dest::Worksheets/ The use -f -R (--relative) along with the /./ path element to signal how much of the source path goes into the copy will make rsync create all the needed dirs on the destination. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20101109/34fc64cf/attachment.html>
Hi, Wayne, Thanks for your suggestion. Your solution is good but not fit my case. If there are a lot of files or folders under l1, rsync -avR source/./l1/l2/l3/abc.txt dest::Worksheets/ will check all the files and folders under l1. Actually, I only care about abc.txt. Thanks, Steven. From: 4wayned at gmail.com [mailto:4wayned at gmail.com] On Behalf Of Wayne Davison Sent: Tuesday, November 09, 2010 4:48 PM To: Steven Xu Cc: rsync at lists.samba.org Subject: Re: Changes made to main.c on implementing real time Rsync On Tue, Nov 9, 2010 at 11:04 AM, Steven Xu <steven.xu at lba.ca> wrote: Example "rsync -av source/l1/l2/l3/abc.txt dest::Worksheets/l1/l2/l3/" Do this instead: rsync -avR source/./l1/l2/l3/abc.txt dest::Worksheets/ The use -f -R (--relative) along with the /./ path element to signal how much of the source path goes into the copy will make rsync create all the needed dirs on the destination. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20101110/d4ed2a9f/attachment.html>
Wayne Davison
2010-Nov-10 23:18 UTC
Changes made to main.c on implementing real time Rsync
On Wed, Nov 10, 2010 at 6:37 AM, Steven Xu <steven.xu at lba.ca> wrote:> Your solution is good but not fit my case. If there are a lot of files > or folders under l1, rsync -avR source/./l1/l2/l3/abc.txt dest::Worksheets/ > will check all the files and folders under l1. >What makes you think that? It just transfers abc.txt including the path elements. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20101110/39227c64/attachment.html>
Hi, Wayne, Thanks for your reply. I just did a test: Here is the Rsync client output: C:\Program Files (x86)\cwRsync\bin\rsync" -av "/cygdrive/e/fileserver/Worksheets/reconcil/./Fiducie Desjardins/Dailies/cl0016062moveq20101110.txt" "lba023::Rsync/" sending incremental file list cl0016062moveq20101110.txt sent 3051907 bytes received 27 bytes 2034622.67 bytes/sec total size is 3051437 speedup is 1.00 Here is the server log: 2010/11/11 10:42:34 [4904] rsync to Rsync/ from lba006 (10.132.208.9) 2010/11/11 10:42:34 [4904] receiving file list 2010/11/11 10:42:35 [4904] recv lba006 [10.132.208.9] Rsync () cl0016062moveq20101110.txt 3051437 2010/11/11 10:42:35 [4904] sent 54 bytes received 3051927 bytes total size 3051437 The path information is not copied over the destinations. If I use the following command, I got error. C:\Program Files (x86)\cwRsync\bin\rsync" -av "/cygdrive/e/fileserver/Worksheets/reconcil/./Fiducie Desjardins/Dailies/cl0016062moveq20101110.txt" "lba023::Rsync/Fiducie Desjardins/Dailies/" sending incremental file list rsync: mkdir "Fiducie Desjardins/Dailies" (in Rsync) failed: No such file or dir ectory (2) rsync error: error in file IO (code 11) at main.c(577) [Receiver=3.0.7] rsync: connection unexpectedly closed (5 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(610) [sender 3.0.7] Thanks. Steven From: 4wayned at gmail.com [mailto:4wayned at gmail.com] On Behalf Of Wayne Davison Sent: Tuesday, November 09, 2010 4:48 PM To: Steven Xu Cc: rsync at lists.samba.org Subject: Re: Changes made to main.c on implementing real time Rsync On Tue, Nov 9, 2010 at 11:04 AM, Steven Xu <steven.xu at lba.ca> wrote: Example "rsync -av source/l1/l2/l3/abc.txt dest::Worksheets/l1/l2/l3/" Do this instead: rsync -avR source/./l1/l2/l3/abc.txt dest::Worksheets/ The use -f -R (--relative) along with the /./ path element to signal how much of the source path goes into the copy will make rsync create all the needed dirs on the destination. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20101111/e0b07a04/attachment.html>
Benjamin R. Haskell
2010-Nov-11 16:41 UTC
Changes made to main.c on implementing real time Rsync
On Thu, 11 Nov 2010, Steven Xu wrote:> > Hi, Wayne, > > Thanks for your reply. > > I just did a test: > > Here is the Rsync client output: > > C:\Program Files (x86)\cwRsync\bin\rsync" -av "/cygdrive/e/fileserver/Worksheets/reconcil/./Fiducie Desjardins/Dailies/cl0016062moveq20101110.txt" > ??"lba023::Rsync/"You've left out the '-R' option, which was the other half of Wayne's response (the '-R' option and the '/./' in the source): "C:\Program Files (x86)\cwRsync\bin\rsync" -Rav "/cygdrive/e/fileserver/Worksheets/reconcil/./Fiducie Desjardins/Dailies/cl0016062moveq20101110.txt"?"lba023::Rsync/" -R (part of the '-Rav' [equivalent to -avR or -a -v -R]), short for --relative, is what makes the difference: it tells rsync that the '/./' in the first path name is where the path starts (the client will send everything after the /./ as the path name). Wayne's original explanation:> Do this instead: > > rsync -avR source/./l1/l2/l3/abc.txt dest::Worksheets/ > > The use of -R (--relative) along with the /./ path element to signal > how much of the source path goes into the copy will make rsync create > all the needed dirs on the destination. > > ..wayne..-- Best, Ben