Hello,
I'm trying to break up my rsync process by separating a directory tree
into multiple rsync processes because I'm witnessing some errors trying
to rsync large directory trees on Windows machines. After breaking up
the tree I tried to rsync each individual directory starting from the
bottom directory on up using the command:
foreach ($array as $directory){ /* $array = list of
directories gathered from the larger directory I'm trying to rsync */
rsync -vz -e ssh username@hostname:/src/$directory/
/dst/$directory/
}
But this comes up with the error:
client: nothing to do: perhaps you need to specify some
filenames or the --recursive option?
which is exactly what I'm trying to avoid.
I've also tried:
rsync -vz -e ssh username@hostname:/src/$directory/*
/dst/$directory/.
but that states that /src/$directory/* does not exist.
Does anyone have a suggestion as to how to perfom this task?
Thanks for your time.
Sincerely,
Mark J. de Jong
,.,.,.,...,.,,.,..,.,....,.,..,.,..,.,.,,.,...,..,,...
Senior Network Engineer - Secure Dog Hosting, Inc.
P.703.256.2869 F.703.256.3810 C.571.212.0027
http://www.secdog.com
> I'm trying to break up my rsync process by separating a directory tree > into multiple rsync processes because I'm witnessing some errors trying > to rsync large directory trees on Windows machines. After breaking up > the tree I tried to rsync each individual directory starting from the > bottom directory on up using the command: > > foreach ($array as $directory){ /* $array = list of > directories gathered from the larger directory I'm trying to rsync */ > rsync -vz -e ssh username@hostname:/src/$directory/ > /dst/$directory/ > } >To synchronize all the files in this directory, I believe you will need the --recursive option. This would work without the recursive option if your 'foreach' statement iterated over all files in the tree. Try this instead: rsync -vrz -e ssh username@hostname:/src/$directory/ /dst/$directory/ Good luck, David
I often use the -R option
example:
system1:/bigdir to system2:/backups/bigdir
list=bigdir/subdir/subdir...
cd /
rsync -HaRz listitem system2:/backups
that will put subdir/subdir as system2:/backups/bigdir/subdir/subdir
now, if you have items above that one, you've got more complexity, as you
can't do bigdir -depth=1 or something like that.
what you can do is
rsync -HRz bigdir/* system2:/backups
What I actually did was to write a recursive shell script to process the
output of find, giving only items with with less than some maximum number
of subcomponents. Of course, a file next to a directory that was an item
would have no subcomponets at all, and would be sent alone.
The only lack in these processes is that you can't send over an empty
directory. The only way to get that is to have the empty directory be
under another directory sent recursively.
Here's my splitter.
+++++++++++++++++++++++++++++++++++++++++++
#!/bin/sh
limit=$1
file=$2
splitdir(){
dir=$1
pathlength=`echo $dir |tr / ' '|wc -w`
pathlength=`echo $pathlength`
searchpat="^$dir/"
[ "$searchpat" = "^/" ] && searchpat='^'
grep $searchpat $file |
cut -d/ -f1-`expr $pathlength + 1` |
uniq -c |
while read dircount subdir
do
if [ "$dircount" -le "$limit" ]
then
echo $subdir
else
(splitdir $subdir)
fi
done
}
splitdir
+++++++++++++++++++++++++++++++++++++++++++++
Obviously, if you go a couple-hundred directories deep, it'll use a lot of
resources.
file ($2) is a file made by find.
in the example above, limiting to 100000 items per rsync run:
cd /
find bigdir -print >listfile
splitter 100000 listfile
cat listfile |while read item
do
rsync -WHaRz $item system2:/backups
done
recursively sending a file is, sensibly, treated as sending the file
itself. The bad thing about breaking things like this up is that it makes
the -H option less meaningful, since if you have multiple links to a file,
but they are all done in seperate runs, the link relationship is lost.
Tim Conway
tim.conway@philips.com
303.682.4917 office, 303.921.0301 cell
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, caesupport2 on AIM
"There are some who call me.... Tim?"
"David Rasch" <rasch@raschnet.com>
Sent by: rsync-admin@lists.samba.org
07/30/2002 02:53 PM
To: <rsync@lists.samba.org>
cc: (bcc: Tim Conway/LMT/SC/PHILIPS)
Subject: RE: Rsync recursion
Classification:
> I'm trying to break up my rsync process by separating a directory tree
> into multiple rsync processes because I'm witnessing some errors trying
> to rsync large directory trees on Windows machines. After breaking up
> the tree I tried to rsync each individual directory starting from the
> bottom directory on up using the command:
>
> foreach ($array as $directory){ /*
$array = list of> directories gathered from the larger directory I'm trying to rsync */
> rsync -vz -e ssh
username@hostname:/src/$directory/> /dst/$directory/
> }
>
To synchronize all the files in this directory, I believe you will need
the --recursive option. This would work without the recursive option if
your 'foreach' statement iterated over all files in the tree.
Try this instead:
rsync -vrz -e ssh username@hostname:/src/$directory/
/dst/$directory/
Good luck,
David
--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html