I am sorry if this has been covered before: I have done a couple of
futile searches in the bug reporting system and is there any way to
search the archive?
I am having a strange symptom: I am synching directories (that have
very long file names, by the time the full path is specified) and a lot
of hard links. It seems that the directory is being copied piecemeal -
that
is, if I run rsync enough times, the entire contents ultimately get
copied. It seems like I am running into some hard limit in the size of
the filelist or something.
I am running 2.4.6 on linux - the source directory is remote mounted on
solaris, the destination is linux.
For instance, I have a directory that is in the tree that contains 173
files at the source - most of which are hard links - here is the effect
of an ls | wc on the destination after five succesive identical runs of
rsync on the source (this is in a subdirectory of one of the directories
in the command below). The directory did not exist before running the
sync.
[root@ks-s0-107-1- SC]# ls | wc
49 49 1286
[root@ks-s0-107-1- SC]# ls | wc
85 85 2234
[root@ks-s0-107-1- SC]# ls | wc
120 120 3243
[root@ks-s0-107-1- SC]# ls | wc
152 152 4112
[root@ks-s0-107-1- SC]# ls | wc
173 173 4739
So that it seems to have synched 49, then 36, then 35, then 32, the
finally the last 21 files in the directory. (The increment seems to vary
if I try it again, i.e. from 49 to 90).
I get no error mesages (that I can see). In fact, the first time I run
the program, it seems to notice all
the files and produces 163 messages that "file blah-de-blah" is a hard
link, but then doesn't
seem to make the link for file "blah-de-blah" - this behavior remains
constant with each succesive run.
Here is the rsync command (generated by a perl script and broken into
little pieces by my mailer)
rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats
--progress -tpgoCHr -v -v --include install/ --include
install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include
install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/
--include install/redhat-7.1/zantaz_rpm_upgrade/ --include
install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/
--include
install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/
--include install/kickstart/zantaz_upgrade/20011121/** --include
install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/**
--include
install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/**
--exclude "*" /net/swdev/staging/* root@10.1.41.92:/
Thanks,
Dave
Unfortunately there is no way to search the archive. That would be very useful. I haven't heard of any similar problems reported with hard links before, and I've been following this list closely for several years. I notice your command line looks pretty complicated, so I suggest that you try to narrow it down to the smallest reproducible case, preferably one that you can completely describe to someone else how to reproduce starting from scratch. Often such an exercise alone will reveal a solution, but if not at least it allows somebody else to debug it. - Dave Dykstra On Tue, Nov 27, 2001 at 11:29:54AM -0800, Dave Madole wrote:> > I am sorry if this has been covered before: I have done a couple of > futile searches in the bug reporting system and is there any way to > search the archive? > I am having a strange symptom: I am synching directories (that have > very long file names, by the time the full path is specified) and a lot > of hard links. It seems that the directory is being copied piecemeal - > that > is, if I run rsync enough times, the entire contents ultimately get > copied. It seems like I am running into some hard limit in the size of > the filelist or something. > > I am running 2.4.6 on linux - the source directory is remote mounted on > solaris, the destination is linux. > > For instance, I have a directory that is in the tree that contains 173 > files at the source - most of which are hard links - here is the effect > of an ls | wc on the destination after five succesive identical runs of > rsync on the source (this is in a subdirectory of one of the directories > in the command below). The directory did not exist before running the > sync. > > [root@ks-s0-107-1- SC]# ls | wc > 49 49 1286 > [root@ks-s0-107-1- SC]# ls | wc > 85 85 2234 > [root@ks-s0-107-1- SC]# ls | wc > 120 120 3243 > [root@ks-s0-107-1- SC]# ls | wc > 152 152 4112 > [root@ks-s0-107-1- SC]# ls | wc > 173 173 4739 > > So that it seems to have synched 49, then 36, then 35, then 32, the > finally the last 21 files in the directory. (The increment seems to vary > if I try it again, i.e. from 49 to 90). > > I get no error mesages (that I can see). In fact, the first time I run > the program, it seems to notice all > the files and produces 163 messages that "file blah-de-blah" is a hard > link, but then doesn't > seem to make the link for file "blah-de-blah" - this behavior remains > constant with each succesive run. > > Here is the rsync command (generated by a perl script and broken into > little pieces by my mailer) > > rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats > --progress -tpgoCHr -v -v --include install/ --include > install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include > install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ > --include install/redhat-7.1/zantaz_rpm_upgrade/ --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ > --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ > --include install/kickstart/zantaz_upgrade/20011121/** --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** > --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** > --exclude "*" /net/swdev/staging/* root@10.1.41.92:/ > > Thanks, > > Dave > > > > > >
On 27 Nov 2001, Dave Dykstra <dwd@bell-labs.com> wrote:> > Unfortunately there is no way to search the archive. That would be very > useful.Just use google and say site:lists.samba.org rsync mbp prototype or whatever. -- Martin
Dave Madole
2001-Nov-28 14:32 UTC
More - cause - not all files synched - program hangs in select
The bug has something to do with verbosity - it works fine without verbosity on. The irony, of course, is that one turns verbosity on to fix these things (I have applied the hang patch, BTW). The last thing I see is the "match_report" message, then the program hangs in select. Here are the symptoms: I am syncing a number of parallel directories that contain large numbers hard links. At the source: dir a: f1 f2 f3 f4 f5 dir b: f1 f2 f3 f4 f5 dir c: f1 f2 f3 f4 f5 f1 in each directory is a hard link to every other f1, etc. (note, there are actually about 10 directories of 100 files in each directory) After my first run of rsync, I get something like this. dir a: f1 (link count 3) f2 (3) f3 (1) f4 (1) f5 (1) dir b: f1 (3) f2 (3) dir c: f1 (3) f2 (3) After my second run of rsync, I get something like this. dir a: f1 (3) f2 (3) f3 (3) f4 (1) f5 (1) dir b: f1 (3) f2 (3) f3 (3) dir c: f1 (3) f2 (3) f3 (3) After my third run of rsync, I get something like this. dir a: f1 (3) f2 (3) f3 (3) f4 (3) f5 (1) dir b: f1 (3) f2 (3) f3 (3) f4 (3) dir c: f1 (3) f2 (3) f3 (3) f4 (3) etc. So that dir a: is completely copied the first time, but only the beginnings of dir b and c. With each successive run, dirs b & c (& d & e & f) fill up, until it really is done. Again, this doesn't happen if I turn verbosity entirely off. Perhaps somebody is closing some file descriptor - I noticed that the program was hanging on the remote box in "select" - and then somebody else (probably having to do with making links) is trying to write to it. Thus, the program hangs but the timing is not always the same and sometimes a few more or less links are made before it hangs. It seems that making the hard links is done in a second phase after the real file transfers are done, at which time something must be closed. I could look through the source code and fix it myself but maybe by the time I get in tomorrow somebody that knows it well will have done so. Dave Dave Madole wrote:> I am sorry if this has been covered before: I have done a couple of > futile searches in the bug reporting system and is there any way to > search the archive? > > I am having a strange symptom: I am synching directories (that have > very long file names, by the time the full path is specified) and a lot > of hard links. It seems that the directory is being copied piecemeal - > that > is, if I run rsync enough times, the entire contents ultimately get > copied. It seems like I am running into some hard limit in the size of > the filelist or something. > > I am running 2.4.6 on linux - the source directory is remote mounted on > solaris, the destination is linux. > > For instance, I have a directory that is in the tree that contains 173 > files at the source - most of which are hard links - here is the effect > of an ls | wc on the destination after five succesive identical runs of > rsync on the source (this is in a subdirectory of one of the directories > in the command below). The directory did not exist before running the > sync. > > [root@ks-s0-107-1- SC]# ls | wc > 49 49 1286 > [root@ks-s0-107-1- SC]# ls | wc > 85 85 2234 > [root@ks-s0-107-1- SC]# ls | wc > 120 120 3243 > [root@ks-s0-107-1- SC]# ls | wc > 152 152 4112 > [root@ks-s0-107-1- SC]# ls | wc > 173 173 4739 > > So that it seems to have synched 49, then 36, then 35, then 32, the > finally the last 21 files in the directory. (The increment seems to vary > if I try it again, i.e. from 49 to 90). > > I get no error mesages (that I can see). In fact, the first time I run > the program, it seems to notice all > the files and produces 163 messages that "file blah-de-blah" is a hard > link, but then doesn't > seem to make the link for file "blah-de-blah" - this behavior remains > constant with each succesive run. > > Here is the rsync command (generated by a perl script and broken into > little pieces by my mailer) > > rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats > --progress -tpgoCHr -v -v --include install/ --include > install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include > install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ > --include install/redhat-7.1/zantaz_rpm_upgrade/ --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ > --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ > --include install/kickstart/zantaz_upgrade/20011121/** --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** > --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** > --exclude "*" /net/swdev/staging/* root@10.1.41.92:/ > > Thanks, > > Dave
On Wed, Nov 28, 2001 at 10:55:55AM +1100, Martin Pool wrote:> On 27 Nov 2001, Dave Dykstra <dwd@bell-labs.com> wrote: > > > > Unfortunately there is no way to search the archive. That would be very > > useful. > > Just use google and say > > site:lists.samba.org rsync mbp prototype > > or whatever.Cool. Any way to list that information on lists.samba.org, or better yet provide a form that uses google to do the search? - Dave