That's the way it is. If it's really a one-off change, a huge change in
your structure, telnet ssh, rsh, and so forth, work really well for dropping in
and deleting stuff (unless you're supplying the master, and other systems
out of your control copy
from you). Rsync is optimized for taking a filesystem in an unknown state, and
making it identical to another filesystem in an unknown state, using network
bandwidth as efficiently as possible.
Once again, please ignore my stupid confidentiality notice. New notes client,
required upgrade, can't delete the danged thing.
The information contained in this message is confidential and may be legally
privileged. The message is intended solely for the addressee(s). If you are not
the intended recipient, you are hereby notified that any use, dissemination, or
reproduction is
strictly prohibited and may be unlawful. If you are not the intended recipient,
please contact the sender by return e-mail and destroy all copies of the
original message.
Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips
Available as n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
".\n" '
"There are some who call me.... Tim?"
Justin Banks
<justinb@tricord. To: rsync@lists.samba.org
com> cc: (bcc: Tim
Conway/LMT/SC/PHILIPS)
Sent by: Subject: rsync recursion
question
rsync-admin@lists
.samba.org Classification:
10/23/2001 11:37
AM
Please respond to
justinb
Hello -
I have a problem/question about rsync's recursion and directory
replication. Here's the problem. Say I have a large filesystem, on the order
of a TB. It's all under /mnt/bigfs on host "local". In /mnt/bigfs,
I have
three directories, a, b, and c. I remove directory a. I want to propagate
this change to the host "remote", and what I'd like to do is have
host
"remote" just remove directory a recursively. The only way I can see
to do
that is to send directory /mnt/bigfs recursively from host "sender".
This
doesn't work very well, though, 'cause it takes a *really* long time to
walk
the whole filesystem. I know the only thing that's changed is the existence
of
directory a - how to I propagate that change, and that change only?
-justinb
--
Justin Banks Tricord, Inc. justinb@tricord.com
The Church says that the earth is flat, but I know that it is round, for
I have seen the shadow on the moon, and I have more faith in a shadow
than in the Church. --Ferdinand Magellan
Hi List, I was going through the rsync archives to find any patches to deal with hanging issues, seems to be there are patches but there are no instructions on how to apply these patches. For instance the following email points to the links where we can download patches ---------------------------------------------------- On Tue, 10 Jul 2001, Steve Ladendorf wrote:> Is there something I'm doing wrong? What can I doto prevent rsync from> hanging all the time??If you avoid using -v, that can help rsync not to hang. However, a better fix is to apply this patch: http://www.clari.net/~wayne/rsync-nohang1.patch This fixes a very common hang problem, and makes -v safe to use again. Since you are also transferring large quantities of files, I would be very interested in knowing if rsync still hangs after applying that patch. If it does, try applying this patch: http://www.clari.net/~wayne/rsync-nohang2.patch The second patch fixes a much rarer bug, and it would be great to know if someone had run into it in simple, real-world conditions. ..wayne.. -------------------------------------------------- How do I apply these patches, could someone help me on that. Thanks in advance Raj ==== __________________________________________________ Do You Yahoo!? Make a great connection at Yahoo! Personals. http://personals.yahoo.com
Now, that's a good solution. if it MUST be done over rsync, and it's
not there, write it in yourself. It's amazing, sometimes, the things people
want added into a generic tool, and they expect Tridge or Dave to write and
maintain it. It's a tool for
non-interactively maintaining directory trees, and adding in an option to do a
"rsh remotehost rm" wouldn't be a normal function. Your situation
is different. Almost nobody writes to rsyncd, especially not huge trees. Glad
to see it's working well for
you.
Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips
Available as n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
".\n" '
"There are some who call me.... Tim?"
Justin Banks
<justinb@tricord. To: Tim
Conway/LMT/SC/PHILIPS@AMEC
com> cc: justinb@tricord.com
Sent by: rsync@lists.samba.org
rsync-admin@lists rsync-admin@lists.samba.org
.samba.org Subject: Re: rsync recursion
question
Classification:
10/23/2001 08:08
PM
Please respond to
justinb
>>>>> "Tim" == tim conway
<tim.conway@philips.com> writes:
Tim> That's the way it is. If it's really a one-off change, a huge
change
Tim> in your structure, telnet ssh, rsh, and so forth, work really well for
Tim> dropping in and deleting stuff (unless you're supplying the
master, and
Tim> other systems out of your control copy from you). Rsync is opTimized
Tim> for taking a filesystem in an unknown state, and making it identical
to
Tim> another filesystem in an unknown state, using network bandwidth as
Tim> efficiently as possible.
Well, that doesn't cut it here. It seemed like all the guts were there - I
mean, the functionality already exists on the receiving side, mostly, right?
Anyway, I took a look, and I added an option (-d). This means that you can do
rsync --delete -d /some/removed/directory foo@wherever::module:/some/removed
and the remote side will remove /some/removed/directory. It will work whether
it's a file or directory. Let me know if anyone wants a patch, if this is a
feature that would help other folks. I can't imagine people are in my
particular circumstances, where the source filesystem is on the order of a
terabyte, but you never know ;)
-justinb
--
Justin Banks Tricord, Inc. justinb@tricord.com
'We have no intention of shipping another bloated operating system and
forcing that down the throats of our Windows customers'
-- Paul Maritz, Microsoft Group Vice President
On Tue, Oct 23, 2001 at 09:08:25PM -0500, Justin Banks wrote:> >>>>> "Tim" == tim conway <tim.conway@philips.com> writes: > > Tim> That's the way it is. If it's really a one-off change, a huge change > Tim> in your structure, telnet ssh, rsh, and so forth, work really well for > Tim> dropping in and deleting stuff (unless you're supplying the master, and > Tim> other systems out of your control copy from you). Rsync is opTimized > Tim> for taking a filesystem in an unknown state, and making it identical to > Tim> another filesystem in an unknown state, using network bandwidth as > Tim> efficiently as possible. > > Well, that doesn't cut it here. It seemed like all the guts were there - I > mean, the functionality already exists on the receiving side, mostly, right? > Anyway, I took a look, and I added an option (-d). This means that you can do > > rsync --delete -d /some/removed/directory foo@wherever::module:/some/removed > > and the remote side will remove /some/removed/directory. It will work whether > it's a file or directory. Let me know if anyone wants a patch, if this is a > feature that would help other folks. I can't imagine people are in my > particular circumstances, where the source filesystem is on the order of a > terabyte, but you never know ;)This rang a bell. Back in December 1998 while I was official maintainer of rsync, Phil Dietz submitted a patch to remotely make directories on rsync --daemons and discussed one to remotely delete directories. I was opposed to the former, and came up with a way to do the latter without a patch. Unfortunately the messages were not on the mailing list so I can't refer them to you. The way to recursively delete a single directory while ignoring the rest is: mkdir /tmp/empty rsync -a --delete --include "/directorytodelete**" --exclude "*" \ /tmp/empty/ foo@wherever::module - Dave Dykstra
On Wed, Oct 24, 2001 at 12:42:29PM -0500, Justin Banks wrote:> >>>>> "Dave" == Dave Dykstra <dwd@bell-labs.com> writes: > Dave> The way to recursively delete a single directory while ignoring the > Dave> rest is: > > Dave> mkdir /tmp/empty rsync -a --delete --include "/directorytodelete**" > Dave> --exclude "*" \ /tmp/empty/ Dave> foo@wherever::module > > This works, if there's nothing else on the receiving side that you want to > keep (unless I'm doing it wrong). In other words, if the sending side looks > like this : > > [root@r1c3n104 foo]# ls -l /tmp/foo > total 4 > drwxr-xr-x 2 root root 4096 Oct 24 12:32 bleen > -rw-r--r-- 1 root root 0 Oct 23 15:27 fred > > and the remote side looks like this : > > [root@r1c1n130 /tmp]# ls -l /tmp/foo > total 8 > drwxr-xr-x 2 root root 4096 Oct 24 12:34 bleen > drwxr-xr-x 2 root root 4096 Oct 24 11:52 bob > -rw-r--r-- 1 root root 0 Oct 24 12:31 fred > > > and I want to remove /tmp/foo/bob on the remote side without walking the > entire /tmp/foo/bleen directory, which may contain several hundred gigabytes, > I can follow your suggestion, as follows : > > [root@r1c3n104 foo]# rsync -a --delete --include "/tmp/foo/bob**" -exclude "*" > /tmp/empty/ user@10.55.11.130::module/tmp/foo/ > > but then the remote side ends up like this : > > [root@r1c1n130 /tmp]# ls -l /tmp/foo > total 0 > > ;(The filename paths that the exclude algorithm sees are relative to the destination path, they do not include it. It should work if you use --include "/bob**". The slash at the beginning does not indicate the root of a filesystem, it indicates "match only the beginning of a path".> What I want to be able to do is have the initial picture as above, and run > something like this (/tmp/foo/bob doesn't exist on the sending side) : > > [root@r1c3n104 foo]# /tmp/rsync -d --delete --force --delete-after /tmp/foo/bob > user@10.55.11.130::module/tmp/foo > > and have the remote side end up like this : > > [root@r1c1n130 /tmp]# ls -l /tmp/foo/ > total 4 > drwxr-xr-x 2 root root 4096 Oct 24 12:36 bleen > -rw-r--r-- 1 root root 0 Oct 24 12:36 fred > > without walking the /tmp/foo/bleen tree at all. This seems to make lots of > sense to me. It seems like the intuitive thing to do. You ought to be able to > sync something that doesn't exist on the sending side, and have it go away on > the remote side, without wierd extra voodoo. > > Anyway, as I said, I have a patch to make this work, and I'd rather it went > in, 'cause then I wouldn't be out of synch with the rest of the world ;)I'm not the official maintainer anymore, but I think there's little chance of your patch being accepted. - Dave Dykstra
Justin Banks [justinb@tricord.com] writes:> If your suggestion worked, that would be just fine with > me. Actually, I guess it's fine anyway, I'll just have to maintain > my patch ;)This is probably obvious, but just in case it isn't, CVS makes this fairly trivial (importing the main rsync releases and then developing your own changes on the mainline) to maintain over time. Tracking local changes to third party sources is one of its strengths. That's how I maintain our internal version of rsync which has a variety of local changes. Most I eventually submit back for possible inclusion in the main release (after some burn-in time in local use), but there are some that aren't general purpose enough, so they just stay in our repository. -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: db3l@fitlinxx.com / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/
On Wed, Oct 24, 2001 at 02:26:29PM -0500, Justin Banks wrote:> >>>>> "Dave" == Dave Dykstra <dwd@bell-labs.com> writes: > > Dave> The filename paths that the exclude algorithm sees are relative to the > Dave> destination path, they do not include it. It should work if you use > Dave> --include "/bob**". The slash at the beginning does not indicate the > Dave> root of a filesystem, it indicates "match only the beginning of a > Dave> path". > > It doesn't, unless I'm misunderstanding what you're saying. > > rsync -a --ignore-errors --delete --include "/bob**" -exclude "*" /tmp/empty/ > user@10.55.11.130::module/tmp/foo/bob > > results in everything in /tmp/foo being removed.I don't understand how it can affect anything above the destination directory you give; you should be giving a destination directory of /tmp/foo. Is that what you meant? I tested it and it worked for me. - Dave Dykstra
On Wed, Oct 24, 2001 at 04:31:06PM -0500, Justin Banks wrote:> >>>>> "Dave" == Dave Dykstra <dwd@bell-labs.com> writes: > > Dave> On Wed, Oct 24, 2001 at 02:26:29PM -0500, Justin Banks wrote: > >> >>>>> "Dave" == Dave Dykstra <dwd@bell-labs.com> writes: > >> > Dave> The filename paths that the exclude algorithm sees are relative to the > Dave> destination path, they do not include it. It should work if you use > Dave> --include "/bob**". The slash at the beginning does not indicate the > Dave> root of a filesystem, it indicates "match only the beginning of a > Dave> path". > >> It doesn't, unless I'm misunderstanding what you're saying. > >> > >> rsync -a --ignore-errors --delete --include "/bob**" -exclude "*" > >> /tmp/empty/ user@10.55.11.130::module/tmp/foo/bob > >> > >> results in everything in /tmp/foo being removed. > > Dave> I don't understand how it can affect anything above the destination > Dave> directory you give; you should be giving a destination directory of > Dave> /tmp/foo. Is that what you meant? I tested it and it worked for me. > > It's not that it affects /tmp/foo itself, it's that it removes /tmp/foo/*, not > just the one I want to remove. It does give a destination of /tmp/foo, but I > want to keep /tmp/foo/bleen, and remove /tmp/foo/bob.I think you misunderstood me. Your destination in the above command is user@10.55.11.130::module/tmp/foo/bob but in the previous message it was user@10.55.11.130::module/tmp/foo/ It should be the one without the "bob" on the end, and I can't see how anything can be affected in the module/tmp/foo directory if you've got "bob" on the end. - Dave Dykstra