So, like a lot of people, I am using --link-dest to do backups. My backup target is ext4 so with a hard link limit of 64K. I do end up with trying to create too many links at some point though and get the following sequence of events: rsync: link "/.snapshots/hourly.0/linux/usr/src/openwrt-r18617/build_dir/host/mpfr-2.4.1/tests/.deps/tui_sub.Po" => usr/src/openwrt-r18617/build_dir/host/bison-2.3/examples/calc++/.deps/calc++-driver.Po failed: Too many links (31) Hlink node data for 2282618 already has path=usr/src/openwrt-r18617/build_dir/host/bison-2.3/examples/calc++/.deps/calc++-driver.Po (usr/src/openwrt-r18617/build_dir/host/mpfr-2.4.1/tests/.deps/tui_sub.Po) rsync error: errors with program diagnostics (code 13) at hlink.c(548) [generator=3.0.9] rsync: connection unexpectedly closed (222362 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] It seems awfully heavy-handed for this "Too many links" condition to be treated as an outright error. Surely it's sufficient to just fail to hard-link and move on, yes? Any thoughts? Cheers, b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20120624/c416e630/attachment.pgp>
On 06/24/2012 10:40:11 AM, Brian J. Murrell wrote:> So, like a lot of people, I am using --link-dest to do backups. My > backup target is ext4 so with a hard link limit of 64K. I do end up > with trying to create too many links at some point though and get the > following sequence of events:<snip>>> It seems awfully heavy-handed for this "Too many links" condition to > be treated as an outright error. Surely it's sufficient to just fail > to hard-link and move on, yes?Typically the pathnames that run out of hard links are those that are already hard linked in the source file system. At least if you're using -H as well to ensure that your backup is a real copy of the source system. The problem is that "just fail(ing) to hard link" can cause subsequent backup(s) to fail depending on the state in which the failure leaves the pathnames in question, more than one pathname because I'm assuming the source system has hardlinking going on at the point where the problem occurs, not to mention the unknown state of the pathnames in the initial rsync. Although "moving on" does get you something, you're not going to get a good backup without some manual work. Speaking as a sysadmin I want some predictability in my backup system and even a small problem with a backup will put me to work until I resolve the issue. Agreed, a "mostly good" backup is better than no backup at all. Even better though would be to have an actual good backup all the time. This could be done by making --link-dest an "atomic(ish)" operation; either all of the source pathhames sharing an inode would be --link-dest-ed or none of them would. The kludgey way to do this is to stat(2) before each --link-dest to see if there's enough free links before proceeding. I say kludgey because of the possibility of race conditions. (I also have not looked into how you'd discover the hard link limit for the filesystem in question.) An alternate approach is to, on "too many links" error, "undo" all the --link-dests on the "problem paths". (I've looked at the code and, at first glance, this seems like it might be sane. I posted an email, of unknown clarity, to this list on this subject and never heard back.) The goal is to have --link-dest work when possible and when not to copy the source and have the same set of hard links on the destination side as the source side. Note that this is all hot air on my part. I've no time now to try to code anything -- even assuming that a patch on this matter would be accepted. Apologies if the above is not clearly written. I have limited opportunity to write and thought a quick brain dump would be better than nothing. Regards, Karl <kop at meme.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein e looked at the code and
Hi all, I have an rsyncd installed on a host and I want to use it from a lot of clients. This rsync is not invoked directly but there is a shell script wrapper co set up parameters and environment. My problem is that I do not want to allow any other rsync to be able to connect to the daemon. Is there any way to implement such feature? Thanks Andras
Hello, You could simply create a dedicated group, set up permissions to your script, and add only the desired remote users to that group. Greg On Tue, Jun 26, 2012 at 2:18 PM, Andr?s Porjesz <andras.porjesz at ericsson.com> wrote:> Hi all, > I have an rsyncd installed on a host and I want to use it from a lot of > clients. This rsync is not invoked directly but there is a shell script > wrapper co set up parameters and environment. > My problem is that I do not want to allow any other rsync to be able to > connect to the daemon. Is there any way to implement such feature? > > Thanks > Andras > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20120626/15fa453c/attachment.html>
And you can also restrict users given their IPs : Add at the top of the rsyncd.conf "hosts allow = 10.42.20.43/32" Greg On Tue, Jun 26, 2012 at 2:38 PM, Greg Deback (rsync) < greg.deb+rsync at gmail.com> wrote:> Hello, > You could simply create a dedicated group, set up permissions to your > script, and add only the desired remote users to that group. > Greg > > On Tue, Jun 26, 2012 at 2:18 PM, Andr?s Porjesz < > andras.porjesz at ericsson.com> wrote: > >> Hi all, >> I have an rsyncd installed on a host and I want to use it from a lot of >> clients. This rsync is not invoked directly but there is a shell script >> wrapper co set up parameters and environment. >> My problem is that I do not want to allow any other rsync to be able to >> connect to the daemon. Is there any way to implement such feature? >> >> Thanks >> Andras >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20120626/71c9ce35/attachment.html>
Hi, thanks, but actually the same users want to use rsync without the wrapper and it will cause inproper transfer. Andras ________________________________ From: greg.deb at gmail.com [mailto:greg.deb at gmail.com] On Behalf Of Greg Deback (rsync) Sent: Tuesday, June 26, 2012 14:38 To: Andr?s Porjesz Cc: rsync at lists.samba.org Subject: Re: Rsync pairing Hello, You could simply create a dedicated group, set up permissions to your script, and add only the desired remote users to that group. Greg On Tue, Jun 26, 2012 at 2:18 PM, Andr?s Porjesz <andras.porjesz at ericsson.com<mailto:andras.porjesz at ericsson.com>> wrote: Hi all, I have an rsyncd installed on a host and I want to use it from a lot of clients. This rsync is not invoked directly but there is a shell script wrapper co set up parameters and environment. My problem is that I do not want to allow any other rsync to be able to connect to the daemon. Is there any way to implement such feature? Thanks Andras -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20120626/dbab0315/attachment.html>
Okay, then you can do the following : - create a dedicated group "rsync" - chown/chmod /bin/rsync for that group only - add a setgid to your script with that group - use IP control in the rsyncd.conf to restrict users Thus only allowed users can access the script, and only the script can launch rsync Greg On Tue, Jun 26, 2012 at 2:41 PM, Andr?s Porjesz <andras.porjesz at ericsson.com> wrote:> ** > Hi, > thanks, but actually the same users want to use rsync without the wrapper > and it will cause inproper transfer. > > Andras > > ------------------------------ > *From:* greg.deb at gmail.com [mailto:greg.deb at gmail.com] *On Behalf Of *Greg > Deback (rsync) > *Sent:* Tuesday, June 26, 2012 14:38 > *To:* Andr?s Porjesz > *Cc:* rsync at lists.samba.org > *Subject:* Re: Rsync pairing > > Hello, > You could simply create a dedicated group, set up permissions to your > script, and add only the desired remote users to that group. > Greg > > On Tue, Jun 26, 2012 at 2:18 PM, Andr?s Porjesz < > andras.porjesz at ericsson.com> wrote: > >> Hi all, >> I have an rsyncd installed on a host and I want to use it from a lot of >> clients. This rsync is not invoked directly but there is a shell script >> wrapper co set up parameters and environment. >> My problem is that I do not want to allow any other rsync to be able to >> connect to the daemon. Is there any way to implement such feature? >> >> Thanks >> Andras >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20120626/7a5788a7/attachment.html>
Hi Greg, thanks, but actually they donwloaded an rsync from somewhere and started to use it.... Andras ________________________________ From: greg.deb at gmail.com [mailto:greg.deb at gmail.com] On Behalf Of Greg Deback (rsync) Sent: Tuesday, June 26, 2012 14:44 To: Andr?s Porjesz Cc: rsync at lists.samba.org Subject: Re: Rsync pairing Okay, then you can do the following : - create a dedicated group "rsync" - chown/chmod /bin/rsync for that group only - add a setgid to your script with that group - use IP control in the rsyncd.conf to restrict users Thus only allowed users can access the script, and only the script can launch rsync Greg On Tue, Jun 26, 2012 at 2:41 PM, Andr?s Porjesz <andras.porjesz at ericsson.com<mailto:andras.porjesz at ericsson.com>> wrote: Hi, thanks, but actually the same users want to use rsync without the wrapper and it will cause inproper transfer. Andras ________________________________ From: greg.deb at gmail.com<mailto:greg.deb at gmail.com> [mailto:greg.deb at gmail.com<mailto:greg.deb at gmail.com>] On Behalf Of Greg Deback (rsync) Sent: Tuesday, June 26, 2012 14:38 To: Andr?s Porjesz Cc: rsync at lists.samba.org<mailto:rsync at lists.samba.org> Subject: Re: Rsync pairing Hello, You could simply create a dedicated group, set up permissions to your script, and add only the desired remote users to that group. Greg On Tue, Jun 26, 2012 at 2:18 PM, Andr?s Porjesz <andras.porjesz at ericsson.com<mailto:andras.porjesz at ericsson.com>> wrote: Hi all, I have an rsyncd installed on a host and I want to use it from a lot of clients. This rsync is not invoked directly but there is a shell script wrapper co set up parameters and environment. My problem is that I do not want to allow any other rsync to be able to connect to the daemon. Is there any way to implement such feature? Thanks Andras -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20120626/bd943884/attachment.html>