I work on software that archives gigabytes of files to multiple sites. Occasionally one or two files have no read permissions: % ls -l dir/foo --w-------+ 1 abcserve myusers 11222 Jan 10 03:14 The error message is: rsync: send_files failed to open "/dir/foo" (in xxx): Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.9] -- To make matters worse, the file is on a read-only snapshot mounted through NFS. In any case root squash is in effect, so super-user is out. I fully realize that the correct answer is "fix the file permissions and regenerate". Which is precisely what we do today. However, regenerating takes several hours, so no one has access to the archives until it is done. The one work-around I have found is to redo the rsync with --exclude='/dir/foo'. I am thinking about adding a facility to my software that generates the rsync commands to allow adding options. It strikes me though, that what I really want is an rsync option '--ignore-files-with-no-read-perms'. This rightfully should be off by default, but once you have a system up and running like we do, it says "if we screw up the read permissions on an occasional file just keep going". I would expect that '--itemize-changes' would make a note that the file was not copied. In my case, the files have always had no read access for anyone, and are owned by id running rsync. That is, if the files *were* on a writable volume, chmod.would fix it (but not --fake-super, nor any combination of --no-perms and --chmod=ugo+r that I tried on /tmp on my desktop). That is, there is no possibility of anyone other than someone with root access to the file system making a copy of this file. Having an option to ignore seems safe enough. Ignoring return code 23 (Partial transfer due to error), similar to the FAQ suggestion for code 24 (Partial transfer due to vanished source files) seems much too risky. I only want to ignore the case of the files being unreadable because they were explicitly denied read permissions, not because of media failure or (my favorite vendor response) "ionizing radiation". Am I missing some obvious settings that can achieve this? An hour spent googling, reading the rsync posts on Stack Overflow, and the last 14 months of archives on this list found nothing. Ps. If this does make sense I am volunteering to make the change. -- Allen.Supynuk at gmail.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I am not really sure what you are requesting here. Rsync does continue after it encounters a permission denied error. It just can't copy that particular file/directory but it will continue on through the tree. The reason for the second error message and the exit 23 is to inform you that there was a problem (if you are running with -v the rest of the tree can easily scroll the real error off the screen). When such an error is encountered --delete will stop deleting stuff unless you use --ignore-errors but rsync will finish copying what it can. On 01/10/13 21:07, Allen Supynuk wrote:> I work on software that archives gigabytes of files to multiple > sites. > > Occasionally one or two files have no read permissions: > > % ls -l dir/foo --w-------+ 1 abcserve myusers 11222 Jan 10 03:14 > > The error message is: rsync: send_files failed to open "/dir/foo" > (in xxx): Permission denied (13) > > rsync error: some files/attrs were not transferred (see previous > errors) (code 23) at main.c(1518) [generator=3.0.9] > > -- > > To make matters worse, the file is on a read-only snapshot mounted > through NFS. In any case root squash is in effect, so super-user > is out. > > I fully realize that the correct answer is "fix the file > permissions and regenerate". Which is precisely what we do today. > However, regenerating takes several hours, so no one has access to > the archives until it is done. > > The one work-around I have found is to redo the rsync with > --exclude='/dir/foo'. I am thinking about adding a facility to my > software that generates the rsync commands to allow adding > options. > > It strikes me though, that what I really want is an rsync option > '--ignore-files-with-no-read-perms'. This rightfully should be off > by default, but once you have a system up and running like we do, > it says "if we screw up the read permissions on an occasional file > just keep going". I would expect that '--itemize-changes' would > make a note that the file was not copied. > > In my case, the files have always had no read access for anyone, > and are owned by id running rsync. That is, if the files *were* on > a writable volume, chmod.would fix it (but not --fake-super, nor > any combination of --no-perms and --chmod=ugo+r that I tried on > /tmp on my desktop). That is, there is no possibility of anyone > other than someone with root access to the file system making a > copy of this file. Having an option to ignore seems safe enough. > > Ignoring return code 23 (Partial transfer due to error), similar > to the FAQ suggestion for code 24 (Partial transfer due to > vanished source files) seems much too risky. I only want to ignore > the case of the files being unreadable because they were explicitly > denied read permissions, not because of media failure or (my > favorite vendor response) "ionizing radiation". > > Am I missing some obvious settings that can achieve this? An hour > spent googling, reading the rsync posts on Stack Overflow, and the > last 14 months of archives on this list found nothing. > > Ps. If this does make sense I am volunteering to make the change. >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlDvdVgACgkQVKC1jlbQAQdyYACgl6xfquP8SbayVESwPIfAvdfg F7kAoMaDolNKoytHRnFI4OMrrUvtxTt8 =Fxor -----END PGP SIGNATURE-----
Hi, On 01/10/2013 08:07:18 PM, Allen Supynuk wrote:> It strikes me though, that what I really want is an rsync option > '--ignore-files-with-no-read-perms'.While you're at it, why not a more generic --ignore-errno foo where foo is (a defined subset of) the Posix error numbers listed in errno(3). The names, alternately, could be used. So your application would use --ignore-errno EACCES Regards, Karl <kop at meme.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein
Francis.Montagnac at inria.fr
2013-Feb-02 16:09 UTC
Does rsync need a --ignore-unreadable-files option?
Hi. On Fri, 11 Jan 2013 03:08:13 +0100 Allen Supynuk wrote:> To make matters worse, the file is on a read-only snapshot mounted > through NFS. In any case root squash is in effect, so super-user is > out.Why not mounting this no_root_squash? Is it not wrong to have partial archives (missing unreadable files) in your case? -- Francis.Montagnac at inria.fr
It also means running the backups as root. Running as root in a root squash environment is risky. In my case the (read-only snapshot of the) source is considered broken if there are unreadable files, so I have no grounds for a security exception. I want the option to have the backup complete with a warning instead of stop part way thru with an error. I realize that the current behavior is the correct default. -Allen Sent from my iPad On Feb 2, 2013, at 11:09, Francis.Montagnac at inria.fr wrote:> > Hi. > > On Fri, 11 Jan 2013 03:08:13 +0100 Allen Supynuk wrote: > >> To make matters worse, the file is on a read-only snapshot mounted >> through NFS. In any case root squash is in effect, so super-user is >> out. > > Why not mounting this no_root_squash? > > Is it not wrong to have partial archives (missing unreadable files) in > your case? > > -- > Francis.Montagnac at inria.fr