rsync - Nov 2005 - [Bug 3277] New: [Feature req] "I couldn't do something you asked" warning

https://bugzilla.samba.org/show_bug.cgi?id=3277

           Summary: [Feature req] "I couldn't do something you
asked"
                    warning
           Product: rsync
           Version: 2.6.7
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: foner-rsync-bugzilla@media.mit.edu
         QAContact: rsync-qa@samba.org


I've recently been burned a couple of times by trying to copy filesystems
with
--numeric-id and forgetting to do it as root.  The transfer succeeds just fine,
but of course the owner/group etc info can't be set, and this was
-particularly- easy to overlook because the same users (mostly) existed at both
ends, but with different UIDs, so it's easy to miss that something went
wrong
because mostly you see what looks like "correct" output from ls &
friends, but
is actually wrong because some of the UIDs weren't set, etc.  (This was
actually what I was mostly doing with your recent file-attr-restore script...)

Even though I was running with -v, I don't recall seeing any errors emitted
when this happened.  I'm wondering whether it would be a good idea to create
an
option to make this much more obvious, so the cautious among us can set it by
default in aliases & scripts & whatnot and hence be warned when
we're about to
make an annoying-to-fix mistake; it would also make it less critical to
carefully scan for these hard-to-notice failures---after all, rsync is supposed
to -save- us from that sort of thing...

Just enabling it with excessive verbosity is probably not the right thing,
since it'd get lost in the noise, and most people probably don't run at
high
verbosities if they're not -already- debugging something.  I'm of mixed
opinions about whether the behavior should be a default, since it could be
irritating to see thousands of "couldn't set uid/gid" messages if
you're okay
with not running as root.  If the option is enabled, a warning should
definitely be emitted at the end of the run (where it's not buried) that
something went wrong (and probably the exit status should change).  And perhaps
there should be another option that emits a warning for each file?  Or
something like --warn-me=none,end,all or something like that.

There are probably other things besides uid/gid that might benefit from this
(tiemstamps? ACLs?), but I fear either a huge number of warnings if they're
all
enabled, or the hair of a warning bitmask ("tell me about UID/GID, but not
about problems setting permissions") or something.

If this looks like the sort of thing that might be better hashed out on the
list, I can forward this there instead; let me know.

Thanks!


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED




------- Comment #1 from wayned@samba.org  2005-12-02 18:34 MST -------
I think that this problem is limited to user and group settings when not run as
root. The complication is that -a implies -o and -g, but we don't want
errors
to be returned when running as a normal user for a normal copy. One possible
solution is to add some kind of an extra mode where rsync knows that it should
definitely try setting the user and/or group and report a failure if it cannot.

There is a diff in the patches dir named owner-group-mod.diff that chooses to
force the user to specify -oo and/or -gg to tell rsync to trying setting all
users and/or all groups no matter what.  I have debated whether -aog should be
enough to also trigger these new actions or if (as the patch used to be
implemented) -oogg is still required even if -a was also specified.  Any
thoughts?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277


foner-rsync-bugzilla@media.mit.edu changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |foner-rsync-
                   |                            |bugzilla@media.mit.edu




------- Comment #2 from foner-rsync-bugzilla@media.mit.edu  2005-12-08 16:16 MST
-------
I think what I''d like to see is a single option called --strict or
something
(surely there''s a better name) which I can either set all the time
(e.g., by
calling rsync via an alias that sets it), or a corresponding environment
variable I can set in my init.  (And, of course, either/both would have to take
--nostrict so I can turn it off on a case-by-case basis if necessary.)  If I
have to remember to set a separate option, I''ll probably forget, and
that''s
exactly what I''m trying to avoid---I typically wind up typing (or
scripting,
even worse) some rsync command and then realizing days or weeks later that I
-meant- to get owners/groups set but didn''t happen to run the command
as root,
by which point it''s way too late. And this seems conceptually simpler
to me
than the peculiar other ways of specifying owner/group you mention below--and
extensible to other cases where the user may want to know that rsync
wasn''t
quite able to fulfill the request (e.g., ACL''s or timestamps maybe).

I''m not sure whether it should complain on every file or just issue a
warning
at the end, though.  The former might produce a lot of output, but OTOH,
it''d
sure be hard to miss, and you''d only get it if you said --strict in the
first
place...


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277





------- Comment #3 from hashproduct@verizon.net  2005-12-09 16:19 MST -------
Instead of treating -o and -g specially, I think it is time rsync had separate
archive options for "running as a normal user for a normal copy", i.e.
using
default destination security settings, and preserving all security settings.

Since there are tons of existing backup scripts that rely on -a/--archive
preserving all security settings, I think -a should continue to behave as it
does now, except that inability to set user or group always causes an error
message.  A new option can modify -a so that it does not imply -pog.  I
recommend calling this option -U/--def-security; -U, mnemonic for "use
umask",
is probably the best letter that is not already taken.

For good measure, one could also add an option -s/--security that preserves all
security settings (-pog).  That way, people can start to use "rsync -as
..." in
their backup scripts, and it might eventually be practical to change the
behavior of unqualified -a to something like the tar-esque "-rltD, plus -s
if
run as root".

Once the two modes are available, I think rsync should take all options
seriously (i.e. -ogD) whether or not it is root.  I also think it should issue
a warning about every failure (i.e. every file) because all three of these
options can succeed on some files and fail on others and it''s important
that
the user gets the message.  A concise output format that coalesces
preservations that failed for the same reason might be good:
    rsync: "/foo/badfile": failed to preserve time, owner, group:
Operation not
permitted (1)
    rsync: "/foo/badfile": failed to preserve ACLs: Operation not
supported (2)

I might try to make some of these changes in my custom rsync.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED




------- Comment #4 from wayned@samba.org  2006-01-29 01:33 MST -------
The CVS version now has an option named --super that tells rsync to attempt to
perform super-user activities even if it is not being run by the super-user
(set file ownership, preserve all groups instead of just the current user's
groups, and to create devices (which are now a separate class from special
files).  This both enables warnings on an rsync run where you're expecting
all
those things to work, and also makes it possible for rsync to work on a system
where it doesn't need a UID of 0 to perform those operations.  Also added
was
the --no-super option that allows root access to the non-super-user version of
the -g option.

If the user wants to make "-a --super" easier to type, consider a popt
alias in
your ~/.popt file to define this as the -s option:

rsync alias -s -a --super

(Perhaps an official -s option should be added that means that...)


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277


hashproduct@verizon.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hashproduct@verizon.net




------- Comment #5 from hashproduct@verizon.net  2006-01-29 10:57 MST -------
I like the idea of --super and --no-super overriding rsync's own detection
of
whether it is running as root.  But at the risk of annoying Wayne, I still
think rsync's behavior with respect to file security settings should be
reconsidered.

Right now, rsync offers two behaviors when -a is given:
    (1) Preserve permissions but not ownership or devices.  Preserve groups to
        which the user running rsync belongs.
    (2) Preserve permissions, owners, groups, and devices, producing an error
        every time it is not possible to do so.
Rsync currently chooses #1 or #2 based on whether it is root, and this choice
can now be overridden with --super or --no-super.

Many people (including me and Lenny) use rsync for backups, and the stricter
checking of #2 is obviously preferred for backups.  Presumably #1 is for a
"normal user for a normal copy", and indeed, I use rsync as a normal
user for a
normal copy several times a day.  I definitely want to recurse and preserve
links, modtimes, and (if there are any) special files.

-a would be perfect except for one thing: I often copy files between areas with
different "ambient" permissions (which I have configured with default
ACLs).
When I copy files from my home directory (700) to my Web site (755) or to a
shared group workspace (2770), I expect them to take on permissions appropriate
for the destination!  When they don't, I fix them manually with chmod; if I
forget to do so, annoyance can result.

For example, visitors to my school's Web site encountered a 403 Forbidden
error
on a PDF file, which surprised me because I had personally configured the site
with a default ACL of 775.  It turned out that another administrator had
uploaded the file, which for some reason had come out of the PDF creator with
600 permissions, using a program that preserved permissions.  I think rsync can
do better.

I want this behavior:
    (3) Preserve everything not security related, but use appropriate
        security settings (permissions, group) for the destination.
-a incorrectly second-guesses my intent; even -p ANDs with source permissions. 
I have been using the following incantation, which I named cp2:
        rsync -rltE --chmod=ugo=rwX ...
(-E is my custom rsync's --executability; ignore it throughout my comment
unless the official rsync adopts this feature.)

I guess I could make a personal popt alias for -rltE --chmod=ugo=rwX, but if -a
--super is to have an official alias, could I please have one for behavior #3? 
I venture to say that it is at least as useful as #1.  In fact, would anyone
object if the non-root meaning of -a changed to behavior #3?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277





------- Comment #6 from wayned@samba.org  2006-01-30 11:56 MST -------
(In reply to comment #5)
> In fact, would anyone object if the non-root meaning of -a changed
> to behavior #3? 
Yes -- I would.  I do a lot of copying where I want the permissions and groups
copied exactly from the source.
>         rsync -rltE --chmod=ugo=rwX ...
I believe you could change the (custom) -E option in that option set into a -p
and it would behave the same way (due to how X works in the chmod option and
the fact that you are forcing the read and write bits on).


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277





------- Comment #7 from hashproduct@verizon.net  2006-01-30 14:36 MST -------
(In reply to comment #6)
> Yes -- I would.  I do a lot of copying where I want the permissions and
groups
> copied exactly from the source.
If you care about the permissions and the groups, I bet you would want to be
warned if there were an owner or a group rsync couldn't preserve.  If so,
ask
for behavior #2 by using -s.
> I believe you could change the (custom) -E option in that option set into a
-p
> and it would behave the same way (due to how X works in the chmod option
and
> the fact that you are forcing the read and write bits on).
I want permissions for existing files unchanged, and I want permissions for new
files masked by destination default permissions; those are the opposites of the
two behaviors that -p activates.

The sole purpose of the --chmod=ugo=rwX is to render ineffectual the AND-ing
with source permissions for new files.  The files come over the wire with 666
or 777 permissions, and the receiver ANDs that with destination default
permissions; that way, a file copied from a private area to a public one can
gain permissions.

For exactly the same reason, I create my tar packages with embedded 666 or 777
permissions using tar --mode=ugo=rwX.  I see it as discourteous to embed other
permissions because that second-guesses the intent of the person extracting the
archive.  I ranted about this on the bug-tar list a while ago, and then the
embedded permissions in kernel source tarballs mysteriously changed from
644/755 to 666/777; I speculate that I may have been the reason.  Enough
digression.

In the rare event that an existing file changes executability on the source, I
want the same to happen on the destination.  That is the purpose of my custom
-E, which affects only existing files.  Subversion set a precedent of
considering a file's executability to be as essential a counterpart to its
logical data as its modtime, even though executability is (unfortunately)
encoded in file security settings.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277





------- Comment #8 from wayned@samba.org  2006-01-30 15:12 MST -------
This is not really the place to hold a tangent conversatation, but no, I
don't
want behavior #2, as I'm not running as root (but I do have multiple groups
associated with my user).

Thanks for the other permissions clarifications.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=3277





------- Comment #9 from hashproduct@verizon.net  2006-01-30 16:31 MST -------
I'm sorry for abusing your Bugzilla.  Let the discussion continue on the
mailing list if necessary.  Never mind changing the behavior of -a, but an
official alias for -rltE --chmod=ugo=rwX would still be nice; I'll request
it
as an enhancement on Bugzilla if that would be appropriate.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.