Hello, I am the guy behind the ext2/ext3 patches for Extended Attributes and ACLs, and I've recently been asked about ACL support in rsync by Eric. Upon investigating I found that you have an ACL patch against rsync-2.5.5 [1]. I also found some other postings to rsync@lists.samba.org concerning rsync and ACLs [3]. Are there any plans for finalizing an integrating that rsync ACL patch? I am posting my own thoughts on that topic with the hope to spur the discussion and accelerate the improvement of rsync in that direction. ACLs are one part of supporting Extended Attributes in general, but they are important enough (and difficult enough to do right) to deserve special treatment. (I would like to see Extended Attributes in rsync too, of course.) Most UNIX systems support some variant of POSIX ACLs. Unfortunately the so-far-final draft 17 document the dissolved POSIX 1003.1e/2c working group has produced does not define how to deal with ACLs on a network. Probably partly because POSIX ACLs didn't ever get standardized, the NFSv4 protocol [4] among other things defines yet another kind of ACLs. NFSv4 ACLs are much more like Windows ACLs than POSIX draft 17 ACLs. What's more, the NFSv4 protocol not only defines the on-the-wire format to be used for ACLs, but also their semantics. This makes them problematic for POSIX ACLs. Nevertheless it seems that NFSv4 ACLs are here to stay. So it seems to make sense to adapt them to POSIX ACLs, and to use them as the underlying transfer format for rsync. The SSH File Transfer Protocol <http://www.ietf.org/internet-drafts/draft-ietf-secsh-filexfer-04.txt> also specifies that scp is to use the NFSv4 ACL format, by the way. Marius Aamodt Eriksen <marius@umich.edu> has thought out a mapping between NFSv4 ACLs and POSIX ACLs [5]. While Marius's mapping most likely is semantically correct, I think that it is too complex to be useful practically. The main problem is to define a mapping for the POSIX ACL mask entry. I would recommend to transfer the ACL MASK entry as a proper ACL entry in NFSv4 ACLs with a who field of "MASK@", and to extend the permission evaluation mechanism of NFSv4 to take care for this additional entry. At least as far as rsync is concerned, this proposed approach could be used without causing compatibility problems. Another issue that has surely been considered for rsync is how to map between users/groups across different systems. On UNIX like systems this mapping can be done based on user/group IDs or names. If it is relevant for rsync to transfer both ID's and names between systems, this will be another problem with the NFSv4 ACL format. (In star [6], an implementation of the PAX archive format defined in POSIX.1-2001, for storing ACLs, we have been using a text based format which is almost identical to what acl_to_text(3) produces, but with ID's added. The exact format used is documented in the file README.ACL inside the package. This approach is less powerful than NFSv4 ACLs, but good enough for POSIX ACL backups.) Best regards, Andreas Gruenbacher. REFERENCES [1] Buck Huppmann: patch: rsync-2.5.5: UNIX ACL support, <http://www.mail-archive.com/rsync@lists.samba.org/msg05573.html>. [2] Gary Fernandez: [PATCH] change rsync to print warning if ACL detected, <http://www.mail-archive.com/rsync@lists.samba.org/msg04988.html>. [3] General rsync ACL discussions, <http://www.mail-archive.com/rsync@lists.samba.org/msg04310.html>, <http://www.mail-archive.com/rsync@lists.samba.org/msg01727.html>. [4] NFS version 4 Protocol, <http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-rfc3010bis-05.txt>. [5] Marius Aamodt Eriksen: Mapping Between NFSv4 and Posix Draft ACLs, <http://www.citi.umich.edu/u/marius/draft-eriksen-nfsv4-acl-01.txt> [6] J?rg Schilling: Star, <http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html>
On Sun, Feb 23, 2003 at 01:13:56PM +0100, Andreas Gruenbacher wrote:> Hello, > > I am the guy behind the ext2/ext3 patches for Extended Attributes and ACLs, > and I've recently been asked about ACL support in rsync by Eric. Upon > investigating I found that you have an ACL patch against rsync-2.5.5 [1]. I > also found some other postings to rsync@lists.samba.org concerning rsync and > ACLs [3]. > > Are there any plans for finalizing an integrating that rsync ACL patch? > > I am posting my own thoughts on that topic with the hope to spur the > discussion and accelerate the improvement of rsync in that direction. > > ACLs are one part of supporting Extended Attributes in general, but they are > important enough (and difficult enough to do right) to deserve special > treatment. (I would like to see Extended Attributes in rsync too, of course.) > Most UNIX systems support some variant of POSIX ACLs. Unfortunately the > so-far-final draft 17 document the dissolved POSIX 1003.1e/2c working group > has produced does not define how to deal with ACLs on a network. > > Probably partly because POSIX ACLs didn't ever get standardized, the NFSv4 > protocol [4] among other things defines yet another kind of ACLs. NFSv4 ACLs > are much more like Windows ACLs than POSIX draft 17 ACLs. What's more, the > NFSv4 protocol not only defines the on-the-wire format to be used for ACLs, > but also their semantics. This makes them problematic for POSIX ACLs. > Nevertheless it seems that NFSv4 ACLs are here to stay. > > So it seems to make sense to adapt them to POSIX ACLs, and to use them as the > underlying transfer format for rsync. The SSH File Transfer Protocol > <http://www.ietf.org/internet-drafts/draft-ietf-secsh-filexfer-04.txt> also > specifies that scp is to use the NFSv4 ACL format, by the way. > > Marius Aamodt Eriksen <marius@umich.edu> has thought out a mapping between > NFSv4 ACLs and POSIX ACLs [5]. While Marius's mapping most likely is > semantically correct, I think that it is too complex to be useful > practically. The main problem is to define a mapping for the POSIX ACL mask > entry. I would recommend to transfer the ACL MASK entry as a proper ACL entry > in NFSv4 ACLs with a who field of "MASK@", and to extend the permission > evaluation mechanism of NFSv4 to take care for this additional entry. > > At least as far as rsync is concerned, this proposed approach could be used > without causing compatibility problems. > > Another issue that has surely been considered for rsync is how to map between > users/groups across different systems. On UNIX like systems this mapping can > be done based on user/group IDs or names. If it is relevant for rsync to > transfer both ID's and names between systems, this will be another problem > with the NFSv4 ACL format. > > (In star [6], an implementation of the PAX archive format defined in > POSIX.1-2001, for storing ACLs, we have been using a text based format which > is almost identical to what acl_to_text(3) produces, but with ID's added. The > exact format used is documented in the file README.ACL inside the package. > This approach is less powerful than NFSv4 ACLs, but good enough for POSIX ACL > backups.) > > Best regards, > Andreas Gruenbacher. > > > REFERENCES > > [1] Buck Huppmann: patch: rsync-2.5.5: UNIX ACL support, > <http://www.mail-archive.com/rsync@lists.samba.org/msg05573.html>.Lots of code but no commentary. I'm not too inclined to wade through this until after an overview of what it does has been given.> [2] Gary Fernandez: [PATCH] change rsync to print warning if ACL detected, > <http://www.mail-archive.com/rsync@lists.samba.org/msg04988.html>.This is mildly usefull. Unfortunately it seems to be Solaris 2+ specific.> [3] General rsync ACL discussions, > <http://www.mail-archive.com/rsync@lists.samba.org/msg04310.html>,Nice wrapper for rsync to apply acls after syncing.> <http://www.mail-archive.com/rsync@lists.samba.org/msg01727.html>.Q: will rsync support ACLs? A: Maybe the vendor tar does, you could sync a tar file. A: no known plans for now, maybe in 3.0, heterogeneous support is difficult. A: Not standardized and problems with out-of-band support. A: Here's a wrapper to get around the deficiency.> [4] NFS version 4 Protocol, > <http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-rfc3010bis-05.txt>.The whole protocol document in plain text. Section 5 (pages 36-58) is the applicable one. Much of this involves negotiation of supported functionality and is further overlaid by other stuff that is runtime oriented rather than storage. This is still worth the read and i will reading it in greater depth later.> [5] Marius Aamodt Eriksen: Mapping Between NFSv4 and Posix Draft ACLs, > <http://www.citi.umich.edu/u/marius/draft-eriksen-nfsv4-acl-01.txt>Brief description of one-way mapping from POSIX to NFSv4 with brief descriptions of how each work.> > [6] J?rg Schilling: Star, > <http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html>homepage of a tar replacement that supports FreeBSD and Linux ACLs. There might be something wothwhile here wouldn't know without reading the source. ACL portability does not seem to be addressed. I'll come back to this later. I want to get these brief assesments of the referencees out quickly. Thanks Andreas Gruenbacher for this posting. It brings up some valuable points for consideration and reviewing these references has given me a better insight of what is and, more importantly, what is not sallient to the ACL discussion. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
On Sun, Feb 23, 2003 at 01:13:56PM +0100, Andreas Gruenbacher wrote:> Hello, > > I am the guy behind the ext2/ext3 patches for Extended Attributes and ACLs,Thanks.> and I've recently been asked about ACL support in rsync by Eric. Upon > investigating I found that you have an ACL patch against rsync-2.5.5 [1]. I > also found some other postings to rsync@lists.samba.org concerning rsync and > ACLs [3]. > > Are there any plans for finalizing an integrating that rsync ACL patch?Not yet. I'd like to see a decent overview of that patch's approach before commenting on its chances.> I am posting my own thoughts on that topic with the hope to spur the > discussion and accelerate the improvement of rsync in that direction. > > ACLs are one part of supporting Extended Attributes in general, but they are > important enough (and difficult enough to do right) to deserve special > treatment. (I would like to see Extended Attributes in rsync too, of course.)As long as i have touched on the issue for rsync i have argued for treating ACLs (file Access Control Lists) as a special-case of EAs (file Extended Attributes). I still believe that is essential. Aside from the blurred line between ACLs and EAs in some filesystems there is the big issue of inconsistent availability of support for these.> Most UNIX systems support some variant of POSIX ACLs. Unfortunately the > so-far-final draft 17 document the dissolved POSIX 1003.1e/2c working group > has produced does not define how to deal with ACLs on a network. > Probably partly because POSIX ACLs didn't ever get standardized, the NFSv4 > protocol [4] among other things defines yet another kind of ACLs. NFSv4 ACLs > are much more like Windows ACLs than POSIX draft 17 ACLs. What's more, the > NFSv4 protocol not only defines the on-the-wire format to be used for ACLs, > but also their semantics. This makes them problematic for POSIX ACLs. > Nevertheless it seems that NFSv4 ACLs are here to stay. > > So it seems to make sense to adapt them to POSIX ACLs, and to use them as the > underlying transfer format for rsync. The SSH File Transfer Protocol > <http://www.ietf.org/internet-drafts/draft-ietf-secsh-filexfer-04.txt> also > specifies that scp is to use the NFSv4 ACL format, by the way.this document actually does a much better job of describing NFSv4 ACLs than the NFS document. So far POSIX ACLs seem to be what is being implemented at the system and filesystem level. In reality rsync doesn't need to deal with the same issues as NFS. Rsync's focus is not in projecting the access rights semantics to determine correct privileges but in preserving as much as possible the original ACL data. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
On Sun, Feb 23, 2003 at 01:13:56PM +0100, Andreas Gruenbacher wrote:> Hello, > > I am the guy behind the ext2/ext3 patches for Extended Attributes and ACLs, > and I've recently been asked about ACL support in rsync by Eric. Upon > investigating I found that you have an ACL patch against rsync-2.5.5 [1]. I > also found some other postings to rsync@lists.samba.org concerning rsync and > ACLs [3].I'm on 2.4 and the only ACL docs i have are from XFS which while consistant with POSIX i'm not using at the moment anyway. getfacl (1) - get file access control lists setfacl (1) - set file access control lists chacl (1) - change the access control list of a file or directory acl (5) - Access Control Lists Where can i find the system calls documented? They don't seem to be part of SUSv3 and Andreas Browne's latest manpages have no ACL related manpages. Thanks much. I'm a bit concerned that onone else has spoken up on this thread?
On Sun, Feb 23, 2003 at 01:13:56PM +0100, Andreas Gruenbacher wrote:> Hello, > > I am the guy behind the ext2/ext3 patches for Extended Attributes and ACLs, > and I've recently been asked about ACL support in rsync by Eric. Upon > investigating I found that you have an ACL patch against rsync-2.5.5 [1]. I > also found some other postings to rsync@lists.samba.org concerning rsync and > ACLs [3]. > > Are there any plans for finalizing an integrating that rsync ACL patch?i hope not seriously, though, the work i did was--we hoped--something to tide us over until something better came along. i have no illusions about or desire to see its incorporation into the official source tree i think that full-blown, general EA support is a laudable goal; my hope was just to preserve--to the naive extent possible by stealing a lot of other people's code and introducing as few of my own bugs as possible-- any additional permissions our folks are putting on their files so that our rsync-ed ``mirrors'' of their stuff aren't any less secure than their source images. in our case, simply preserving extended/discretionary ACLs between our Solaris (and, it's hoped some day soon now, Linux) systems was all we were looking for, and i'm content to live in ignorance of the more arcane^H^H^H general issue of EA support, since it's not anything that bears on security for our Solaris filesystems. (at least, i'm not aware of any other attributes that have significance security-wise; which is to say that i hope our users haven't figured out any yet) nevertheless, it seems to me also that the NFSv4 approach is a bit verbose and a bad enough fit for POSIX and Solaris/SysV ACLs, at least, that i would think twice about dumping my patch in favor of an implementation along those lines--at least if there weren't EAs that would benefit from a fully generalized, all singing, all dancing implementation. (but then again i am the sort who has trouble coping with the additional dimension- ality of access control beyond the POSIX triple, think NT permissions are a perplexing if exotic security nightmare, prefer single-letter options to --gnu-style options, prefer man to info, coffee to latte etc.). and yet i understand that somebody's going to figure out something cool to do with extended attributes that users are going to want to have rsync preserve for them and, in the future, it may be useless to transfer files at all with- out such things (just think, filesystem cookies and referral paths--gag). so maybe it's best to bite the bullet now, but i'm certainly not qualified to do it. then again, maybe you wait and see if NFSv4 and supporting file- systems force the issue, then you worry about it, just like seems to be the present rsync development strategy . . . (whence, bad patches like mine crop up to fill the void) sorry to ramble cynically, but thanks for showing an interest in clearing up the state of affairs and for your excellent work on ext2/ext3 EAs and ACLs and libattr and libacl
Access Control Lists (ACLs) and Extended Attributes (EA) are
an area i have seen for some time as something rsync will
need to address. I've put a tighter focus on this issue for
the past week or so and have reached a few conclusions.
1. ACL and EA OS support is growing but not really there yet.
Most of the UNIX players have POSIX ACL support.
I have no data on UNIX EA support but as far as i
can tell it is mostly absent.
Linux does not have consistent support yet. ACL
support is not part of mainstream production
kernels. If you want it you have to either apply
patches or run a development kernel. Some
distribution kernels are patched with ACL support
for one or more filesystems.
I know that XFS supports ACLs and EAs, later
versions of EXT2, EXT3 and JFS do also. I'm not
entirely sure of the status of rieserfs.
2. Utility support is almost completely missing.
Not only does rsync not support ACLs and EAs yet but
neither does cpio, tar (with the exception of star)
nor most of the backup utilities.
3. Use lags support dramatically.
The vast majority of sites do not use either ACLs
nor EAs even when they can. The lack of utility
support aggravates this.
Until support is ubiquitous across production-grade
OSs, filesystems and utilities the adoption of ACLs
and EAs will be delayed.
4. ACLs and EAs are a part of the future.
Users and admins are coming to linux and Unix with
the expectation of ACLs. While intelligent use of
group IDs can more simply deal with _almost_ all
permissions issues, and by being simpler tend to be
more secure, many will prefer the quick fix ACLs
provide.
Some of the new security models i've seen are going
to require both ACLs and EAs.
The potential value of EAs in GUI environments
should not be underestimated. Imagine file-manager
thumbnails and application icons that really are
attached to the file.
Even rsync may find good use for EAs. I can
envision optional storing of blocksums as an
extended attribute. A 64KB EA could support the
blocksums for a 127MB file using 4byte sums and 16KB
blocks.
Who knows what else the future might hold. Once
people expect EA support.
Those applicable utilities that fail to support ACLs
and EAs will become irrelevant.
So while demand is currently low i believe that rsync will
need to support ACLs and EAs in the near future or it will
become little more than a limited download tool like ftp.
When that will be is an unknown but i think the release of
the Linux 2.6 kernel will be a major factor. This means
that the widespread use of ACLs on the filesystems may well
begin in as little as one year if 2.6 comes out on schedule.
Where does that leave us? What am i going to do about it?
Based on the level of comments on this and other threads
there is little demand presently for ACL or EA support in
the rsync user or developer base. There are a few people
for whom this is a production issue today and a few like
myself who see it as a future issue that should be
anticipated.
I do not use ACLs or EAs currently. I believe that support
for them should be added sooner rather than later and i
think that how that support is implemented is very
important. I however do not use them at this time and
unless someone will pay me to do so i'm not going to set up
a lab or start slinging code to add that support. Sorry but
this just isn't my itch yet. While i care how it is done i
don't care enough yet to do it myself and build the test
apparatus.
The remainder of this missive shall be a bit more technical.
Since i do care how they are implemented and have some
informed ideas in that direction it seems good to me to
relate those ideas. If you like design documents think of
this as a start on a high-level one. I don't generally care
for design documents but the design should be discussed
before such significant code is generated. If you aren't
interested in such discussion, or don't care what i have to
say please move on instead of complaining. I've broken this
down into several sections.
If you have comments to make please address one issue per
followup so they form separate sub-threads and only quote the
relevant text. Unless there are comments i expect this will
be the last i'll discuss this issue for a while.
-- How have OSs implemented Access Control Lists --
In UNIX ACLs are implemented in various, sometimes
non-standard, ways. For the most part it looks like they
are largely compatible with POSIX ACLs.
In Linux ACLs are implemented as an EA. In order to
support ACLs on Linux and non-linux platforms we have
to treat them separately from the EAs.
Although NTFS supports ACLs the cygwin environment does not
_yet_ reflect that.*** NTFS ACLs do not quite translate to
and from POSIX. The semantic differences mean that
information is lost translating each direction.
Netware also has a form of ACLs similar to NTFS.
The order of POSIX ACLs does not affect them. The only
consistent compact expression of them is textual in the form:
[d:]type:id:perms
type is one of u,g,o,m
id is either the name or ascii id number.
perms is a the symbolic rwx string as shown in ls
As near as i can tell the acl_t structure used by the
libraries is an opaque data type and working on it directly
is likely to break on some platforms and may be subject to
change in future.
-- How have OSs implemented Extended Attributes --
EAs seem to be unsupported by most of the UNIX platforms.
Pipe up now if you know of support for them on any
mainstream UNIX.
Limited EAs are supported on the MacOS and OS-X in the form
of the resource fork. I have a vague recollection that NTFS
also has some form of EAs.
In Linux EAs have been growing in importance. In addition
to ACLs other security features are being implemented with
them.
Extended attributes are simple name, value pairs. Names are
partitioned into namespaces ("user." and "system." at
present) are null terminated fully qualified text up to 256
bytes long. Values are BLOBS up to 64KB long. The count
and total size of a file's EAs is implementation dependant
(varies from 1KB to unlimited). The order on retrieval of
them is indeterminate.
-- What should rsync support --
Extended Attributes and POSIX ACLs.
Like it or not rsync is POSIX semantics oriented. Further,
when running on the one major non-POSIX platform it does so
in a POSIX emulating environment (cygwin) so it doesn't even
have access to non-POSIX file semantics. Full
cross-platform functionality isn't really rsync's bailiwick.
Perhaps a future tool that supercedes rsync might
interoperate with native, non-POSIX semantics but i don't
see rsync going there.
If all we supported were Linux we could simply provide
Extended attributes and the ACLs would be automatically
included. However, we should support the other POSIX
(Unix mostly) platforms.
It would be worthwhile to have some limited support for
non-POSIX ACLs in some cases. I'll discuss that in the
"how" section.
-- How should rsync support Access Control Lists --
As i've already said the focus should be on POSIX ACLs.
As much as i am inclined to deal with ACLs as binary data
short type/flags
short perms
int id
or something similar, the library routines treat the acl_t
structure as an opaque object and don't really support that.
So we would wind up converting the acl_t to the text form
and then to the binary and visa-versa. I don't think the
compaction (10-15 chars/ACE -> 8 or 12 bytes/ACE) is worth
the extra computation. If the acl_t structure turns out to
be less opaque i'd be more than happy to revisit the binary
format.
Given the potential size of an ACL we don't want to transmit
the ACL for every file that has an extended one. An
extended ACL is one with entries not in the standard POSIX
UGO permissions mask. What we should do instead is assemble
each ACL into a consistent, sorted form that lends itself to
checksum comparison. A single block checksum for a file's
ACL would allow us to identify those files who's ACLs had
changed. This way we would only transmit an ACL if it had
changed.
Limited support for non-POSIX ACLs could be provided.
During the protocol discovery phase** ACL capabilities should
be determined. If two like systems (windows<->windows for
example) are communicating they could use their native ACL
format instead of POSIX. Otherwise non-POSIX systems would
be expected to convert their ACL format to and from our
chosen POSIX representation.
-- How should rsync support Extended Attributes --
Extended attributes can be quite large. The EAs of a file
should be built into a single contiguous object. The object
would be built with the extended attributes sorted by name.
This EA object would be compared and transmitted just like a
regular file using the rsync algorithm. Possible
alternatives would be to generate a separate checksums for
each EA or to somehow start a new blocksum with each EA to
take advantage of the fact that each EA is a discrete entity
requiring a separate syscall to retrieve, remove, set or
change.
Where ACLs are being supported by EAs the ACL EA(s) would
not be included in the EA synchronization. For this reason
it will be necessary to write the ACLs after the EAs. So
the order of ops on a file update would be file-data, EA,
ACL, chown, chgrp, chmod, and finally mtime.
-- Unresolved issues --
A drawback to checksum approach to EA and ACL objects is that
it would often be necessary to create the objects twice on
the receiver, once to generate the blocksums and a second
time to merge any changes.
Also at issue is whether and what timestamps are modified by
changes to ACLs and EAs. Such updates should change ctime
but not necessarily mtime. If that is the case it may be
necessary to generate total checksum and size on the sender
during the file-scan phase or things could get quite ugly.
Such a early sender checksum would mean the objects might
have to be generated twice on the sender as well.
The regeneration may be partially avoidable by allowing a
limited amount to be attached to the flist or otherwise
cached. The ACLs in particular may be small enough to keep.
Because ACLs and EA lists will often be the same on many
files a hash to the ACLs and EAs objects could be maintained
and duplicates could be identified reducing inter-phase
storage requirements.
--------
** Some of my comments may conflict with the rsync
protocol. This is because i have not so far needed to get
into the protocol itself and i have yet to see any decent
documentation on the protocol. Tridge's algorithm is well
enough documented but the protocol implementing it is not.
*** I don't use windows so my cygwin and NTFS knowledge is
limited.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: jw@pegasys.ws
Remember Cernan and Schmitt
How many bytes per file on average do you think EAs/ACLs will take up? -- Ben Escoto -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 229 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20030309/6d04a3e7/attachment.bin