samba - Nov 2021 - permissions, and maybe a violation of the least surprise principle

>
>
> >> What happens to these permissions if I migrate the data (say, via
> >> rsync) to another server? Seems like all the Windows ACLs will be
lost
> >> unless I transfer the relevant database as well.
> >
> > So long as rsync transfers EA's then all will work.
>
>
>
> That's kind of the question, I guess. For ext4 the xattrs are stored in
> a separate data block referenced from inodes via inode.i_file_acl*
> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
>
> I suppose rsync could/would be smart enough to follow this pointer to
> the xattrs.
>
>
Yes, rsync will copy/synchronize extended attributes and POSIX ACLs.
However, keep in mind that POSIX ACLs and EAs are different things, and so
copying over one does not necessarily mean the other is copied over - if
you've applied Windows-type permissions that are stored in xattrs, and you
successfully copy over files with the EAs intact, the underlying POSIX ACLs
will not be copied/updated unless you make sure that you specifically copy
those over. The linkage between the POSIX ACLs and the EAs is done
completely by Samba and not by any Linux-level linkage between the two.

> This got me interested, so I'm running some tests.  For starters, if
you
> `scp -p` files from an XFS filesystem to ext4, POSIX ALC's are not
> preserved:
>
>
> root at kraken:/EM/21sep30a# getfacl 20210930_YD04_68_004_0.0_14.08.45.eer
> # file: 20210930_YD04_68_004_0.0_14.08.45.eer
> # owner: jonesimages
> # group: jonesimages
> user::rwx
> user:abril:r-x
> user:jones:rwx
> user:cryosparc_user:r-x
> user:afb938:r-x
> group::r-x
> group:cns-joneslabusers:r-x
> mask::rwx
> other::---
>
>
> root at kraken:/EM/21sep30a# scp -p  20210930_YD04_68_004_0.0_14.08.45.eer
> pgoetz at frog.my-domain.com:/home/pgoetz/temp
>
> [pgoetz at frog temp]$ getfacl 20210930_YD04_68_004_0.0_14.08.45.eer
> # file: 20210930_YD04_68_004_0.0_14.08.45.eer
> # owner: pgoetz
> # group: pgoetz
> user::rwx
> group::rwx
> other::---
>
>
> Looks like rsync'ing from XFS to ZFS doesn't preserve POSIX
extended
> ACLs, either. Damn.  I need to find an ext4 system I'm using POSIX ACLs
> on; maybe ext4 performs better in this regard.
>
> Has anyone tested to see if copying files from ext4 to ext4 preserves
> POSIX extended ACLs? Under some conditions (say same UID's on each
system)?
>
>
What flags are you using with rsync? There's a flag specifically for ACLs
(-A), and one for xattrs (-X) so you have to make sure to specify those. I
generally have very good success with copying ACL across filesystems,
though I cannot say for sure that I've tried from XFS to ZFS, etc. My rsync
command generally looks like this:

rsync -aADXHv <source> <destination>

Also, one thing I've found with rsync regarding ID mapping is this:
* If the user account exists on the destination system, ownership will be
properly changed so that the username matches. So, if account "nick"
has ID
1001 on system A, and account "nick" has ID 2001 on system B, if I
rsync
from system A to B, the account "nick" will own all of the files on
System
B that the account owned on system A, regardless of the fact that the
numeric ID does not match.
* If the user account does not exist on system B, or files are owned by a
UID that has no user on system A, then the ownership will just be copied
over numerically. So, if account "nick" exists on system A with ID
1001,
and there is no account "nick" on system B, then the files will be
copied
over with owner 1001. Also, if files on system A have an owner ID of 1002
with no account, those will just be copied over with the same numeric ID to
system B.

> If I have to restore a 1PB filesystem from backup and need to
> reconstruct all the ACLs by hand it's going to be a sad, sad day. 
Week,
> I meant; possibly month. Actually infinite, since I'll probably quit
and
> will open a bakery instead. Bread doesn't have ACLs.
>
>
I like bread. With butter. I will visit your bakery.

-Nick

On Tue, 2021-11-09 at 08:02 -0500, Nick Couchman via samba
wrote:
> > 
> 
> > 
> > 
> > 
> > Looks like rsync'ing from XFS to ZFS doesn't preserve POSIX
> > extended
> > ACLs, either. Damn. 
Could this be on freebsd ? 
ZFS is known to be a problem on Samba, because it uses NFSv4 ACLs
> >  I need to find an ext4 system I'm using POSIX ACLs
> > on; maybe ext4 performs better in this regard.
It at least uses ACL's
> > 
> > Has anyone tested to see if copying files from ext4 to ext4
> > preserves
> > POSIX extended ACLs? Under some conditions (say same UID's on each
> > system)?
No it doesn't, not unless the users has the same UID on both systems.
> > 
> > 
> What flags are you using with rsync? There's a flag specifically for
> ACLs
> (-A), and one for xattrs (-X) so you have to make sure to specify
> those. I
> generally have very good success with copying ACL across filesystems,
> though I cannot say for sure that I've tried from XFS to ZFS, etc. My
> rsync
> command generally looks like this:
> 
> rsync -aADXHv <source> <destination>
> 
> Also, one thing I've found with rsync regarding ID mapping is this:
> * If the user account exists on the destination system, ownership
> will be
> properly changed so that the username matches. So, if account
"nick"
> has ID
> 1001 on system A, and account "nick" has ID 2001 on system B, if
I
> rsync
> from system A to B, the account "nick" will own all of the files
on
> System
> B that the account owned on system A, regardless of the fact that the
> numeric ID does not match.
So, how do explain the fact that you have to sync idmap.ldb between
DC's before the directories and files are owned by the correct user and
groups ?
 
Rowland

On 11/9/21 07:02, Nick Couchman wrote:
> 
>      >> What happens to these permissions if I migrate the data (say,
via
>      >> rsync) to another server? Seems like all the Windows ACLs
will
>     be lost
>      >> unless I transfer the relevant database as well.
>      >
>      > So long as rsync transfers EA's then all will work.
> 
> 
> 
>     That's kind of the question, I guess. For ext4 the xattrs are
stored in
>     a separate data block referenced from inodes via inode.i_file_acl*
>    
https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
>    
<https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes>
> 
>     I suppose rsync could/would be smart enough to follow this pointer to
>     the xattrs.
> 
> 
> Yes, rsync will copy/synchronize extended attributes and POSIX ACLs. 
> However, keep in mind that POSIX ACLs and EAs are different things, and 
> so copying over one does not necessarily mean the other is copied over - 
> if you've applied Windows-type permissions that are stored in xattrs, 
> and you successfully copy over files with the EAs intact, the underlying 
> POSIX ACLs will not be copied/updated unless you make sure that you 
> specifically copy those over. The linkage between the POSIX ACLs and the 
> EAs is done completely by Samba and not by any Linux-level linkage 
> between the two.
> 
>     This got me interested, so I'm running some tests.? For starters,
if
>     you
>     `scp -p` files from an XFS filesystem to ext4, POSIX ALC's are not
>     preserved:
> 
> 
>     root at kraken:/EM/21sep30a# getfacl
20210930_YD04_68_004_0.0_14.08.45.eer
>     # file: 20210930_YD04_68_004_0.0_14.08.45.eer
>     # owner: jonesimages
>     # group: jonesimages
>     user::rwx
>     user:abril:r-x
>     user:jones:rwx
>     user:cryosparc_user:r-x
>     user:afb938:r-x
>     group::r-x
>     group:cns-joneslabusers:r-x
>     mask::rwx
>     other::---
> 
> 
>     root at kraken:/EM/21sep30a# scp -p?
20210930_YD04_68_004_0.0_14.08.45.eer
>     pgoetz at frog.my-domain.com:/home/pgoetz/temp
> 
>     [pgoetz at frog temp]$ getfacl 20210930_YD04_68_004_0.0_14.08.45.eer
>     # file: 20210930_YD04_68_004_0.0_14.08.45.eer
>     # owner: pgoetz
>     # group: pgoetz
>     user::rwx
>     group::rwx
>     other::---
> 
> 
>     Looks like rsync'ing from XFS to ZFS doesn't preserve POSIX
extended
>     ACLs, either. Damn.? I need to find an ext4 system I'm using POSIX
ACLs
>     on; maybe ext4 performs better in this regard.
> 
>     Has anyone tested to see if copying files from ext4 to ext4 preserves
>     POSIX extended ACLs? Under some conditions (say same UID's on each
>     system)?
> 
> 
> What flags are you using with rsync? There's a flag specifically for 
> ACLs (-A), and one for xattrs (-X) so you have to make sure to specify 
> those. I generally have very good success with copying ACL across 
> filesystems, though I cannot say for sure that I've tried from XFS to 
> ZFS, etc. My rsync command generally looks like this:
> 
> rsync -aADXHv <source> <destination>

That's it! I was just assuming -a covered everything, but the rsync man 
page indicates otherwise.  In particular, I was not using -A or -X.

Wow, you just averted a potential disaster I wasn't even aware of due to 
having too few brain cells. On the other hand, you saved me from the 
happy life of a baker when my PB filesystem crashes and has to be 
restored from backup, so pros and cons.

Question: why are you backing up devices? Does this only apply to root 
filesystem backups?
> 
> Also, one thing I've found with rsync regarding ID mapping is this:
> * If the user account exists on the destination system, ownership will 
> be properly changed so that the username matches. So, if account
"nick"
> has ID 1001 on system A, and account "nick" has ID 2001 on system
B, if
> I rsync from system A to B, the account "nick" will own all of
the files
> on System B that the account owned on system A, regardless of the fact 
> that the numeric ID does not match.
Wait, what?! Why would rsync care or even know about the uid <-> 
username mapping? This is a feature of the directory service/files and 
outside the filesystem entirely AFAIK.

> * If the user account does not exist on system B, or files are owned by 
> a UID that has no user on system A, then the ownership will just be 
> copied over numerically. So, if account "nick" exists on system A
with
> ID 1001, and there is no account "nick" on system B, then the
files will
> be copied over with owner 1001. Also, if files on system A have an owner 
> ID of 1002 with no account, those will just be copied over with the same 
> numeric ID to system B.
> 
>     If I have to restore a 1PB filesystem from backup and need to
>     reconstruct all the ACLs by hand it's going to be a sad, sad day. 
>     Week,
>     I meant; possibly month. Actually infinite, since I'll probably
quit
>     and
>     will open a bakery instead. Bread doesn't have ACLs.
> 
> 
> I like bread. With butter. I will visit your bakery.
> 
> -Nick