samba - Apr 2018 - Is it possible to clone an NT ACL from one file or dir to a totally different file or dir ?

On Sat, 2018-03-24 at 20:38 -0400, Ken McDonald via samba
wrote:
> 
> > By default, step 4 takes forever to run on large datasets because it 
> > uses named entities. My ultimate plan was to use the numeric 
> > user/group id's in that step instead of named ones so the Winbind
cost
> > is not incurred. Seems for the whole process, the calls to Winbind to 
> > resolve the named entity to it's numeric ID are the reason for the
> > slowdown. That's why, even when using the normal Windows security
tab
> > or samba-tool, it takes days to update large datasets. I'm
exploring
> > options around that issue.
If this is on the AD DC, then I fixed part of the slowness here a
couple of week ago with 

commit d418d0ca33afb41a793a2fff19ca68871aa5e9ef
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Wed Mar 21 20:43:10 2018 +1300

    winbindd: Add a cache of the samr and lsa handles for the passdb
domain

Otherwise, I'm not sure folks have looked into this terribly much, you
are already working at a pretty low level.

I hope this helps a little.

Andrew Bartlett
-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba

On 04/05/2018 08:32 PM, Andrew Bartlett wrote:
> On Sat, 2018-03-24 at 20:38 -0400, Ken McDonald via samba wrote:
>>> By default, step 4 takes forever to run on large datasets because
it
>>> uses named entities. My ultimate plan was to use the numeric
>>> user/group id's in that step instead of named ones so the
Winbind cost
>>> is not incurred. Seems for the whole process, the calls to Winbind
to
>>> resolve the named entity to it's numeric ID are the reason for
the
>>> slowdown. That's why, even when using the normal Windows
security tab
>>> or samba-tool, it takes days to update large datasets. I'm
exploring
>>> options around that issue.
> If this is on the AD DC, then I fixed part of the slowness here a
> couple of week ago with
>
> commit d418d0ca33afb41a793a2fff19ca68871aa5e9ef
> Author: Andrew Bartlett <abartlet at samba.org>
> Date:   Wed Mar 21 20:43:10 2018 +1300
>
>      winbindd: Add a cache of the samr and lsa handles for the passdb
> domain
>
> Otherwise, I'm not sure folks have looked into this terribly much, you
> are already working at a pretty low level.
>
> I hope this helps a little.
>
> Andrew Bartlett
I have found a working solution using parallel processing on multiple 
cores that greatly reduces the time required to clone permissions. On a 
user directory with > 1TB data of 100,000s of dirs/files with all kinds 
of exotic naming styles, the process in the script below completes in 
about 60 minutes instead of the 5 days it took with either samba-tool or 
Windows Explorer.

This kind of performance was needed to move a bunch of user home 
directories over to the Samba file server, and I also have some cron 
jobs that periodically run to ensure a few shared file directories 
always have a standard set of permissions. In prior email list replies, 
I mentioned the perms were not properly cloning because the cloned copy 
had "extras" in them. This turned was because the top-level POSIX user
&
group were not the same in the cloned target. While I was correctly 
cloning POSIX extended ACL & attributes from the source, I did not 
realize that would not include the top-level permissions entries. Once 
that setting was added to the process, cloning Samba/POSIX permissions & 
attributes worked perfectly and ended up with exactly the same 
permission structure as would have been created by using Windows Explorer.

While I haven't looked at Andrew's code update helping Samba cache 
winbind lookups, I will mention that the process in the script below 
does not make use of any Samba/Winbind username/group lookup calls 
UNLESS the Windows ACL dir/file ownership is being set with a Windows 
username or group. For my case, all the dirs/files are owned by 
Administrator (root) and user access rights are delegated through 
Windows ACL entries, so no winbind lookups were needed when the script 
runs. If a Windows user/group account is configured for dir/file 
ownership, there will be slight delays during script execution, but 
since it's only for dir/file ownership, not actual Windows ACL perms, 
the delay is relatively tiny.

This process is extremely fast compared to samba-tool and I hope it's 
helpful to anyone that finds this thread. If you need to configure a 
standard suite of Windows ACL permissions on a large amount of 
directories & files this is currently the fastest way I've found to do
it.

It's possible "sneak in" pure POSIX ACL entries manually using 
Linux-side only users/groups and the full cloning still works. In those 
case, once the POSIX ACL entry is created outside of Samba/Windows, you 
have to go back into the Windows Explorer Security tab for the perms 
template source dir/file and clean up some unneeded entries like 
"Everyone" and "CREATER OWNER." In the end, I was able to
clone a source
template dir/file with both POSIX/Windows ACL entries from both 
Linux-side & Samba-winbind sourced usernames/groups!

I'd appreciate any helpful comments on making my script more efficient 
or better simplified. The one portion I wished could have been compacted 
is the shell'ing-out of xargs to the separate tiny helper script 
"setperm.sh". I couldn't get that helper snippet to run as a shell
function while still allowing xargs to call it, especially because of 
the variables being passed. Also, as dir/file names can be complex and 
contain all kinds of crazy special characters, there were a lot of 
challenges making the script and command/program calls handle these 
issues without running into escape'ing problems. I think all of these 
leaky problems have been corrected but I'd certainly appreciate any 
feedback there.

By using deep extended attribute cloning, only getfattr is needed 
because it includes the ACL entries normally configured by using 
getfacl. This helped ...a lot... in reducing winbind username/group 
lookup calls. The key line that clones POSIX/Windows ACL entries is this.

getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" | setfattr 
--restore=-

THE FULL SCRIPTS FOLLOW:

####################################### cloneperms.sh 
#######################################
#!/bin/bash
# Clone permissions & attributes from a template source to a target 
destination,
# possibly recursing all subdirectories & files. Also supports parallelism.
#
# USAGE: cloneperms.sh [-r] [-p numcores] template target
#
# -r        : recurse all subdirectories. pointless if template & target 
are files
# -p        : number of cores for parallel processing, defaults to 1. 
pointless if template & target are files
# template  : source directory or file with permissions to clone
# target    : destination directory or file for cloning perms to
#
# version 1.0

USAGE="Usage: $0 [-r] [-p numcores] template target"

# determine parameter type
test_type()
{
     TYPE_TEST=$1
     if   [ -d "${TYPE_TEST}" ]
     then return 1; # directory
     elif [ -f "${TYPE_TEST}" ]
     then return 2; # file
     else
         echo "${TYPE_TEST} is not found!"; echo $USAGE; exit 1;
     fi
}

# check for options
CORES=1
RECURSE=0
while getopts rp: o
     do    case "$o" in
         r)    RECURSE=1;;
         p)    CORES=$OPTARG
                 if [ -z "${CORES##*[!0-9]*}" ] || [ $CORES -le 0 ]
                 then
                     echo "numcores must be number greater than 0";
echo
$USAGE; exit 1;
                 fi;;
         [?])    echo $USAGE; exit 1;;
     esac
done
shift $(($OPTIND - 1))

# enough arguments?
if [ "$#" -ne 2 ]
then
     echo "missing template or target"; echo $USAGE; exit 1;
fi

TEMPLATE=$1
TARGET=$2

# no self targets
if [ $TEMPLATE == $TARGET ]
then
     echo "template & target cannot be the same!"; echo $USAGE;
exit 1;
fi

# test template & target types
test_type $TEMPLATE
TEMPLATE_TYPE=$?
test_type $TARGET
TARGET_TYPE=$?

# ensure template, target, recurse, and cores combinations are valid
if [ $TEMPLATE_TYPE == 1 ]
then
     FIND_FLAG="d";
     if [ $TARGET_TYPE == 2 ]
     then
         echo "template cannot be directory and target a file!"; echo 
$USAGE; exit 1;
     fi
else
     FIND_FLAG="f";
     if [ $TARGET_TYPE == 1 ]
     then
         if [ $RECURSE == 0 ]
         then
             echo "template cannot be a file and target a directory 
without recurse option"; echo $USAGE; exit 1;
         fi
     else
         if [ $CORES -gt 1 ]
         then
             echo "template & target are files, ignoring numcores
option";
             CORES=1;
         fi
         if [ $RECURSE == 1 ]
         then
             echo "template & target are files, ignoring recurse
option";
             RECURSE=0;
fi fi fi

TEMPLATE_OWN=$(stat -c %U $TEMPLATE)      # get the template top-level 
POSIX owner
TEMPLATE_GROUP=$(stat -c %G $TEMPLATE)    # get the template top-level 
POSIX group
TEMPLATE_PERMS=$(stat -c %a $TEMPLATE)    # get the template top-level 
POSIX permissions
current_dir=$(dirname $(readlink -f $0))  # script can run from another 
directory
if [ $RECURSE == 1 ]
     then
         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES 
chown $TEMPLATE_OWN;
         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES 
chgrp $TEMPLATE_GROUP;
         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES 
chmod $TEMPLATE_PERMS;
         # get the template POSIX extended attributes, which include 
extended ACL
         find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES 
-I {} $current_dir/setperm.sh $TEMPLATE {};
     else
         chown $TEMPLATE_OWN   $TARGET;
         chgrp $TEMPLATE_GROUP $TARGET;
         chmod $TEMPLATE_PERMS $TARGET;
         # get the template POSIX extended attributes, which include 
extended ACL
         $current_dir/setperm.sh $TEMPLATE $TARGET;
fi
######################################################################################


####################################### setperm.sh 
#######################################
#!/bin/bash
# Sets a single permissions & attributes entry on a target based on a 
template source example
#
# There's no parameter checking in this script, so don't run this script
directly.
# It's meant to be run from cloneperms.sh.
#
# USAGE: setperm.sh template target
#
# template  : source directory or file with permissions to clone
# target    : destination directory or file for cloning perms to
#
# version 1.0

ENCODED=$(printf '%s\n' "$2" | sed 's|\\|\\\\134|g') #
setfattr syntax
for target name containing forward slashes
getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" | setfattr 
--restore=-
######################################################################################


I also have a script for recursively removing all extended ACL & 
attribute entries if anyone is interested. Strangely there is a way to 
remove the ACL "setfacl -R -b" recursively but I found no way to 
similarly remove ALL attributes. I made a script to do it regardless of 
what the attributes are named...

TLDR?

-Ken

On 07.04.2018 at 22:02 Ken McDonald wrote:
> By using deep extended attribute cloning, only getfattr is needed 
> because it includes the ACL entries normally configured by using 
> getfacl. This helped ...a lot... in reducing winbind username/group 
> lookup calls. The key line that clones POSIX/Windows ACL entries is this.
> 
> getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" |
setfattr
> --restore=-
I found this to be insufficient, and are using these two commands:

# transfer Unix-ACLs (and owner, group, and mode-bits):
cd $src; getfacl -R -n . | (cd $dst; setfacl --restore=-)

# transfer ATTRs (including Windows-ACLs, but not owner, group, or 
mode-bits):
cd $src; getfattr -R -d -m - . | (cd $dst; setfattr --restore=-)

If I remember correct, the Unix owner, group and mode bits must also be 
copied, otherwise the Windows-ACLs will be ignored.

This requires traversing all directories twice, but the second run is 
usually a lot faster because caching, independend on the order in which 
the commands are run.