samba - Mar 2019 - classicupgrade, net rpc rights grant NT_STATUS_IO_TIMEOUT and NT_STATUS_INTERNAL_ERROR

Dear all,

we are transitioning from an openldap / MIT KDC setup to a samba4 AD. I
am doing this by setting up a samba NT4 domain, populating it from LDAP
and sticking in the password hashes which I automatically extract from
the MIT KDC arc4-hmac keys. Then I run the classicupgrade. I do this
whole thing from cron in a script once a day to be able to slowly
migrate services. The MIT / openldap and samba4 AD servers are on
different machines. My script (based on LPHvB's instructions) sets
privileges in the following way:

[..]

systemctl restart bind9 ntp samba-ad-dc
sleep 5
SAMBA_DC_ADMIN_GROUP_CHOICE="BUILTIN\Administrators"
PRIVS="SeDiskOperatorPrivilege SeTakeOwnershipPrivilege \
SeBackupPrivilege SeRestorePrivilege SeRemoteShutdownPrivilege \
SePrintOperatorPrivilege SeAddUsersPrivilege SeDiskOperatorPrivilege \
SeSecurityPrivilege SeSystemtimePrivilege SeShutdownPrivilege \
SeDebugPrivilege SeSystemEnvironmentPrivilege SeSystemProfilePrivilege \
SeProfileSingleProcessPrivilege SeIncreaseBasePriorityPrivilege \
SeLoadDriverPrivilege SeCreatePagefilePrivilege \
SeIncreaseQuotaPrivilege SeChangeNotifyPrivilege SeUndockPrivilege \
SeManageVolumePrivilege SeImpersonatePrivilege SeCreateGlobalPrivilege \
SeEnableDelegationPrivilege"

samba-tool user setpassword Administrator \
  --newpassword="$SAMBA_NT_ADMIN_PASS"
echo "$SAMBA_NT_ADMIN_PASS" | kinit Administrator
for priv in $PRIVS ; do
  while ! net rpc rights grant "${SAMBA_DC_ADMIN_GROUP_CHOICE}" $priv
\
            -U "Administrator%$SAMBA_NT_ADMIN_PASS" ; do
    echo "Failed to grant $priv ... Retrying ..."
    sleep 10
  done
done

Upon running this, I often get NT_STATUS_INTERNAL_ERROR or
NT_STATUS_IO_TIMEOUT:

Changed password OK
Password for Administrator at XXXXXXXXXXXXXXXXX:
Could not connect to server 127.0.0.1
Connection failed: NT_STATUS_IO_TIMEOUT
Failed to grant SeDiskOperatorPrivilege ... Retrying ...
Could not connect to server 127.0.0.1
Connection failed: NT_STATUS_IO_TIMEOUT
Failed to grant SeDiskOperatorPrivilege ... Retrying ...
Could not connect to server 127.0.0.1
Connection failed: NT_STATUS_INTERNAL_ERROR
Failed to grant SeDiskOperatorPrivilege ... Retrying ...
Successfully granted rights.
Successfully granted rights.
Successfully granted rights.

Why would that happen? I can put in as much wait time as I want after
the initial service restart, and it still happens. I obviously work
around it by repeating until it proceeds, but I do not understand why
this is necessary. Any hints would be appreciated... Thanks,

Christian

On Mon, 2019-03-11 at 07:16 +0100, Christian via samba
wrote:
> Dear all,
> 
> we are transitioning from an openldap / MIT KDC setup to a samba4 AD. I
> am doing this by setting up a samba NT4 domain, populating it from LDAP
> and sticking in the password hashes which I automatically extract from
> the MIT KDC arc4-hmac keys. Then I run the classicupgrade. I do this
> whole thing from cron in a script once a day to be able to slowly
> migrate services. The MIT / openldap and samba4 AD servers are on
> different machines. My script (based on LPHvB's instructions) sets
> privileges in the following way:
> 
> [..]
> 
> systemctl restart bind9 ntp samba-ad-dc
> sleep 5
> SAMBA_DC_ADMIN_GROUP_CHOICE="BUILTIN\Administrators"
> PRIVS="SeDiskOperatorPrivilege SeTakeOwnershipPrivilege \
> SeBackupPrivilege SeRestorePrivilege SeRemoteShutdownPrivilege \
> SePrintOperatorPrivilege SeAddUsersPrivilege SeDiskOperatorPrivilege \
> SeSecurityPrivilege SeSystemtimePrivilege SeShutdownPrivilege \
> SeDebugPrivilege SeSystemEnvironmentPrivilege SeSystemProfilePrivilege \
> SeProfileSingleProcessPrivilege SeIncreaseBasePriorityPrivilege \
> SeLoadDriverPrivilege SeCreatePagefilePrivilege \
> SeIncreaseQuotaPrivilege SeChangeNotifyPrivilege SeUndockPrivilege \
> SeManageVolumePrivilege SeImpersonatePrivilege SeCreateGlobalPrivilege \
> SeEnableDelegationPrivilege"
> 
> samba-tool user setpassword Administrator \
>   --newpassword="$SAMBA_NT_ADMIN_PASS"
> echo "$SAMBA_NT_ADMIN_PASS" | kinit Administrator
> for priv in $PRIVS ; do
>   while ! net rpc rights grant "${SAMBA_DC_ADMIN_GROUP_CHOICE}"
$priv \
>             -U "Administrator%$SAMBA_NT_ADMIN_PASS" ; do
>     echo "Failed to grant $priv ... Retrying ..."
>     sleep 10
>   done
> done
> 
> Upon running this, I often get NT_STATUS_INTERNAL_ERROR or
> NT_STATUS_IO_TIMEOUT:
> 
> Changed password OK
> Password for Administrator at XXXXXXXXXXXXXXXXX:
> Could not connect to server 127.0.0.1
> Connection failed: NT_STATUS_IO_TIMEOUT
> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
> Could not connect to server 127.0.0.1
> Connection failed: NT_STATUS_IO_TIMEOUT
> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
> Could not connect to server 127.0.0.1
> Connection failed: NT_STATUS_INTERNAL_ERROR
> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
> Successfully granted rights.
> Successfully granted rights.
> Successfully granted rights.
> 
> Why would that happen? I can put in as much wait time as I want after
> the initial service restart, and it still happens. I obviously work
> around it by repeating until it proceeds, but I do not understand why
> this is necessary. Any hints would be appreciated... Thanks,
That certainly is strange, and perhaps the server-side logs or a
network trace might provide a clue.  I would however note that
privileges are not very important in AD because they are not
replicated.  Instead things tend to be provided by extended rights on
magic AD objects.

Perhaps avoid re-generating the whole AD and just migrate the changed
passwords?  A bit more fiddly but then things like GUIDs won't keepchanging.

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

On Mon, 11 Mar 2019 07:16:30 +0100
Christian via samba <samba at lists.samba.org> wrote:
> Dear all,
> 
> we are transitioning from an openldap / MIT KDC setup to a samba4 AD.
> I am doing this by setting up a samba NT4 domain, populating it from
> LDAP and sticking in the password hashes which I automatically
> extract from the MIT KDC arc4-hmac keys. Then I run the
> classicupgrade. I do this whole thing from cron in a script once a
> day to be able to slowly migrate services. The MIT / openldap and
> samba4 AD servers are on different machines. My script (based on
> LPHvB's instructions) sets privileges in the following way:
> 
> [..]
> 
> systemctl restart bind9 ntp samba-ad-dc
> sleep 5
> SAMBA_DC_ADMIN_GROUP_CHOICE="BUILTIN\Administrators"
> PRIVS="SeDiskOperatorPrivilege SeTakeOwnershipPrivilege \
> SeBackupPrivilege SeRestorePrivilege SeRemoteShutdownPrivilege \
> SePrintOperatorPrivilege SeAddUsersPrivilege SeDiskOperatorPrivilege \
> SeSecurityPrivilege SeSystemtimePrivilege SeShutdownPrivilege \
> SeDebugPrivilege SeSystemEnvironmentPrivilege
> SeSystemProfilePrivilege \ SeProfileSingleProcessPrivilege
> SeIncreaseBasePriorityPrivilege \ SeLoadDriverPrivilege
> SeCreatePagefilePrivilege \ SeIncreaseQuotaPrivilege
> SeChangeNotifyPrivilege SeUndockPrivilege \ SeManageVolumePrivilege
> SeImpersonatePrivilege SeCreateGlobalPrivilege \
> SeEnableDelegationPrivilege"
> 
> samba-tool user setpassword Administrator \
>   --newpassword="$SAMBA_NT_ADMIN_PASS"
> echo "$SAMBA_NT_ADMIN_PASS" | kinit Administrator
> for priv in $PRIVS ; do
>   while ! net rpc rights grant "${SAMBA_DC_ADMIN_GROUP_CHOICE}"
$priv
> \ -U "Administrator%$SAMBA_NT_ADMIN_PASS" ; do
>     echo "Failed to grant $priv ... Retrying ..."
>     sleep 10
>   done
> done
> 
> Upon running this, I often get NT_STATUS_INTERNAL_ERROR or
> NT_STATUS_IO_TIMEOUT:
> 
> Changed password OK
> Password for Administrator at XXXXXXXXXXXXXXXXX:
> Could not connect to server 127.0.0.1
> Connection failed: NT_STATUS_IO_TIMEOUT
> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
> Could not connect to server 127.0.0.1
> Connection failed: NT_STATUS_IO_TIMEOUT
> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
> Could not connect to server 127.0.0.1
> Connection failed: NT_STATUS_INTERNAL_ERROR
> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
> Successfully granted rights.
> Successfully granted rights.
> Successfully granted rights.
> 
> Why would that happen? I can put in as much wait time as I want after
> the initial service restart, and it still happens. I obviously work
> around it by repeating until it proceeds, but I do not understand why
> this is necessary. Any hints would be appreciated... Thanks,
> 
> Christian
> 
> 
This looks like a DNS problem, what do you have in /etc/resolv.conf ?
Is Bind9 setting up correctly and running ?
What OS ?

You do not need the 'kinit', you are not doing the changes via kerberos.

Rowland

Am 11.03.2019 um 09:24 schrieb Rowland Penny via samba:
> On Mon, 11 Mar 2019 07:16:30 +0100
> Christian via samba <samba at lists.samba.org> wrote:
>
>> Dear all,
>>
>> we are transitioning from an openldap / MIT KDC setup to a samba4 AD.
>> I am doing this by setting up a samba NT4 domain, populating it from
>> LDAP and sticking in the password hashes which I automatically
>> extract from the MIT KDC arc4-hmac keys. Then I run the
>> classicupgrade. I do this whole thing from cron in a script once a
>> day to be able to slowly migrate services. The MIT / openldap and
>> samba4 AD servers are on different machines. My script (based on
>> LPHvB's instructions) sets privileges in the following way:
>>
>> [..]
>>
>> systemctl restart bind9 ntp samba-ad-dc
>> sleep 5
>> SAMBA_DC_ADMIN_GROUP_CHOICE="BUILTIN\Administrators"
>> PRIVS="SeDiskOperatorPrivilege SeTakeOwnershipPrivilege \
>> SeBackupPrivilege SeRestorePrivilege SeRemoteShutdownPrivilege \
>> SePrintOperatorPrivilege SeAddUsersPrivilege SeDiskOperatorPrivilege \
>> SeSecurityPrivilege SeSystemtimePrivilege SeShutdownPrivilege \
>> SeDebugPrivilege SeSystemEnvironmentPrivilege
>> SeSystemProfilePrivilege \ SeProfileSingleProcessPrivilege
>> SeIncreaseBasePriorityPrivilege \ SeLoadDriverPrivilege
>> SeCreatePagefilePrivilege \ SeIncreaseQuotaPrivilege
>> SeChangeNotifyPrivilege SeUndockPrivilege \ SeManageVolumePrivilege
>> SeImpersonatePrivilege SeCreateGlobalPrivilege \
>> SeEnableDelegationPrivilege"
>>
>> samba-tool user setpassword Administrator \
>>   --newpassword="$SAMBA_NT_ADMIN_PASS"
>> echo "$SAMBA_NT_ADMIN_PASS" | kinit Administrator
>> for priv in $PRIVS ; do
>>   while ! net rpc rights grant
"${SAMBA_DC_ADMIN_GROUP_CHOICE}" $priv
>> \ -U "Administrator%$SAMBA_NT_ADMIN_PASS" ; do
>>     echo "Failed to grant $priv ... Retrying ..."
>>     sleep 10
>>   done
>> done
>>
>> Upon running this, I often get NT_STATUS_INTERNAL_ERROR or
>> NT_STATUS_IO_TIMEOUT:
>>
>> Changed password OK
>> Password for Administrator at XXXXXXXXXXXXXXXXX:
>> Could not connect to server 127.0.0.1
>> Connection failed: NT_STATUS_IO_TIMEOUT
>> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
>> Could not connect to server 127.0.0.1
>> Connection failed: NT_STATUS_IO_TIMEOUT
>> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
>> Could not connect to server 127.0.0.1
>> Connection failed: NT_STATUS_INTERNAL_ERROR
>> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
>> Successfully granted rights.
>> Successfully granted rights.
>> Successfully granted rights.
>>
>> Why would that happen? I can put in as much wait time as I want after
>> the initial service restart, and it still happens. I obviously work
>> around it by repeating until it proceeds, but I do not understand why
>> this is necessary. Any hints would be appreciated... Thanks,
>>
>> Christian
>>
>>
> This looks like a DNS problem, what do you have in /etc/resolv.conf ?
> Is Bind9 setting up correctly and running ?
Yep. /etc/resolv.conf points to 127.0.0.1 only. Not sure DNS is the
issue here as it says it cannot somehow talk to 127.0.0.1...
> What OS ?
Debian stable with LPHvB 4.8 packages.
> You do not need the 'kinit', you are not doing the changes via
kerberos.
I do that for something later down the road in the script when I use
samba-tool to add DNS records...

Thanks for any insights,

Christian

Am 11.03.2019 um 07:44 schrieb Andrew Bartlett:
> On Mon, 2019-03-11 at 07:16 +0100, Christian via samba wrote:
>> Dear all,
>>
>> we are transitioning from an openldap / MIT KDC setup to a samba4 AD. I
>> am doing this by setting up a samba NT4 domain, populating it from LDAP
>> and sticking in the password hashes which I automatically extract from
>> the MIT KDC arc4-hmac keys. Then I run the classicupgrade. I do this
>> whole thing from cron in a script once a day to be able to slowly
>> migrate services. The MIT / openldap and samba4 AD servers are on
>> different machines. My script (based on LPHvB's instructions) sets
>> privileges in the following way:
>>
>> [..]
>>
>> systemctl restart bind9 ntp samba-ad-dc
>> sleep 5
>> SAMBA_DC_ADMIN_GROUP_CHOICE="BUILTIN\Administrators"
>> PRIVS="SeDiskOperatorPrivilege SeTakeOwnershipPrivilege \
>> SeBackupPrivilege SeRestorePrivilege SeRemoteShutdownPrivilege \
>> SePrintOperatorPrivilege SeAddUsersPrivilege SeDiskOperatorPrivilege \
>> SeSecurityPrivilege SeSystemtimePrivilege SeShutdownPrivilege \
>> SeDebugPrivilege SeSystemEnvironmentPrivilege SeSystemProfilePrivilege
\
>> SeProfileSingleProcessPrivilege SeIncreaseBasePriorityPrivilege \
>> SeLoadDriverPrivilege SeCreatePagefilePrivilege \
>> SeIncreaseQuotaPrivilege SeChangeNotifyPrivilege SeUndockPrivilege \
>> SeManageVolumePrivilege SeImpersonatePrivilege SeCreateGlobalPrivilege
\
>> SeEnableDelegationPrivilege"
>>
>> samba-tool user setpassword Administrator \
>>   --newpassword="$SAMBA_NT_ADMIN_PASS"
>> echo "$SAMBA_NT_ADMIN_PASS" | kinit Administrator
>> for priv in $PRIVS ; do
>>   while ! net rpc rights grant
"${SAMBA_DC_ADMIN_GROUP_CHOICE}" $priv \
>>             -U "Administrator%$SAMBA_NT_ADMIN_PASS" ; do
>>     echo "Failed to grant $priv ... Retrying ..."
>>     sleep 10
>>   done
>> done
>>
>> Upon running this, I often get NT_STATUS_INTERNAL_ERROR or
>> NT_STATUS_IO_TIMEOUT:
>>
>> Changed password OK
>> Password for Administrator at XXXXXXXXXXXXXXXXX:
>> Could not connect to server 127.0.0.1
>> Connection failed: NT_STATUS_IO_TIMEOUT
>> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
>> Could not connect to server 127.0.0.1
>> Connection failed: NT_STATUS_IO_TIMEOUT
>> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
>> Could not connect to server 127.0.0.1
>> Connection failed: NT_STATUS_INTERNAL_ERROR
>> Failed to grant SeDiskOperatorPrivilege ... Retrying ...
>> Successfully granted rights.
>> Successfully granted rights.
>> Successfully granted rights.
>>
>> Why would that happen? I can put in as much wait time as I want after
>> the initial service restart, and it still happens. I obviously work
>> around it by repeating until it proceeds, but I do not understand why
>> this is necessary. Any hints would be appreciated... Thanks,
> 
> That certainly is strange, and perhaps the server-side logs or a
> network trace might provide a clue.  
Not found anything yet...
> I would however note that
> privileges are not very important in AD because they are not
> replicated.  Instead things tend to be provided by extended rights on
> magic AD objects.
Sure. Though if something mysterious is going on, it may be a sign of
another problem..
> Perhaps avoid re-generating the whole AD and just migrate the changed
> passwords?  A bit more fiddly but then things like GUIDs won't
keepchanging.
GUIDs changing are not a problem right now, as we will only be starting
to use the AD functionality once the migration is complete. For now it
just has to behave like a KDC / ldap combination. I did not find how to
stick the kerberos keys extracted from MIT into samba in a reasonable
amount of time; therefore I went for this classicupgrade approach... Thanks,

Christian