samba - Oct 2011 - Win7pro can't authenticate dcom identity

I have a Samba 3.4.7 PDC set up (Ubuntu 10.10/OpenLDAP) and have configured
a Windows 7 Pro 64 bit workstation as a domain member.  Logins and shares
work without a hitch.

I'm running into a problem with ArcGIS 10 Server on Windows using domain
accounts for the services.  ArcGIS post-install fails because it won't take
the username/password combination of the domain account.  I initially
thought this was an ArcGIS problem, but looking at the Samba logs, I find
"check_ntlm_password: sam authentication for user [ArcGISSOC] FAILED with
error NT_STATUS_WRONG_PASSWORD" entries whenever I try to configure the
service.

ArcGISSOC is the user account set up on our domain for the service, it has
both "log on as a service" and "log on as a batch account"
privileges, and I
have tried every permutation of settings.

By-passing ArcGIS's installation routines, if I go into
dcomcnfg->Component
Services->Computers->My Computer->DCOM Config->ArcSOC on the Windows
box and
under Identity, I set the username and password of the domain account, I
get:

The user account or password entered is invalid.  If you entered a domain
account, make sure the name is prefixed with the domain name.

Which, of course, it is.  I select the user by hitting the browse button to
make sure it is resolving correctly and the password is the same password I
used to log on to the machine.  I believe that this error is the cause of
the ArcGIS post-installation failure.

I have scoured for a couple of days now trying to find a cause for this and
I have come up empty.  Is this a Samba error, or a Windows 7 configuration
problem, or simply my own stupidity?  We have had this working on older
versions of Windows/ArcGIS.

Everything else works as expected, is there something weird about the way MS
DCOM authenticates for the identity?  Anyone who can shed some light on this
for me would have my eternal gratitude.

Thanks,

Chris

On Wed, Oct 26, 2011 at 7:25 AM, Chris Perry <outtascope at gmail.com>
wrote:
> I have a Samba 3.4.7 PDC set up (Ubuntu 10.10/OpenLDAP) and have configured
> a Windows 7 Pro 64 bit workstation as a domain member. Logins and shares
> work without a hitch.
>
> I'm running into a problem with ArcGIS 10 Server on Windows using
domain
> accounts for the services.  ArcGIS post-install fails because it won't
take
> the username/password combination of the domain account.  I initially
> thought this was an ArcGIS problem, but looking at the Samba logs, I find
> "check_ntlm_password: sam authentication for user [ArcGISSOC] FAILED
with
> error NT_STATUS_WRONG_PASSWORD" entries whenever I try to configure
the
> service.
>
Ok, I have fixed this problem on my site by patching Samba 3.4.7 on Ubuntu
10.04 (I incorrectly said 10.10 originally).  It isn't clear to me if this
is a Samba bug, a Windows bug, or a misconfiguration on either end of my
systems, but the following does fix my problem.

The issue appears to (my neophyte eyes) to be that login requests that fail
ntlmv2 authentication are not falling through gracefully to lm auth.  In
Samba, the code that determines whether or not to process is an ntlmv2
request checks the length parameter of the nt field in the RPC request.  If
this field is greater than or equal to 24, it processes as ntlmv2.  If it
processes the ntlmv2 check and it fails, it returns an
NT_STATUS_WRONG_PASSWORD to the client before it ever has a chance to check
lm.
Now I have played around with these auth settings on my Windows machine
endlessly and it does not seem to have an effect on what authentication
dcomcnfg uses when setting the identity of a component.
My instinct on this issue is that whatever initially processes the rpc
request in Samba is creating a zero-filled data element for both methods and
setting the length accordingly, regardless of whether the particular method
was requested or not (probably to avoid null pointer errors).  The ntlmv2
check assumes that the length should be zero for the nt data element if it
wasn't requested.  My patch additionally checks that the nt response
property is not just a zero-filled array before deciding that it is an
ntlmv2 request.

Again, I can't say that this is the right solution and I'm sure there
are
some enormous deficits in my understanding of how this is supposed to work,
but this patch does work if you are willing to compile yourself.  This
section of code has been changed quite a bit in Samba 3.5 and up, so it may
not be an issue on those new versions, but I can't say for sure.

If anyone has any additional insight into this (ie., I have an obvious
configuration error, I have misunderstood something, or I have created a
gaping security hole with this patch) I would appreciate the feedback.

Thanks,

- Chris




Extract from log files of failing configuration (real data replaced with
01234, 0000... sequences are directly from log).

in.logon.network.nt and in.logon.network.lm properties of netr_LogonSamLogon
struct at rpc_server/srv_pipe.c:2327(api_rpcTNP)
>From request generated when setting Identity property of DCOM object in
DComCnfg

nt: struct netr_ChallengeResponse
  length : 0x002c (44)
  size   : 0x002c (44)
  data   : *
    data :
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
lm: struct netr_ChallengeResponse
  length : 0x0018 (24)
  size   : 0x0018 (24)
  data   : *
    data : 012345678901234567890123456789010000000000000000
>From domain login on the same workstation
nt: struct netr_ChallengeResponse
  length : 0x010e (270)
  size   : 0x010e (270)
  data   : *
    data : 01234567890123456789012345 ... 01234567890123456789012345
lm: struct netr_ChallengeResponse
  length : 0x0018 (24)
  size   : 0x0018 (24)
  data   : *
    data : 000000000000000000000000000000000000000000000000



Patch to source3/libsmb/ntlm_check.c that makes this work:

--- samba-3.4.7~dfsg/source3/libsmb/ntlm_check.c        2011-10-28
11:05:38.000000000 -0400
+++ samba-3.4.7~dfsg-modified/source3/libsmb/ntlm_check.c       2011-10-28
11:41:12.000000000 -0400
@@ -28,6 +28,21 @@
 /****************************************************************************
  Core of smb password checking routine.
 ****************************************************************************/
+static bool nt_response_empty(TALLOC_CTX *mem_ctx,
+                              const DATA_BLOB *nt_response)
+{
+        if(nt_response->length == 0)
+                return true;
+
+        DATA_BLOB empty_response = data_blob_talloc_zero(mem_ctx,
nt_response->length);
+        if(memcmp(nt_response->data, empty_response.data,
nt_response->length)) {
+                data_blob_free(&empty_response);
+                return false;
+        }
+
+        data_blob_free(&empty_response);
+        return true;
+}

 static bool smb_pwd_check_ntlmv1(const DATA_BLOB *nt_response,
                                 const uchar *part_passwd,
@@ -283,7 +298,7 @@
                         (unsigned long)nt_response->length, username));
        }

-       if (nt_response->length >= 24 && nt_pw) {
+       if (nt_response->length >= 24 && nt_pw &&
!nt_response_empty(mem_ctx, nt_response)) {
                if (nt_response->length > 24) {
                        /* We have the NT MD4 hash challenge available - see
if we can
                           use it