samba - Oct 2008 - Bad passwords from Vampire / NT migration

I just did a clean install of Fedora 9 onto a Pentium 4 machine.  It is actually
running SELinux enforcing, but I haven't found any errors related to that so
far.  It is running Samba 3.2.3-0.20. I am trying to migrate a Windows NT 4.0
domain with about 30 users and 30 computers onto a pure Samba setup, using Samba
as the PDC and tdbsam.



I have read the HOWTO guide extensively, and everything seems to work in the
beginning.  My smb.conf is shown below in the BDC configuration.  I started with
no tdbsam database, so I started by creating the root account using pdbedit.  I
join the domain using net rpc join -S PDC.  No problem.  I setup the group maps
using by creating a script, as recommended.



Then I run the vampire command:

sudo net rpc vampire -S cls01



At this point I noticed it does not need a password, though the documentation
says to use -U administrator; I assume it's relying on the machine trust
account and its BDC membership.  It appears to create all user, workstation, and
group accounts properly.  The only errors are pertaining to the inability to add
groups with spaces:



[2008/10/13 23:41:26,  0] groupdb/mapping.c:smb_create_group(215)

  smb_create_group: Running the command `/usr/sbin/groupadd "Account
Operators"' gave 3



This is okay because I manually created unix groups and mapped them.  Now the
problem is that apparently all the passwords are coming over incorrectly.  If I
attempt to promote Linux to PDC and shutdown the old server, workstations
can't login, and Samba logs show complaints about a machine trust account
needing a password:



[2008/10/13 22:57:04,  0] rpc_server/srv_netlog_nt.c:get_md4pw(357)

  get_md4pw: Workstation GRADE7-8$: account does not have a password

[2008/10/13 22:57:04,  0]
rpc_server/srv_netlog_nt.c:_netr_ServerAuthenticate2(502)

  _netr_ServerAuthenticate2: failed to get machine password for account
GRADE7-8$: NT_STATUS_LOGON_FAILURE



I started looking at all the password hashes using pdbedit -Lw, and found the
following:



* There are no LanMan password hashes (all set to XXXX...)

* All machine accounts have neither LanMan nor NT password hashes:

WS6$:576:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:[W   
]:LCT-48D8EC0E:

* The NT hashes for the user accounts appear to be incorrect.



Otherwise everything looks right.  The account flags look to be correct, and
they all appear under the CLS domain.  The Logon time field looks correct,
though the password last set field shows the time of the vampire run.  Group
membership is correct when using net rpc user info.  When I test the logins by
using smbclient, in PDC mode, none of the existing passwords work, except the
root password I created before the vampire.



The weird part is that I ran the net rpc samdump command, and its output all
appears to be correct!  Almost all accounts besides workstations have LanMan
password hashes, and I went as far as to verify the NT hashes.  I did this by
creating a new user (pdbedit -a) and setting the password to what it is on the
NT DC.  The NT hash is the same as what appears in the samdump output.  However,
all NT hashes are different from what is stored in the tdbsam after vampire.


My conclusion is that the samdump appears to be doings its job, but somehow the
tdbsam is being written incorrectly with respect to password hashes.  It seems
as though the tdbsam database version is not in sync with the vampire tool,
though it's odd that there aren't any visible error messages. 
Shouldn't the LanMan / NT hashes be the same beteween pdbedit and samdump?

thanks,
Cooper



[global]

        workgroup = CLS

        security = user

        passdb backend = tdbsam

        domain master = no

        wins support = no

        domain logons = yes

        local master = yes

        preferred master = yes

        os level = 33



        logon path 
        logon home = 

        add user script = /usr/sbin/useradd "%u" -n -g users -s
/bin/false

        add user to group script = /usr/sbin/usermod -aG "%g"
"%u"

        add group script = /usr/sbin/groupadd "%g"

        add machine script = /usr/sbin/useradd -n -c "Workstation
(%u)" -M -d /nohome -s /bin/false "%u"

        delete user script = /usr/sbin/userdel "%u"

        delete group script = /usr/sbin/groupdel "%g"

        log file = /var/log/samba/log.%m

        max log size = 50



[homes]

        comment = Home Directories

        browseable = no

        writable = yes



[netlogon]

        comment = Network Logon Service

        path = /var/lib/samba/netlogon

        read only = yes

> I started looking at all the password hashes using pdbedit -Lw, and
> found the following:
>
> * There are no LanMan password hashes (all set to XXXX...)
> * Most machine accounts have neither LanMan nor NT password hashes:
> WS6$:576:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:
>	XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:[W          ]:LCT-48D8EC0E:
> * The NT hashes for the user accounts appear to be incorrect.
Can anyone tell me why net rpc samdump gets the correct LM and NT
password hashes, but net rpc vampire gets incorrect hashes?  What's
funny is that vampire seems to produce consistent results, but
they're consistently wrong.

Is it possible that the NT PDC doesn't trust the Samba server so
it gives it bad hashes?

If there was an easy way to update the NT password hashes in the
passdb tdb file then I would do it.  I could import the samdump via
the pdbedit import feature, but then I have to remap SIDs, which I
have been unable to do.  Perhaps using tdbtool I could locate the NT
hash bytes and replace them, but this would be tedious.

Any thoughts?  Suggestions?

thanks,
Cooper

> Can anyone tell me why net rpc samdump gets the correct LM and NT
> password hashes, but net rpc vampire gets incorrect hashes?  What's
> funny is that vampire seems to produce consistent results, but
> they're consistently wrong.
> 
> Is it possible that the NT PDC doesn't trust the Samba server so
> it gives it bad hashes?
I have a few more comments.
1. net rpc vampire does not set the machine or domain SID.  This has
to be done manually using net setlocalsid and net setdomainsid.
Calling net rpc getsid appears to work but does not change anything.
I have seen multiple other people with this same problem going back
a couple years, so it appears to be a longstanding bug.

2. I checked the event log on the Windows NT PDC.  It's interesting
because each time I run the vampire command, it logs 2 or 5 5722
events in a row (the error is that the session setup from my samba
BDC failed to authenticate with the error Access is denied.)  This
error is indicating an invalid machine password from Samba.

Then immediately after the error messages I get two 5713 events,
indicating that the full synchronization request from the BDC
completed successfully.  The first event refers to over 100 objects
and the second event refers to a much smaller number.  So despite
the 5722 error, everything synchronizes.

3. The only evidence of any problem from the vampire command is the
events logged on the PDC, and the invalid passwords.  I tried
deleting the trust account on the PDC and rejoining several times,
with Samba on, off, and nmbd on and off.  The result is always the
same.  The bad password hashes are always the same for each account.
If I change a password on the PDC then run vampire again, the NT
hash changes on the Samba box.  It just seems like the NT hash is
somehow being scrambled, but in a consistent way.

4. It does not seem to matter if I create the BDC trust account on
the PDC using Server Manager, or whether I just join the domain
using net rpc join.  The former step seems unncessary.

5. Here is the stderr output from the vampire command:
[2008/10/20 21:08:23,  0] passdb/pdb_tdb.c:tdb_update_samacct_only(1117)

I did also save the debug level 10 output, but it really doesn't look
to contain anything interesting.

-Cooper

> Great catch. Both look valid to me. I think the best fix for
> 3.2 is to always set rid_crypt to true, and remove all the
> other sam_pwd_hash() calls - just do it in the one place.
> 
> Ok, here is a quick patch for 3.2. It removes some silly
> static buffers and changes all calls to samsync_fix_delta_array()
> to set rid_crypt = true and then removes all the extra
> crypto sam_pwd_hash() calls that are no longer needed.
> 
> Can you confirm it works for you and I'll check it in
> with your credit, and then fix 3.3 and master in the
> same way.
> 
> Thanks a *LOT*,
> 
> Jeremy.
Thanks for taking a look.  Unfortunately I'm not really setup to
verify the patch or compile the code, as I was just browsing through
the git repositories over the web to try to find the problem.

My current setup is Fedora 9 using packages.  Maybe in the next few
days I can work on downloading and building the 3.2 branch and
trying the fix.  I'm still confused about why my vampire would
result in the LM hash being null but the NT hash being set, but
hopefully this will take care of it.

In either case I will post results if I can get it running.

thanks,
Cooper

> Ok, I've pushed the third versino into the git repositories,
> so if you can confirm the fix I'd appreciate it.
> 
> Thanks a *LOT* for that bug report - absolutely perfect !
> 
> Jeremy.
OK, well it was easier than I thought to compile the code, so I
did it tonight.  Yes indeed, that was the problem!  I did a fresh
net rpc join to the domain, then ran net rpc vampire and it
successfully loaded all accounts with proper NT hashes.  The
LanMan hashes are all NULL, but I assume that's by design for
security reasons.  I tested using the v3-2-test branch from git.

This is why open source is cool:  being able to test a patch on
the same day is pretty impressive.  Thanks for the help, and keep
up the good work.

-Cooper