Puppet users - Jul 2010 - A WTF moment brought to you by Puppet + [confused] nscd

To start with, I don’t think Puppet did anything wrong here. I’ll pin this
mainly on `nscd`, but it’s something you might want to know about if you’re
using both services. I’ll skip the long story of discovery, panic, and
investigation.

For historical reasons that don’t matter, we do the following on each system:

    user { "root":
      comment => "$hostname",
    }

Yesterday, `nscd` on one of our Puppet nodes lost its frakking mind and decided
there were no users on the box. Puppet comes along, looking to make sure the
comment for root has the correct value, and sees that there’s no such user… so
it adds it. I don’t know the exact command Puppet used (since it didn’t fail)
but either `useradd` or `usermod` was somehow aware of the next available UID
and that root already existed, so root had its UID (and home directory) changed.

That was a fun afternoon. “Why is root’s home set to `/home/root`? Holy shit!
Why is its UID 1273?!” And of course, though I was able to log in as root, I
didn’t have permission to examine most log files since its UID wasn’t 0.

Thankfully, the Puppet client was still up and running with UID 0, so I was able
to throw in a temporary rule to set root’s UID back to 0. After 30 minutes, I
was able to get in and figure out what happened.

Like I said, Puppet didn’t do anything wrong, but I wonder: If its internal
logic leads it to conclude that it needs to create new user named “root”, should
it maybe think twice?

-- 
Rob McBroom
<http://www.skurfer.com/>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Any reason for not specifying the uid parameter for the user?

   user { "root":
     uid => 0,
     comment => "$hostname",
   }

Regards,

Jonathan

On 20 July 2010 16:28, Rob McBroom <mailinglist0@skurfer.com>
wrote:
> To start with, I don’t think Puppet did anything wrong here. I’ll pin this
mainly on `nscd`, but it’s something you might want to know about if you’re
using both services. I’ll skip the long story of discovery, panic, and
investigation.
>
> For historical reasons that don’t matter, we do the following on each
system:
>
>    user { "root":
>      comment => "$hostname",
>    }
>
> Yesterday, `nscd` on one of our Puppet nodes lost its frakking mind and
decided there were no users on the box. Puppet comes along, looking to make sure
the comment for root has the correct value, and sees that there’s no such user…
so it adds it. I don’t know the exact command Puppet used (since it didn’t fail)
but either `useradd` or `usermod` was somehow aware of the next available UID
and that root already existed, so root had its UID (and home directory) changed.
>
> That was a fun afternoon. “Why is root’s home set to `/home/root`? Holy
shit! Why is its UID 1273?!” And of course, though I was able to log in as root,
I didn’t have permission to examine most log files since its UID wasn’t 0.
>
> Thankfully, the Puppet client was still up and running with UID 0, so I was
able to throw in a temporary rule to set root’s UID back to 0. After 30 minutes,
I was able to get in and figure out what happened.
>
> Like I said, Puppet didn’t do anything wrong, but I wonder: If its internal
logic leads it to conclude that it needs to create new user named “root”, should
it maybe think twice?
>
> --
> Rob McBroom
> <http://www.skurfer.com/>
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> Yesterday, `nscd` on one of our Puppet nodes lost its frakking mind
> and decided there were no users on the box. Puppet comes along,
> looking to make sure the comment for root has the correct value, and
> sees that there’s no such user… so it adds it. I don’t know the exact
> command Puppet used (since it didn’t fail) but either `useradd` or
> `usermod` was somehow aware of the next available UID and that root
> already existed, so root had its UID (and home directory) changed.
if you look into the provider code, you''ll see that it uses useradd,
BUT
afair puppet tries to predict the next available UID for you and passes
that to useradd as well. This might be the reason that it was
successfull as useradd didn''t choke, hence puppet ran successful.

In general there are some parts of a system which I still fear to touch
automatically. The root user is one of the last remaining parts, but as
Jonathan mentioned, setting uid => 0, might have helped.

cheers pete
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxFyL0ACgkQbwltcAfKi39IJgCaAjDfgA8uUibXbzpcpp0Nc8Ef
3WUAn3KBeoOiUPpXLqrUOjHh2ex83mz8
=fkMT
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jul 20, 2010, at 10:40 AM, Jonathan Share wrote:
> Any reason for not specifying the uid parameter for the user?
Well, just that I’m not looking to “change” the UID so I shouldn’t need to
mention it at all. After what happened yesterday though, I’m considering it.

-- 
Rob McBroom
<http://www.skurfer.com/>


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Some admins that I have known in the past would change root to something
else and have a different account with UID=0.

The UID is all that matters, you could have the account bob52 be UID 0
if you wanted to.

Trevor

On 07/20/2010 01:07 PM, Rob McBroom wrote:
> On Jul 20, 2010, at 10:40 AM, Jonathan Share wrote:
> 
>> Any reason for not specifying the uid parameter for the user?
> 
> Well, just that I’m not looking to “change” the UID so I shouldn’t need to
mention it at all. After what happened yesterday though, I’m considering it.
> 
- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaughan@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQEcBAEBAgAGBQJMRlSMAAoJECNCGV1OLcyppF8H/0bap/hY/8/ksjz3unzFKjdD
w0DprOYSmtQQ5RCHcWNSX5nwmdAIJFFCwWbHjfbEjqGDofTo/TBrqff7rwnKxatI
Nz1yUkXuhcyFR5hYnRka9QuWTOS4ScKxJhkInE6732Ox2QzdQlXAcK5FLvq/ItpB
9qT5GpM0ixy3GKn4mqtCh7LHy+wbI1ZF5VIkpC+ujFB2NsgZBq9R5dx7T0ef8ZE3
CfLPoZytxlrht5Lx3a3WQXrCksUofQpX0Fm96oL0gaFGQQnZmQXJexETqlKXtp7i
bUC/gVpCsFuNckHLzyQUWRbd21jMpTfz0BX3BT02SfeRUsaK7V6H7WRVlf431M4=2RMM
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.