We have an environment where we have to place some files on systems owned by
''ldap'' users... that is, users that are not local, but are
held in LDAP. We''ve done everything we can to stabalize our LDAP
environment, but we still run into an issue where hosts randomly pop out
failures like:
err Could not evaluate: Could not find user xxx
/Stage[main]//Node[cseng_base]/Auth::Key[xxx]/File[/home/xxx] 2010-11-01 20:46
UTC
warning Skipping because of failed dependencies
/Stage[main]//Node[cseng_base]/Auth::Key[xxx]/File[/home/xxx/.ssh] 2010-11-01
20:46 UTC
warning Skipping because of failed dependencies
/Stage[main]//Node[cseng_base]/Auth::Key[xxx]/File[/home/xxx/.ssh/authorized_keys]
2010-11-01 20:46 UTC
notice Dependency File[/home/xxx] has failures: true
/Stage[main]//Node[cseng_base]/Auth::Key[xxx]/File[/home/xxx/.ssh/authorized_keys]
2010-11-01 20:46 UTC
notice Dependency File[/home/xxx] has failures: true
/Stage[main]//Node[cseng_base]/Auth::Key[xxx]/File[/home/xxx/.ssh] 2010-11-01
20:46 UTC
notice created
/Stage[main]//Node[cseng_base]/Auth::Key[yyy]/File[/home/yyy]/ensure 2010-11-01
20:46 UTC
notice created
/Stage[main]//Node[cseng_base]/Auth::Key[yyy]/File[/home/yyy/.ssh]/ensure
2010-11-01 20:46 UTC
notice defined content as
''{md5}71e80c50ae9c737cbfbc82fb56a792af''
/Stage[main]//Node[cseng_base]/Auth::Key[yyy]/File[/home/yyy/.ssh/authorized_keys]/ensure
You can see two different users that are both in LDAP.. but one fails when
puppet tries to set its permissions ,and the other succeeds.
1) Any thoughts on why puppet randomly fails the lookup of the user id?
2) Why is puppet looking up the userid anyways — i just want it to do verify
that the username is ''xxx'' and walk away. Whats the mechanism
puppet is using for this?
On the systems we use ''files db ldap'' as our nsswitch.conf
priority, and ''db'' is a local copy of the ldap data using
''nsscache'' on a regular basis. Looking up a user should never
fail.... and it doesnt seem to when we log in and check by hand, so I
don''t know why puppet is failing.
—Matt
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Bruce Richardson
2010-Nov-02 01:53 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Mon, Nov 01, 2010 at 02:10:23PM -0700, Matt Wise wrote:> > On the systems we use ''files db ldap'' as our nsswitch.conf priority, > and ''db'' is a local copy of the ldap data using ''nsscache'' on a > regular basis. Looking up a user should never fail.... and it doesnt > seem to when we log in and check by hand, so I don''t know why puppet > is failing.nsscache is probably at fault. Try removing it from nsswitch on a machine which has shown this problem and then see how that affects things. -- Bruce I must admit that the existence of Disneyland (which I know is real) proves that we are not living in Judea in AD 50. -- Philip K. Dick
nsscache was an attempt at solving the issue... even without nscd installed, or nsscache, the issue happens extremely frequently. so far the only fix i have is to do everything with hard-coded GID/UIDs ... On Nov 1, 2010, at 6:53 PM, Bruce Richardson wrote:> On Mon, Nov 01, 2010 at 02:10:23PM -0700, Matt Wise wrote: >> >> On the systems we use ''files db ldap'' as our nsswitch.conf priority, >> and ''db'' is a local copy of the ldap data using ''nsscache'' on a >> regular basis. Looking up a user should never fail.... and it doesnt >> seem to when we log in and check by hand, so I don''t know why puppet >> is failing. > > nsscache is probably at fault. Try removing it from nsswitch on > a machine which has shown this problem and then see how that affects > things. > > -- > Bruce > > I must admit that the existence of Disneyland (which I know is real) > proves that we are not living in Judea in AD 50. -- Philip K. Dick-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2010-Nov-02 04:00 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Mon, Nov 1, 2010 at 8:27 PM, Matt Wise <wise@wiredgeek.net> wrote:> nsscache was an attempt at solving the issue... even without nscd installed, or nsscache, the issue happens extremely frequently. so far the only fix i have is to do everything with hard-coded GID/UIDs ...You can mitigate this a little with some global variables. Did you notice this happen in other versions of Puppet? When you say "extremely frequently", how often is that?> > On Nov 1, 2010, at 6:53 PM, Bruce Richardson wrote: > >> On Mon, Nov 01, 2010 at 02:10:23PM -0700, Matt Wise wrote: >>> >>> On the systems we use ''files db ldap'' as our nsswitch.conf priority, >>> and ''db'' is a local copy of the ldap data using ''nsscache'' on a >>> regular basis. Looking up a user should never fail.... and it doesnt >>> seem to when we log in and check by hand, so I don''t know why puppet >>> is failing. >> >> nsscache is probably at fault. Try removing it from nsswitch on >> a machine which has shown this problem and then see how that affects >> things. >> >> -- >> Bruce >> >> I must admit that the existence of Disneyland (which I know is real) >> proves that we are not living in Judea in AD 50. -- Philip K. Dick > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten - Puppet Labs - http://www.puppetlabs.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Bruce Richardson
2010-Nov-02 08:03 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Mon, Nov 01, 2010 at 08:27:30PM -0700, Matt Wise wrote:> nsscache was an attempt at solving the issue... even without nscd > installed, or nsscache, the issue happens extremely frequently. so far > the only fix i have is to do everything with hard-coded GID/UIDs ...Is it only Puppet which has this problem, or do other processes have difficulties resolving usernames and uids? If it''s an LDAP/nsswitch configuration issue, may be hard for us to diagnose just by looking at Puppet configs. Are these errors happening with exisiting users or new ones which are being created during puppet runs? -- Bruce I must admit that the existence of Disneyland (which I know is real) proves that we are not living in Judea in AD 50. -- Philip K. Dick -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Michael Gliwinski
2010-Nov-02 09:35 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Tuesday 02 Nov 2010 01:53:34 Bruce Richardson wrote:> nsscache is probably at fault. Try removing it from nsswitch on > a machine which has shown this problem and then see how that affects > things.Why do you think nsscache could be a problem? The reason I''m asking is I''m having some issues with nss_ldap + nscd (on CentOS) as well and was actually going to look into nsscache as possible workaround. -- Michael Gliwinski Henderson Group Information Services 9-11 Hightown Avenue, Newtownabby, BT36 4RT Phone: 028 9034 3319 ********************************************************************************************** The information in this email is confidential and may be legally privileged. It is intended solely for the addressee and access to the email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients, any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing client engagement leter or contract. If you have received this email in error please notify support@henderson-group.com John Henderson (Holdings) Ltd Registered office: 9 Hightown Avenue, Mallusk, County Antrim, Northern Ireland, BT36 4RT. Registered in Northern Ireland Registration Number NI010588 Vat No.: 814 6399 12 ********************************************************************************* -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Bruce Richardson
2010-Nov-02 12:44 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Tue, Nov 02, 2010 at 09:35:24AM +0000, Michael Gliwinski wrote:> On Tuesday 02 Nov 2010 01:53:34 Bruce Richardson wrote: > > nsscache is probably at fault. Try removing it from nsswitch on > > a machine which has shown this problem and then see how that affects > > things. > > Why do you think nsscache could be a problem?It and nscd often are. Better to ensure that your LDAP directory is resilient and responsive. Configure pam and nsswitch so that you can still get into a box as a non-LDAP user in the worst case scenario where LDAP is inaccessible. Caching the directory just causes a whole new special kind of latency issues while making problems harder to debug. -- Bruce I unfortunately do not know how to turn cheese into gold. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Michael Gliwinski
2010-Nov-02 15:57 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Tuesday 02 Nov 2010 12:44:38 Bruce Richardson wrote:> On Tue, Nov 02, 2010 at 09:35:24AM +0000, Michael Gliwinski wrote: > > On Tuesday 02 Nov 2010 01:53:34 Bruce Richardson wrote: > > > nsscache is probably at fault. Try removing it from nsswitch on > > > a machine which has shown this problem and then see how that affects > > > things. > > > > Why do you think nsscache could be a problem? > > It and nscd often are. Better to ensure that your LDAP directory is > resilient and responsive. Configure pam and nsswitch so that you can > still get into a box as a non-LDAP user in the worst case scenario where > LDAP is inaccessible. Caching the directory just causes a whole new > special kind of latency issues while making problems harder to debug.Hmm, I know what you mean, but then without any caching this can mean quite a load on the LDAP servers considering even simple ''ls -l'' has to go over network to resolve UIDs/GIDs. Anyway, do you happen to know of a good guide, howto, etc. for nss_ldap configuration (specifically on CentOS with AD)? I''m thinking maybe client configuration is at fault but haven''t yet encountered a good guide that would explain what is needed and why (e.g. many also mention winbind, not sure what for, also one of our admins here was playing with Kerberos which seems isn''t needed and could be contributing to the issues). -- Michael Gliwinski Henderson Group Information Services 9-11 Hightown Avenue, Newtownabby, BT36 4RT Phone: 028 9034 3319 ********************************************************************************************** The information in this email is confidential and may be legally privileged. It is intended solely for the addressee and access to the email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients, any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing client engagement leter or contract. If you have received this email in error please notify support@henderson-group.com John Henderson (Holdings) Ltd Registered office: 9 Hightown Avenue, Mallusk, County Antrim, Northern Ireland, BT36 4RT. Registered in Northern Ireland Registration Number NI010588 Vat No.: 814 6399 12 ********************************************************************************* -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
We use nsscache because nscd is so unreliable. Nsscache is simple enough that it works, and it works pretty well. As Michael said, without it your system is sending LDAP queries for almost every operation that uses getpw/getuser. We do not see random LDAP failures from other processes on our systems — but we see at least 50% of Puppet runs fail if they try to check/set ownership of a file to an ldap user. These failures go away completely when we do UID/GID only. (which is our work-around for now) —Matt On Nov 2, 2010, at 8:57 AM, Michael Gliwinski wrote:> On Tuesday 02 Nov 2010 12:44:38 Bruce Richardson wrote: >> On Tue, Nov 02, 2010 at 09:35:24AM +0000, Michael Gliwinski wrote: >>> On Tuesday 02 Nov 2010 01:53:34 Bruce Richardson wrote: >>>> nsscache is probably at fault. Try removing it from nsswitch on >>>> a machine which has shown this problem and then see how that affects >>>> things. >>> >>> Why do you think nsscache could be a problem? >> >> It and nscd often are. Better to ensure that your LDAP directory is >> resilient and responsive. Configure pam and nsswitch so that you can >> still get into a box as a non-LDAP user in the worst case scenario where >> LDAP is inaccessible. Caching the directory just causes a whole new >> special kind of latency issues while making problems harder to debug. > > Hmm, I know what you mean, but then without any caching this can mean quite a > load on the LDAP servers considering even simple ''ls -l'' has to go over > network to resolve UIDs/GIDs. > > Anyway, do you happen to know of a good guide, howto, etc. for nss_ldap > configuration (specifically on CentOS with AD)? I''m thinking maybe client > configuration is at fault but haven''t yet encountered a good guide that would > explain what is needed and why (e.g. many also mention winbind, not sure what > for, also one of our admins here was playing with Kerberos which seems isn''t > needed and could be contributing to the issues). > > > -- > Michael Gliwinski > Henderson Group Information Services > 9-11 Hightown Avenue, Newtownabby, BT36 4RT > Phone: 028 9034 3319 > > ********************************************************************************************** > The information in this email is confidential and may be legally privileged. It is intended solely for the addressee and access to the email by anyone else is unauthorised. > If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. > When addressed to our clients, any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing client engagement leter or contract. > If you have received this email in error please notify support@henderson-group.com > > John Henderson (Holdings) Ltd > Registered office: 9 Hightown Avenue, Mallusk, County Antrim, Northern Ireland, BT36 4RT. > Registered in Northern Ireland > Registration Number NI010588 > Vat No.: 814 6399 12 > ********************************************************************************* > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2010-Nov-03 22:58 UTC
Re: [Puppet Users] Help with LDAP, Users, and Puppet..
On Wed, Nov 3, 2010 at 3:38 PM, Matt Wise <wise@wiredgeek.net> wrote:> We use nsscache because nscd is so unreliable. Nsscache is simple enough that it works, and it works pretty well. As Michael said, without it your system is sending LDAP queries for almost every operation that uses getpw/getuser. > > We do not see random LDAP failures from other processes on our systems — but we see at least 50% of Puppet runs fail if they try to check/set ownership of a file to an ldap user. These failures go away completely when we do UID/GID only. (which is our work-around for now)That does sound like you''ve found a bug, and I''m not particularly fond of this section of our codebase. Can you reproduce it with repeated applications of a standalone puppet manifest with a single resource?> > —Matt > > On Nov 2, 2010, at 8:57 AM, Michael Gliwinski wrote: > >> On Tuesday 02 Nov 2010 12:44:38 Bruce Richardson wrote: >>> On Tue, Nov 02, 2010 at 09:35:24AM +0000, Michael Gliwinski wrote: >>>> On Tuesday 02 Nov 2010 01:53:34 Bruce Richardson wrote: >>>>> nsscache is probably at fault. Try removing it from nsswitch on >>>>> a machine which has shown this problem and then see how that affects >>>>> things. >>>> >>>> Why do you think nsscache could be a problem? >>> >>> It and nscd often are. Better to ensure that your LDAP directory is >>> resilient and responsive. Configure pam and nsswitch so that you can >>> still get into a box as a non-LDAP user in the worst case scenario where >>> LDAP is inaccessible. Caching the directory just causes a whole new >>> special kind of latency issues while making problems harder to debug. >> >> Hmm, I know what you mean, but then without any caching this can mean quite a >> load on the LDAP servers considering even simple ''ls -l'' has to go over >> network to resolve UIDs/GIDs. >> >> Anyway, do you happen to know of a good guide, howto, etc. for nss_ldap >> configuration (specifically on CentOS with AD)? I''m thinking maybe client >> configuration is at fault but haven''t yet encountered a good guide that would >> explain what is needed and why (e.g. many also mention winbind, not sure what >> for, also one of our admins here was playing with Kerberos which seems isn''t >> needed and could be contributing to the issues). >> >> >> -- >> Michael Gliwinski >> Henderson Group Information Services >> 9-11 Hightown Avenue, Newtownabby, BT36 4RT >> Phone: 028 9034 3319 >> >> ********************************************************************************************** >> The information in this email is confidential and may be legally privileged. It is intended solely for the addressee and access to the email by anyone else is unauthorised. >> If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. >> When addressed to our clients, any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing client engagement leter or contract. >> If you have received this email in error please notify support@henderson-group.com >> >> John Henderson (Holdings) Ltd >> Registered office: 9 Hightown Avenue, Mallusk, County Antrim, Northern Ireland, BT36 4RT. >> Registered in Northern Ireland >> Registration Number NI010588 >> Vat No.: 814 6399 12 >> ********************************************************************************* >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten - Puppet Labs - http://www.puppetlabs.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.