Puppet users - Feb 2012 - Mounts occasionally not applied on first run

Hi All,

Strange problem specifically to do with mounts. When I run Puppet for the
first time on a server, occasionally everything will get applied except for
mounts, and other times mounts are applied as part of the first run. No
errors in /var/log/messages unfortunately.

Has anyone else experienced this? Version 2.7.9.

Regards
Gonzalo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 23, 6:18 pm, Gonzalo Servat <gser...@gmail.com>
wrote:
> Hi All,
>
> Strange problem specifically to do with mounts. When I run Puppet for the
> first time on a server, occasionally everything will get applied except for
> mounts, and other times mounts are applied as part of the first run. No
> errors in /var/log/messages unfortunately.

I have not noticed this behavior, but you''re a little vague.  When you
say that the Mount resources are not applied, what exactly does that
mean?  Is /etc/fstab not modified?  Is fstab modified but the
filesystem is not mounted?  Either way, does the mount point exist
(and is it a directory) before the Mount is applied?  What shows up in
the client log?  Can you provide a simple manifest that exhibits the
behavior you describe (even if only intermitently)?


John




-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sat, Feb 25, 2012 at 2:06 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
> I have not noticed this behavior, but you''re a little vague.

Yes, I guess I wasn''t sure what I could provide in terms of detail so
your
questions help!

When you say that the Mount resources are not applied, what exactly
does
> that
> mean?  Is /etc/fstab not modified? Is fstab modified but the filesystem is
> not mounted?

/etc/fstab is not modified at all. I rebooted the server and made it
rebuild again and this time Puppet ran fine and all the mounts were applied
correctly.


> Either way, does the mount point exist (and is it a directory) before the
> Mount is applied?  What shows up in
> the client log?  Can you provide a simple manifest that exhibits the
> behavior you describe (even if only intermitently)?
>
I have a define that creates the directory and then mounts it. Nothing in
the client log indicating an error with the mounts. Unfortunately I
don''t
have a simple manifest that shows this behaviour. I was hoping someone
would say "oh yeah, seen that happen" based on my description, as
it''s not
something I can easily show in a manifest. It happens occasionally too
making the troubleshooting effort even harder. If I run Puppet again right
after the first run, it gets all the mounts done correctly. Most of the
time the first run is enough to get the entire catalog applied including
mounts.

- Gonzalo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 24, 9:16 am, Gonzalo Servat <gser...@gmail.com>
wrote:
> On Sat, Feb 25, 2012 at 2:06 AM, jcbollinger
<John.Bollin...@stjude.org>wrote:
>
>
>
> > I have not noticed this behavior, but you''re a little vague.
>
> Yes, I guess I wasn''t sure what I could provide in terms of detail
so your
> questions help!
>
> When you say that the Mount resources are not applied, what exactly does
>
> > that
> > mean?  Is /etc/fstab not modified? Is fstab modified but the
filesystem is
> > not mounted?
>
> /etc/fstab is not modified at all. I rebooted the server and made it
> rebuild again and this time Puppet ran fine and all the mounts were applied
> correctly.
>
> > Either way, does the mount point exist (and is it a directory) before
the
> > Mount is applied?  What shows up in
> > the client log?  Can you provide a simple manifest that exhibits the
> > behavior you describe (even if only intermitently)?
>
> I have a define that creates the directory and then mounts it. Nothing in
> the client log indicating an error with the mounts.

Does the client log show any errors at all?  Are there any other
resources that don''t always get applied?

If Puppet neither applies the Mount resource nor logs any client-side
message about it, then almost certainly the agent isn''t even *trying*
to apply the Mount.  That would imply that either the master doesn''t
think the client should have it (based on the classes assigned to the
node and/or conditional statements in them), or that some other
resource failed that must be applied first (because of resource
relationships).

> Unfortunately I don''t
> have a simple manifest that shows this behaviour. I was hoping someone
> would say "oh yeah, seen that happen" based on my description, as
it''s not
> something I can easily show in a manifest.

It doesn''t look like that''s happening.  Although we cannot
rule out a
Puppet bug, I am inclined to think that the problem is in your
manifests.

> It happens occasionally too
> making the troubleshooting effort even harder. If I run Puppet again right
> after the first run, it gets all the mounts done correctly. Most of the
> time the first run is enough to get the entire catalog applied including
> mounts.

Are you sure that Puppet *always* gets it on the second run, or could
your observation simply be a result of the unexpected behavior being
intermitent and not node-specific?  I mean, if it happens 10% of the
time, randomly, then there''s only a 1% chance of it happening twice in
a row on the same node.

Without seeing any code I don''t have much specific insight on the
problem, but here are some things you can try, individually or in
combination:

1) Run the agent in --debug mode
2) Sprinkle Notify resources through strategic parts of your manifests
to help you track which classes and definition resources are applied.
3) Run the agent many times in --noop mode to try to trigger the
misbehavior (and determine whether the misbehavior occurs via the
client-side log or YAML catalog).  When the misbehavior is detected,
look carefully at which resources are applied and which aren''t.
4) When the problem occurs, look at the client node''s facts as cached
on the master to verify that they don''t include anything unexpected
that might be related to the problem


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.