madAndroid
2011-Nov-09 15:00 UTC
[Puppet Users] NFS clients - puppet hang when mount point
We''ve only recently discovered that puppet can manage mount points using the mount directive; however, a short while back we built an nfs client and server classes without using this resource, and we''ve encountered a problem where puppet seems to hang when the nfs server is unavailable. Using --debug doesn''t seem to specify exactly at which point the run is failing, which could steer us in the right direction around putting something in place in the classes in question. Is there anything we can do, short of switching over to using the mount directive/resource, in order to mitigate the problem when the nfs server is unavailable? It''s preventing us from managing other resources on the clients when this happens.. thanks in advance, Andrew -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Guy Matz
2011-Nov-09 19:12 UTC
Re: [Puppet Users] NFS clients - puppet hang when mount point
OK. This may seem like a bad idea, but it''s a workaround that has worked for me: I add the nfs server to the 127.0.0.1 entry of the hosts file which causes NFS to time out pretty immediately. :-\ On Wed, Nov 9, 2011 at 10:00 AM, madAndroid <andrewstangl@gmail.com> wrote:> We''ve only recently discovered that puppet can manage mount points > using the mount directive; > however, a short while back we built an nfs client and server classes > without using this resource, and we''ve encountered a problem where > puppet seems to hang when the nfs server is unavailable. > > Using --debug doesn''t seem to specify exactly at which point the run > is failing, which could steer us in the right direction around putting > something in place in the classes in question. > > Is there anything we can do, short of switching over to using the > mount directive/resource, in order to mitigate the problem when the > nfs server is unavailable? It''s preventing us from managing other > resources on the clients when this happens.. > > thanks in advance, > Andrew > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
madAndroid
2011-Nov-10 12:02 UTC
[Puppet Users] Re: NFS clients - puppet hang when mount point
That sounds a little heavy handed ... also, I feel that it would probably stop the mount from happening at all? how would the fstab initiate the nfs mount if it''s not able to resolve the address of the nfs server correctly? unless I''m missing something .. On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote:> OK. This may seem like a bad idea, but it''s a workaround that has worked > for me: > I add the nfs server to the 127.0.0.1 entry of the hosts file which causes > NFS to time out pretty immediately. :-\ > > > > > > > > On Wed, Nov 9, 2011 at 10:00 AM, madAndroid <andrewsta...@gmail.com> wrote: > > We''ve only recently discovered that puppet can manage mount points > > using the mount directive; > > however, a short while back we built an nfs client and server classes > > without using this resource, and we''ve encountered a problem where > > puppet seems to hang when the nfs server is unavailable. > > > Using --debug doesn''t seem to specify exactly at which point the run > > is failing, which could steer us in the right direction around putting > > something in place in the classes in question. > > > Is there anything we can do, short of switching over to using the > > mount directive/resource, in order to mitigate the problem when the > > nfs server is unavailable? It''s preventing us from managing other > > resources on the clients when this happens.. > > > thanks in advance, > > Andrew > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group.> To post to this group, send email topuppet-users@googlegroups.com. > > To unsubscribe from this group, send email to>puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Len Rugen
2011-Nov-10 13:51 UTC
Re: [Puppet Users] Re: NFS clients - puppet hang when mount point
If you can reproduce this, you might try adding the "bg" option to the mount options (via options => in puppet). I suspect that when puppet tries to do the nfs mount with the nfs server unavailable, it will hang for 120 seconds or more. I think it you tried the same mount command at that time outside of puppet, it would hang. I suspect you have ensure => mounted, if ensure => present and atboot => true would be adequate, you might circumvent this problem. Having puppet ensure the mount is cool, but the 30 minute lag isn''t adequate in our shop to detect and correct a missing resource. I would have our monitoring system (Zabbix) more frequently check resources, maybe every minute or two and not rely on puppet alone. On Thu, Nov 10, 2011 at 6:02 AM, madAndroid <andrewstangl@gmail.com> wrote:> That sounds a little heavy handed ... > > also, I feel that it would probably stop the mount from happening at > all? > how would the fstab initiate the nfs mount if it''s not able to resolve > the address of the nfs server correctly? > unless I''m missing something .. > > > On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote: > > OK. This may seem like a bad idea, but it''s a workaround that has worked > > for me: > > I add the nfs server to the 127.0.0.1 entry of the hosts file which > causes > > NFS to time out pretty immediately. :-\ > > > > > > > > > > > > > > > > On Wed, Nov 9, 2011 at 10:00 AM, madAndroid <andrewsta...@gmail.com> > wrote: > > > We''ve only recently discovered that puppet can manage mount points > > > using the mount directive; > > > however, a short while back we built an nfs client and server classes > > > without using this resource, and we''ve encountered a problem where > > > puppet seems to hang when the nfs server is unavailable. > > > > > Using --debug doesn''t seem to specify exactly at which point the run > > > is failing, which could steer us in the right direction around putting > > > something in place in the classes in question. > > > > > Is there anything we can do, short of switching over to using the > > > mount directive/resource, in order to mitigate the problem when the > > > nfs server is unavailable? It''s preventing us from managing other > > > resources on the clients when this happens.. > > > > > thanks in advance, > > > Andrew > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Puppet Users" group.> To post to this group, send email > topuppet-users@googlegroups.com. > > > To unsubscribe from this group, send email to> > puppet-users+unsubscribe@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2011-Nov-10 15:10 UTC
[Puppet Users] Re: NFS clients - puppet hang when mount point
On Nov 10, 6:02 am, madAndroid <andrewsta...@gmail.com> wrote:> That sounds a little heavy handed ... > > also, I feel that it would probably stop the mount from happening at > all? > how would the fstab initiate the nfs mount if it''s not able to resolve > the address of the nfs server correctly? > unless I''m missing something .. > > On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote: > > > OK. This may seem like a bad idea, but it''s a workaround that has worked > > for me: > > I add the nfs server to the 127.0.0.1 entry of the hosts file which causes > > NFS to time out pretty immediately. :-\ > > > On Wed, Nov 9, 2011 at 10:00 AM, madAndroid <andrewsta...@gmail.com> wrote: > > > We''ve only recently discovered that puppet can manage mount points > > > using the mount directive; > > > however, a short while back we built an nfs client and server classes > > > without using this resource, and we''ve encountered a problem where > > > puppet seems to hang when the nfs server is unavailable. > > > > Using --debug doesn''t seem to specify exactly at which point the run > > > is failing, which could steer us in the right direction around putting > > > something in place in the classes in question. > > > > Is there anything we can do, short of switching over to using the > > > mount directive/resource, in order to mitigate the problem when the > > > nfs server is unavailable? It''s preventing us from managing other > > > resources on the clients when this happens..It all comes down to mount options. I''d recommend you absorb the material in nfs(5), but options of particular interest include ''retry'', ''retrans'', and ''timeo''. Do notice that the latter two will affect all operations on the mounted filesystem, not just the initial mount (and maybe not the initial mount at all -- the docs aren''t quite clear on that). Unlike Len, I would not recommend using the "bg" option. Doing so likely would prevent Puppet from hanging for a long time attempting the mount, but it would also prevent Puppet from correctly managing the resource. Puppet needs to know whether it has succeeded in mounting the remote filesystem. It may also cause you trouble later if the client never does succeed in mounting the filesystem. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
madAndroid
2011-Nov-11 11:03 UTC
[Puppet Users] Re: NFS clients - puppet hang when mount point
Thank you so much to all who responded - we''ve managed to rewrite our classes to use the mount directive rather (and quite quickly and painlessly), and it seems a much better way than we had before (using file and exec resources/directives). Still experiencing the timeouts during the puppet run when the NFS box goes away (in a dev environment during testing the new class structure), but we haven''t tried the two options that Len suggested - will be giving those a go, with the trade- offs in mind. Yes, I agree with John that it does feel like more of a problem with the way we are implementing the mounts, will give those suggestions a try as well. Thanks again, Andrew On Nov 10, 3:10 pm, jcbollinger <John.Bollin...@stJude.org> wrote:> On Nov 10, 6:02 am, madAndroid <andrewsta...@gmail.com> wrote: > > > > > That sounds a little heavy handed ... > > > also, I feel that it would probably stop the mount from happening at > > all? > > how would the fstab initiate the nfs mount if it''s not able to resolve > > the address of the nfs server correctly? > > unless I''m missing something .. > > > On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote: > > > > OK. This may seem like a bad idea, but it''s a workaround that has worked > > > for me: > > > I add the nfs server to the 127.0.0.1 entry of the hosts file which causes > > > NFS to time out pretty immediately. :-\ > > > > On Wed, Nov 9, 2011 at 10:00 AM, madAndroid <andrewsta...@gmail.com> wrote: > > > > We''ve only recently discovered that puppet can manage mount points > > > > using the mount directive; > > > > however, a short while back we built an nfs client and server classes > > > > without using this resource, and we''ve encountered a problem where > > > > puppet seems to hang when the nfs server is unavailable. > > > > > Using --debug doesn''t seem to specify exactly at which point the run > > > > is failing, which could steer us in the right direction around putting > > > > something in place in the classes in question. > > > > > Is there anything we can do, short of switching over to using the > > > > mount directive/resource, in order to mitigate the problem when the > > > > nfs server is unavailable? It''s preventing us from managing other > > > > resources on the clients when this happens.. > > It all comes down to mount options. I''d recommend you absorb the > material in nfs(5), but options of particular interest include > ''retry'', ''retrans'', and ''timeo''. Do notice that the latter two will > affect all operations on the mounted filesystem, not just the initial > mount (and maybe not the initial mount at all -- the docs aren''t quite > clear on that). > > Unlike Len, I would not recommend using the "bg" option. Doing so > likely would prevent Puppet from hanging for a long time attempting > the mount, but it would also prevent Puppet from correctly managing > the resource. Puppet needs to know whether it has succeeded in > mounting the remote filesystem. It may also cause you trouble later > if the client never does succeed in mounting the filesystem. > > John-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
madAndroid
2011-Nov-15 15:58 UTC
[Puppet Users] Re: NFS clients - puppet hang when mount point
FYI,
We ended up going with a mix of the two suggestions: having
''bg''
specified as one of the mount point options,
and also having the mount resource specify => present instead of =>
mounted, and then we use an exec to force
the remount, constrained by an unless and onlyif:
exec { "remount-storage":
command => "mount -a",
unless => "grep $brand /proc/mounts",
onlyif => "nc -zv $nfs_server 2049 -w 3",
}
The works for us in all use cases, and seems to be the best way
forwards :)
- Andrew
On Nov 11, 11:03 am, madAndroid <andrewsta...@gmail.com>
wrote:> Thank you so much to all who responded - we''ve managed to rewrite
our
> classes to use the mount directive rather (and quite quickly and
> painlessly), and it seems a much better way than we had before (using
> file and exec resources/directives). Still experiencing the timeouts
> during the puppet run when the NFS box goes away (in a dev environment
> during testing the new class structure), but we haven''t tried the
two
> options that Len suggested - will be giving those a go, with the trade-
> offs in mind. Yes, I agree with John that it does feel like more of a
> problem with the way we are implementing the mounts, will give those
> suggestions a try as well.
>
> Thanks again,
> Andrew
>
> On Nov 10, 3:10 pm, jcbollinger <John.Bollin...@stJude.org> wrote:
>
> > On Nov 10, 6:02 am, madAndroid <andrewsta...@gmail.com> wrote:
>
> > > That sounds a little heavy handed ...
>
> > > also, I feel that it would probably stop the mount from happening
at
> > > all?
> > > how would the fstab initiate the nfs mount if it''s not
able to resolve
> > > the address of the nfs server correctly?
> > > unless I''m missing something ..
>
> > > On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote:
>
> > > > OK. This may seem like a bad idea, but it''s a
workaround that has worked
> > > > for me:
> > > > I add the nfs server to the 127.0.0.1 entry of the hosts
file which causes
> > > > NFS to time out pretty immediately. :-\
>
> > > > On Wed, Nov 9, 2011 at 10:00 AM, madAndroid
<andrewsta...@gmail.com> wrote:
> > > > > We''ve only recently discovered that puppet can
manage mount points
> > > > > using the mount directive;
> > > > > however, a short while back we built annfs clientand
server classes
> > > > > without using this resource, and we''ve
encountered a problem where
> > > > > puppet seems to hang when the nfs server is
unavailable.
>
> > > > > Using --debug doesn''t seem to specify exactly
at which point the run
> > > > > is failing, which could steer us in the right direction
around putting
> > > > > something in place in the classes in question.
>
> > > > > Is there anything we can do, short of switching over to
using the
> > > > > mount directive/resource, in order to mitigate the
problem when the
> > > > > nfs server is unavailable? It''s preventing us
from managing other
> > > > > resources on the clients when this happens..
>
> > It all comes down to mount options. I''d recommend you absorb
the
> > material in nfs(5), but options of particular interest include
> > ''retry'', ''retrans'', and
''timeo''. Do notice that the latter two will
> > affect all operations on the mounted filesystem, not just the initial
> > mount (and maybe not the initial mount at all -- the docs
aren''t quite
> > clear on that).
>
> > Unlike Len, I would not recommend using the "bg" option.
Doing so
> > likely would prevent Puppet from hanging for a long time attempting
> > the mount, but it would also prevent Puppet from correctly managing
> > the resource. Puppet needs to know whether it has succeeded in
> > mounting the remote filesystem. It may also cause you trouble later
> > if the client never does succeed in mounting the filesystem.
>
> > John
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
madAndroid
2011-Nov-15 15:59 UTC
[Puppet Users] Re: NFS clients - puppet hang when mount point
FYI,
We ended up going with a mix of the two suggestions: having
''bg''
specified as one of the mount point options,
and also having the mount resource specify => present instead of =>
mounted, and then we use an exec to force
the remount, constrained by an unless and onlyif:
exec { "remount-storage":
command => "mount -a",
unless => "grep $brand /proc/mounts",
onlyif => "nc -zv $nfs_server 2049 -w 3",
}
The works for us in all use cases, and seems to be the best way
forwards :)
- Andrew
On Nov 11, 11:03 am, madAndroid <andrewsta...@gmail.com>
wrote:> Thank you so much to all who responded - we''ve managed to rewrite
our
> classes to use the mount directive rather (and quite quickly and
> painlessly), and it seems a much better way than we had before (using
> file and exec resources/directives). Still experiencing the timeouts
> during the puppet run when the NFS box goes away (in a dev environment
> during testing the new class structure), but we haven''t tried the
two
> options that Len suggested - will be giving those a go, with the trade-
> offs in mind. Yes, I agree with John that it does feel like more of a
> problem with the way we are implementing the mounts, will give those
> suggestions a try as well.
>
> Thanks again,
> Andrew
>
> On Nov 10, 3:10 pm, jcbollinger <John.Bollin...@stJude.org> wrote:
>
> > On Nov 10, 6:02 am, madAndroid <andrewsta...@gmail.com> wrote:
>
> > > That sounds a little heavy handed ...
>
> > > also, I feel that it would probably stop the mount from happening
at
> > > all?
> > > how would the fstab initiate the nfs mount if it''s not
able to resolve
> > > the address of the nfs server correctly?
> > > unless I''m missing something ..
>
> > > On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote:
>
> > > > OK. This may seem like a bad idea, but it''s a
workaround that has worked
> > > > for me:
> > > > I add the nfs server to the 127.0.0.1 entry of the hosts
file which causes
> > > > NFS to time out pretty immediately. :-\
>
> > > > On Wed, Nov 9, 2011 at 10:00 AM, madAndroid
<andrewsta...@gmail.com> wrote:
> > > > > We''ve only recently discovered that puppet can
manage mount points
> > > > > using the mount directive;
> > > > > however, a short while back we built annfs clientand
server classes
> > > > > without using this resource, and we''ve
encountered a problem where
> > > > > puppet seems to hang when the nfs server is
unavailable.
>
> > > > > Using --debug doesn''t seem to specify exactly
at which point the run
> > > > > is failing, which could steer us in the right direction
around putting
> > > > > something in place in the classes in question.
>
> > > > > Is there anything we can do, short of switching over to
using the
> > > > > mount directive/resource, in order to mitigate the
problem when the
> > > > > nfs server is unavailable? It''s preventing us
from managing other
> > > > > resources on the clients when this happens..
>
> > It all comes down to mount options. I''d recommend you absorb
the
> > material in nfs(5), but options of particular interest include
> > ''retry'', ''retrans'', and
''timeo''. Do notice that the latter two will
> > affect all operations on the mounted filesystem, not just the initial
> > mount (and maybe not the initial mount at all -- the docs
aren''t quite
> > clear on that).
>
> > Unlike Len, I would not recommend using the "bg" option.
Doing so
> > likely would prevent Puppet from hanging for a long time attempting
> > the mount, but it would also prevent Puppet from correctly managing
> > the resource. Puppet needs to know whether it has succeeded in
> > mounting the remote filesystem. It may also cause you trouble later
> > if the client never does succeed in mounting the filesystem.
>
> > John
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.