Puppet users - Jul 2012 - The Puppet Way to handle slow resources? (newbie)

Hi

I have a resource that is slow to come into service and I have other resources
that are dependent on the service in order to be configured. I''ve been
searching for the "puppet way" to do this, but all the options
I''ve found don''t seem optimal, so I''m assuming there
is a better way that I''m missing.

I have several instances, so I want puppet to kick them off in parallel. And I
want the reliant services to be configured as soon as possible when the slow
services are ready

So I don''t really want to use long timeout/multiple retry because I
believe that won''t be executed in parallel - and anyway, it''s
not a retry I want, it''s a wait

I don''t want to have Mcollective/AMQP just for this one problem, it
feels like a sledgehammer to crack a nut

I don''t want some external process to repeatedly kick puppet repeatedly
until the dependency is resolved and the reliant service can be configured

I''m imagining something like a semaphore, puppet sleeps and wakes up
when the semaphore is flipped (the slow resource is available)

Sorry if this is a dumb question, first post etc ...

Thanks

Dave
-- 
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/zZfbu37DrsQJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

(inline)

On Tue, Jul 10, 2012 at 11:39:58AM -0700, Dave Anderson
wrote:
> Hi
> 
> I have a resource that is slow to come into service and I have other
resources that are dependent on the service in order to be configured.
I''ve been searching for the "puppet way" to do this, but all
the options I''ve found don''t seem optimal, so I''m
assuming there is a better way that I''m missing.
> 
> I have several instances, so I want puppet to kick them off in parallel.
And I want the reliant services to be configured as soon as possible when the
slow services are ready
> 
> So I don''t really want to use long timeout/multiple retry because
I believe that won''t be executed in parallel - and anyway,
it''s not a retry I want, it''s a wait
> 
> I don''t want to have Mcollective/AMQP just for this one problem,
it feels like a sledgehammer to crack a nut

I don''t understand your issue with mcollective. You have an
orchestration issue but don''t want to use the puppet-ecosystem
orchestration tool? This is the part that confuses me, especially since this
won''t be your last orchestration requirement.

Speaking from ignorance, maybe look into exported resources and have the slow
services export something.

http://docs.puppetlabs.com/guides/exported_resources.html

> I don''t want some external process to repeatedly kick puppet
repeatedly until the dependency is resolved and the reliant service can be
configured
> 
> I''m imagining something like a semaphore, puppet sleeps and wakes
up when the semaphore is flipped (the slow resource is available)
> 
> Sorry if this is a dumb question, first post etc ...
> 
> Thanks
> 
> Dave
> -- 
> This email and any files transmitted with it are confidential and intended 
> solely for the use of the individual or entity to whom they are addressed. 
> If you have received this email in error please notify the system manager. 
> This message contains confidential information and is intended only for the
> individual named. If you are not the named addressee you should not 
> disseminate, distribute or copy this e-mail.
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/zZfbu37DrsQJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 07/10/2012 08:52 PM, Christopher Wood wrote:
> I don''t understand your issue with mcollective. You have an
orchestration issue but don''t want to use the puppet-ecosystem
orchestration tool? This is the part that confuses me, especially since this
won''t be your last orchestration requirement.
I agree with the OP that there is quite some overhead involved and there
should be a good reason for implementing the requried infrastructure.

What''s not clear to me is: Should all this happen during the same agent
run?

I disbelieve puppet can handle this. Puppet really only cares about
this: Is service A running? If not, start it. If starting is implemented
synchronously, i.e. the start operation does not finish before the
service is ready, puppet will wait for it to get ready. Nothing works in
parallel in this scenario.

On the other hand, if the start operation finishes immediately, puppet
will consider the start to be complete and apply all depending resources.

So if you wish your start procedures to work asynchronously, you will
probably have to design your depending services accordingly, i.e. when
you start services B and C, their initscritps (or whatever) need to
linger in the background and wait for service A to be completely
initialized before actually performing their true startup operations.

HTH,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Dave,

This is definitely not a dumb question. I wrote a post on what I
thought a puppet semaphore system should do quite some time ago.
Unfortunately, you don''t really want to hang an instance until the
semaphore flips so you end up having to wait through puppet runs.

If you''ve got a solid SSH infrastructure working, then I would suggest
looking at something like Capistrano or Func to give you that one-off,
lightweight punch that you need.

The big thing to watch out for is that you need to make sure that your
manifests work properly even if they don''t run in order since your
nodes will be calling in over time.

One way of doing this is to use a custom fact and drop a file onto the
system when you''re ready for that extra bit to run or have it based on
hitting the remote system to see if your target service is actually
functional.

Ideally though, the startup scripts for the different services would
be smart enough to figure out if the environment is ready for them to
run. The puppet can do what it does best and the services can merrily
go on their way trying to come up until they succeed (or ultimately
fail).

Trevor

On Tue, Jul 10, 2012 at 2:39 PM, Dave Anderson <Dave.Anderson@ammeon.com>
wrote:
> Hi
>
> I have a resource that is slow to come into service and I have other
resources that are dependent on the service in order to be configured.
I''ve been searching for the "puppet way" to do this, but all
the options I''ve found don''t seem optimal, so I''m
assuming there is a better way that I''m missing.
>
> I have several instances, so I want puppet to kick them off in parallel.
And I want the reliant services to be configured as soon as possible when the
slow services are ready
>
> So I don''t really want to use long timeout/multiple retry because
I believe that won''t be executed in parallel - and anyway,
it''s not a retry I want, it''s a wait
>
> I don''t want to have Mcollective/AMQP just for this one problem,
it feels like a sledgehammer to crack a nut
>
> I don''t want some external process to repeatedly kick puppet
repeatedly until the dependency is resolved and the reliant service can be
configured
>
> I''m imagining something like a semaphore, puppet sleeps and wakes
up when the semaphore is flipped (the slow resource is available)
>
> Sorry if this is a dumb question, first post etc ...
>
> Thanks
>
> Dave
> --
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager.
> This message contains confidential information and is intended only for the
> individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail.
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/zZfbu37DrsQJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaughan@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

If I understand the OPs question correctly, he has an ordering problem:
some services aren''t waiting for other services to come online before
starting themselves?

If that''s the case, you handle this with dependencies and chaining,
e.g.
service { fast_service:  require =>
Service[''slow_service'']...}

If the real issue is that the initial startup of a service involves a long
initialization process, then you need to insert some kind of test in
between the two services that verifies the slow service is truly up and
running, e.g.
exec { slow_service_check:  require =>
Service[''slow_service''], command =>
"process that will wait for the slow service"...}
service { fast_service:  require =>
Exec[''slow_service_check'']...}

If the real issue is that the slow service takes longer than puppet_timeout
to complete its startup, then you have an insurmountable problem without
either MCollective or multiple puppet runs.

One way to address this with multiple puppet runs involves having a
progression of environments (assuming this happens once on provisioning and
never again).  Essentially, you have systems boot up into a
"provisioning"
environment, where puppet runs, say, every 5 minutes.  Then you have your
ENC set up such that when a provisioning run completes with no changes, it
switches to a "longterm" environment, which is generally much more
lightweight and runs puppet on a slower schedule (e.g. once a day outside
of business hours).  This methodology can make the majority of your puppet
runs very lightweight and fast (which can matter if you are paying for CPU
cycles), but requires that everyone be good citizens who never change a
system without using Puppet.

On Wed, Jul 11, 2012 at 7:12 AM, Trevor Vaughan
<tvaughan@onyxpoint.com>wrote:
> Hi Dave,
>
> This is definitely not a dumb question. I wrote a post on what I
> thought a puppet semaphore system should do quite some time ago.
> Unfortunately, you don''t really want to hang an instance until the
> semaphore flips so you end up having to wait through puppet runs.
>
> If you''ve got a solid SSH infrastructure working, then I would
suggest
> looking at something like Capistrano or Func to give you that one-off,
> lightweight punch that you need.
>
> The big thing to watch out for is that you need to make sure that your
> manifests work properly even if they don''t run in order since your
> nodes will be calling in over time.
>
> One way of doing this is to use a custom fact and drop a file onto the
> system when you''re ready for that extra bit to run or have it
based on
> hitting the remote system to see if your target service is actually
> functional.
>
> Ideally though, the startup scripts for the different services would
> be smart enough to figure out if the environment is ready for them to
> run. The puppet can do what it does best and the services can merrily
> go on their way trying to come up until they succeed (or ultimately
> fail).
>
> Trevor
>
> On Tue, Jul 10, 2012 at 2:39 PM, Dave Anderson
<Dave.Anderson@ammeon.com>
> wrote:
> > Hi
> >
> > I have a resource that is slow to come into service and I have other
> resources that are dependent on the service in order to be configured.
I''ve
> been searching for the "puppet way" to do this, but all the
options I''ve
> found don''t seem optimal, so I''m assuming there is a
better way that I''m
> missing.
> >
> > I have several instances, so I want puppet to kick them off in
parallel.
> And I want the reliant services to be configured as soon as possible when
> the slow services are ready
> >
> > So I don''t really want to use long timeout/multiple retry
because I
> believe that won''t be executed in parallel - and anyway,
it''s not a retry I
> want, it''s a wait
> >
> > I don''t want to have Mcollective/AMQP just for this one
problem, it
> feels like a sledgehammer to crack a nut
> >
> > I don''t want some external process to repeatedly kick puppet
repeatedly
> until the dependency is resolved and the reliant service can be configured
> >
> > I''m imagining something like a semaphore, puppet sleeps and
wakes up
> when the semaphore is flipped (the slow resource is available)
> >
> > Sorry if this is a dumb question, first post etc ...
> >
> > Thanks
> >
> > Dave
> > --
> > This email and any files transmitted with it are confidential and
> intended
> > solely for the use of the individual or entity to whom they are
> addressed.
> > If you have received this email in error please notify the system
> manager.
> > This message contains confidential information and is intended only
for
> the
> > individual named. If you are not the named addressee you should not
> > disseminate, distribute or copy this e-mail.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> > To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/zZfbu37DrsQJ.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
> >
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699
> tvaughan@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tuesday, July 10, 2012 1:39:58 PM UTC-5, Dave Anderson
wrote:
>
> Hi 
>
> I have a resource that is slow to come into service and I have other 
> resources that are dependent on the service in order to be configured.
I''ve
> been searching for the "puppet way" to do this, but all the
options I''ve
> found don''t seem optimal, so I''m assuming there is a
better way that I''m
> missing. 
>
> I have several instances, so I want puppet to kick them off in parallel. 
> And I want the reliant services to be configured as soon as possible when 
> the slow services are ready 
>
The Puppet agent is single-threaded, and it actively prevents multiple 
(Puppet) instances from running simultaneously.  The only way you can start 
multiple resources in parallel is if Puppet is used only to trigger an 
asynchronous startup.  Although that''s doable, it''s not very
consistent
with the Puppet resource model, because Puppet relies on knowing whether 
its attempts to synchronize a resource succeed.  To put it another way, 
Puppet resource synchronization is (assumed to be) atomic when viewed at 
the DSL level.

If you can model your group of slow resources so that together they look 
like a single resource to Puppet, then perhaps it would work out.  Such a 
construction cannot be done in Puppet DSL, but you can either write a 
script that you will wrap an an Exec resource, or you can have a go at 
writing a custom resource type.  Possibly you could use Puppet''s Ruby
DSL,
but my intuition suggests that might be dangerous for this kind of task.
 
> So I don''t really want to use long timeout/multiple retry because
I
> believe that won''t be executed in parallel - and anyway,
it''s not a retry I
> want, it''s a wait 
>
Good, because Puppet assumes that waiting is what happens.  Or bad, because 
Puppet isn''t going to sync multiple (Puppet) resources on the same node
in
parallel no matter what you do. 


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/q5xaacPWRIUJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Everyone,

First, thanks for all the great feedback!

Chris, I''ll take a look at exported resources. I don''t have a
problem with
MCollective per se, I just don''t want to add a bunch of other software
if
there''s a "native puppet" way to solve the problem. From what
I''ve seen,
Puppet itself isn''t supposed to solve this problem, MCollective is.

Felix, yes my aim was to have everything complete in one Puppet run. My 
next-best option would be that Puppet would run again as soon as the 
slow-starting-service is ready - as opposed to waiting until the next 
scheduled run.

Trevor, I''ll take a look at Capistrano (that was mentioned at the
recent
Dublin Puppet Camp) but I have the same reservations as I would have with 
MCollective. Func isn''t possible for us, due to arcane/licensing
issues.

Brian, thanks for your comments too. I will definitely model the service 
dependencies.

My "plan A" right now is that when the slow-running service is up and 
running it will tell Puppet to run. I haven''t really thought about how
this
would work for multiple instances of the slow-service, I''m pretty sure 
that''s not a hard problem to solve though.

Cheers everyone!

Dave


On Tuesday, 10 July 2012 19:39:58 UTC+1, Dave Anderson
wrote:
>
> Hi 
>
> I have a resource that is slow to come into service and I have other 
> resources that are dependent on the service in order to be configured.
I''ve
> been searching for the "puppet way" to do this, but all the
options I''ve
> found don''t seem optimal, so I''m assuming there is a
better way that I''m
> missing. 
>
> I have several instances, so I want puppet to kick them off in parallel. 
> And I want the reliant services to be configured as soon as possible when 
> the slow services are ready 
>
> So I don''t really want to use long timeout/multiple retry because
I
> believe that won''t be executed in parallel - and anyway,
it''s not a retry I
> want, it''s a wait 
>
> I don''t want to have Mcollective/AMQP just for this one problem,
it feels
> like a sledgehammer to crack a nut 
>
> I don''t want some external process to repeatedly kick puppet
repeatedly
> until the dependency is resolved and the reliant service can be configured 
>
> I''m imagining something like a semaphore, puppet sleeps and wakes
up when
> the semaphore is flipped (the slow resource is available) 
>
> Sorry if this is a dumb question, first post etc ... 
>
> Thanks 
>
> Dave 
> -- 
> This email and any files transmitted with it are confidential and intended 
> solely for the use of the individual or entity to whom they are addressed. 
> If you have received this email in error please notify the system manager. 
> This message contains confidential information and is intended only for 
> the 
> individual named. If you are not the named addressee you should not 
> disseminate, distribute or copy this e-mail. 
>
>
-- 
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/pSQNRuLUHgAJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

In regard to: [Puppet Users] Re: The Puppet Way to handle slow resources?...:
> Chris, I''ll take a look at exported resources. I don''t
have a problem with
> MCollective per se, I just don''t want to add a bunch of other
software if
> there''s a "native puppet" way to solve the problem. From
what I''ve seen,
> Puppet itself isn''t supposed to solve this problem, MCollective
is.
Agreed.
> My "plan A" right now is that when the slow-running service is up
and
> running it will tell Puppet to run. I haven''t really thought about
how this
> would work for multiple instances of the slow-service, I''m pretty
sure
> that''s not a hard problem to solve though.
I''ve only partially followed this thread so I don''t know if
someone else
has already suggested this, but if the real issue is that the interaction
between software, init script, and puppet isn''t working correctly, then
why not have puppet manage and use a "wrapper" init script?  You keep
the
init script that came with the software, but instead of having puppet use
that for start/stop/status, you write your own "local-service" or
"mycompany-service" init script and have that script call the original
script and augment the logic in start/stop/status/whatever to do whatever
is needed to work correctly with puppet.

Tim
-- 
Tim Mooney                                             Tim.Mooney@ndsu.edu
Enterprise Computing & Infrastructure                  701-231-1076 (Voice)
Room 242-J6, IACC Building                             701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

That sounds like a very elegant solution! 
I''m thinking that this pattern could be useful for others, not just me.
I''ll start a separate thread asking for a location to publish and
critique recipes.
/d
-- 
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/10e0e6HN-xsJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.