Puppet users - Dec 2011 - how to link items together -- all happen together?

I''ve found some problems due to the extremely random ordering puppet
does.  It is necessary for some of these items to all happen together, with no
other random resources executed in between.  Is there some way to arrange this
with puppet?

-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 2011-12-08 11:07 , Jo Rhett wrote:
> I''ve found some problems due to the extremely random ordering
puppet
> does.  It is necessary for some of these items to all happen together,
> with no other random resources executed in between.  Is there some way
> to arrange this with puppet?
> 
> -- 
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
> other randomness
> 
Have you tried specifying the dependencies between your resources?

package { ''foo'':
  ensure => installed,
}

file { ''/etc/foo.conf'':
  ensure => file,
  source => ''puppet:///module/foo/foo.conf'',
  require => Package[''foo''],
}

service { ''foo'':
  ensure => running,
  subscribe => File[''/etc/foo/foo.conf''],
Package[''foo''],
}

Both subscribe and require will setup ordering between resources.

-- 
Jacob Helwig
http://technosorcery.net/about/me

It sounds like he wants to not only assue that foo happens before bar,
but that nothing happens between foo and bar. I think the only way to
ensure that is to use stages, and have a separate stage for the two
items that need to be executed together.

On Dec 8, 1:58 pm, Jacob Helwig <ja...@puppetlabs.com>
wrote:
> On 2011-12-08 11:07 , Jo Rhett wrote:
>
> > I''ve found some problems due to the extremely random ordering
puppet
> > does.  It is necessary for some of these items to all happen together,
> > with no other random resources executed in between.  Is there some way
> > to arrange this with puppet?
>
> > --
> > Jo Rhett
> > Net Consonance : consonant endings by net philanthropy, open source
and
> > other randomness
>
> Have you tried specifying the dependencies between your resources?
>
> package { ''foo'':
>   ensure => installed,
>
> }
>
> file { ''/etc/foo.conf'':
>   ensure => file,
>   source => ''puppet:///module/foo/foo.conf'',
>   require => Package[''foo''],
>
> }
>
> service { ''foo'':
>   ensure => running,
>   subscribe => File[''/etc/foo/foo.conf''],
Package[''foo''],
>
> }
>
> Both subscribe and require will setup ordering between resources.
>
> --
> Jacob Helwighttp://technosorcery.net/about/me
>
>  signature.asc
> < 1KViewDownload
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Dec 8, 1:07 pm, Jo Rhett <jrh...@netconsonance.com>
wrote:
> I''ve found some problems due to the extremely random ordering
puppet does.  It is necessary for some of these items to all happen together,
with no other random resources executed in between.

That''s rather unusual.  I''ll discuss in a minute how you might
address
this requirement, but are you sure you really need to do so?  Would
you care to satisfy my curiosity as to why?  Are you also going to
have a problem if some other process (that is, not the Puppet agent)
does any work between?

>  Is there some way to arrange this with puppet?

Puppet''s resource model assumes that each resource is atomic.  If
multiple resources really must all be applied one after another, with
no others in between, then they oughtn''t really to be modeled as
separate resources.  Rather, they are a single composite resource, and
the right way to handle the situation is therefore to write a custom
type (and provider) to handle the situation.

There is also a quick and dirty solution: use run stages.  Assign all
the resources that need to be applied consecutively to one run stage,
and make sure no others are assigned to it.  Or, if you cannot exclude
all other resources (since run stages work at class granularity) then
use explicit resource relationships to push other resources before or
behind the group as needed.  Be warned: doing this is likely to cause
you continuing grief with dependency cycles, even with classes not
unrelated to your resource group.  Also be aware that run stages are
an entry point to several of the problems attending use of
parameterized classes, even if the classes involved are not explicitly
parameterized.

In principle, you could also do this with ordinary resource
relationships.  That would keep you away from parameterized class
syntax, but it would be horendously messy.  I cannot possibly
recommend it except maybe if you have some kind of code generator
writing your manifests for you.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Dec 8, 2011, at 11:58 AM, Jacob Helwig wrote:
> Have you tried specifying the dependencies between your resources?

Yes, extensively.  The problem I am having is that just because A->B->C
happens in order, does not mean that sometimes puppet won''t take
Z,Y,X,V and do them between A->B.  With 26 modules and 2k resources, you can
actually have 4-5 minutes of "other stuff" done between two things. 
There are some things which I would prefer to have happen in tight order -- do
A, then B, then C then go do stuff in other modules.

-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> On Dec 8, 1:07 pm, Jo Rhett <jrh...@netconsonance.com> wrote:
>> I''ve found some problems due to the extremely random ordering
puppet does.  It is necessary for some of these items to all happen together,
with no other random resources executed in between.
On Dec 8, 2011, at 1:55 PM, jcbollinger wrote:
> That''s rather unusual.  I''ll discuss in a minute how you
might address
> this requirement, but are you sure you really need to do so?  Would
> you care to satisfy my curiosity as to why?  Are you also going to
> have a problem if some other process (that is, not the Puppet agent)
> does any work between?
I''m not worried about normal multitasking.  I''m just seeing
problems where puppet does step 1 (which causes a small but acceptable outage
with a service) then steps aside and does dozens of other operations from other
modules, like package security checks, log handling, etc.  Then it comes back to
finish the changes to the service.  What should have been a minute outage (5-10
seconds when they occur in order) to happen with 12-15 minutes in between.

Now, I would prefer to use tags to do this, but because tags are inclusive but
not exclusive I can''t prevent this change from being done during a
normal run.
> Puppet''s resource model assumes that each resource is atomic.  If
> multiple resources really must all be applied one after another, with
> no others in between, then they oughtn''t really to be modeled as
> separate resources.  Rather, they are a single composite resource, and
> the right way to handle the situation is therefore to write a custom
> type (and provider) to handle the situation.
That seems fairly extreme for a simple enough ordering issue.  All we really
need here is the basic idea of a closure.
> There is also a quick and dirty solution: use run stages.  Assign all
> the resources that need to be applied consecutively to one run stage,
> and make sure no others are assigned to it.  Or, if you cannot exclude
> all other resources (since run stages work at class granularity) then
> use explicit resource relationships to push other resources before or
> behind the group as needed.  Be warned: doing this is likely to cause
> you continuing grief with dependency cycles, even with classes not
Bingo. At this point almost every class has dependencies within every other
class which make it impossible.  Mostly shared path names, but sometimes
services and config file references.

-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Dec 10, 12:20 am, Jo Rhett <jrh...@netconsonance.com>
wrote:
> > On Dec 8, 1:07 pm, Jo Rhett <jrh...@netconsonance.com> wrote:
> >> I''ve found some problems due to the extremely random
ordering puppet does.  It is necessary for some of these items to all happen
together, with no other random resources executed in between.
>
> On Dec 8, 2011, at 1:55 PM, jcbollinger wrote:
>
> > That''s rather unusual.  I''ll discuss in a minute how
you might address
> > this requirement, but are you sure you really need to do so?  Would
> > you care to satisfy my curiosity as to why?  Are you also going to
> > have a problem if some other process (that is, not the Puppet agent)
> > does any work between?
>
> I''m not worried about normal multitasking.  I''m just
seeing problems where puppet does step 1 (which causes a small but acceptable
outage with a service) then steps aside and does dozens of other operations from
other modules, like package security checks, log handling, etc.  Then it comes
back to finish the changes to the service.  What should have been a minute
outage (5-10 seconds when they occur in order) to happen with 12-15 minutes in
between.

So you are using one or more Execs in your command grouping?  If you
were relying on the Service resource alone then I don''t see how the
lengthy outage you describe would happen.

> Now, I would prefer to use tags to do this, but because tags are inclusive
but not exclusive I can''t prevent this change from being done during a
normal run.
>
> > Puppet''s resource model assumes that each resource is atomic.
 If
> > multiple resources really must all be applied one after another, with
> > no others in between, then they oughtn''t really to be modeled
as
> > separate resources.  Rather, they are a single composite resource, and
> > the right way to handle the situation is therefore to write a custom
> > type (and provider) to handle the situation.
>
> That seems fairly extreme for a simple enough ordering issue.  All we
really need here is the basic idea of a closure.

I submit that you are looking at it from the wrong perspective and
struggling to make Puppet work contrary to its designed mode of
operation.  It is a basic premise of Puppet''s design that native
resources are atomic as far as Puppet is concerned.  It follows that
the right way to compose an atomic resource is as a native type.
That''s not extreme, it''s the Puppet way.

Furthermore, if I have deduced correctly that your command group
includes one or more Execs, then that is further confirmation that you
are really looking for a native type.  Most individual execs are stand-
ins for for-purpose native resources anyway, but if indeed your group
of resources includes Execs then that''s a siren and flashing light
telling you that you want a native type.

Depending on the exact nature of your problem, however, it may be that
you don''t need to write a custom type.  If you take a high-level view,
then you can interpret the built-in Package resource type as
representing a collection of files and scripts to be applied in a
concerted manner.  That seems at least close to what you are looking
for, so perhaps you can approach your problem by building packages and
having Puppet manage those.

> > There is also a quick and dirty solution: use run stages.  Assign all
> > the resources that need to be applied consecutively to one run stage,
> > and make sure no others are assigned to it.  Or, if you cannot exclude
> > all other resources (since run stages work at class granularity) then
> > use explicit resource relationships to push other resources before or
> > behind the group as needed.  Be warned: doing this is likely to cause
> > you continuing grief with dependency cycles, even with classes not
>
> Bingo. At this point almost every class has dependencies within every other
class which make it impossible.  Mostly shared path names, but sometimes
services and config file references.

If it is truly impossible to do the job with run stages then it is
inconsistent with the context of your overall manifest set.  You need
to change it somehow, such as by altering which resources are involved
or by eliminating some resource relationships.

The bigger picture here is that such a high degree of resource
interrelationship constitutes a modularity problem that will bite you
again if not fixed.  It may even be a direct contributor to your
present problem, rather than merely a complication.  You should work
to minimize cross-class relationships, and especially cross-module
relationships in your manifests.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Dec 9, 2011 at 22:20, Jo Rhett <jrhett@netconsonance.com>
wrote:
>> On Dec 8, 1:07 pm, Jo Rhett <jrh...@netconsonance.com> wrote:
>>> I''ve found some problems due to the extremely random
ordering puppet does.  It is necessary for some of these items to all happen
together, with no other random resources executed in between.
>
> On Dec 8, 2011, at 1:55 PM, jcbollinger wrote:
>> That''s rather unusual.  I''ll discuss in a minute how
you might address
>> this requirement, but are you sure you really need to do so?  Would
>> you care to satisfy my curiosity as to why?  Are you also going to
>> have a problem if some other process (that is, not the Puppet agent)
>> does any work between?
>
> I''m not worried about normal multitasking.  I''m just
seeing problems where puppet does step 1 (which causes a small but acceptable
outage with a service) then steps aside and does dozens of other operations from
other modules, like package security checks, log handling, etc.  Then it comes
back to finish the changes to the service.  What should have been a minute
outage (5-10 seconds when they occur in order) to happen with 12-15 minutes in
between.
So, I want to be clear up front: this is the absolute opposite of a
promise.  I want to understand what people expect here, and there is
*nothing* resembling a hint that we are changing the model of Puppet.

So, Jo, if we wanted to solve this, would it work for you if we
strongly prefer to process resources in the same class at the same
time, rather than separating them over time?

Would you need that to work between multiple classes?  How would you
like that to be specified, or just do it in the general case?

Would it work for you if we treated the relationship arrow as
something that tried to bind as close as possible?

Is there a situation where that wouldn''t work?


Other folks, what do you think, in your environments?

Daniel
-- 
⎋ Puppet Labs Developer – http://puppetlabs.com
♲ Made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Dec 12, 2011 at 11:33:37AM -0800, Daniel Pittman
wrote:
> On Fri, Dec 9, 2011 at 22:20, Jo Rhett <jrhett@netconsonance.com>
wrote:
> >> On Dec 8, 1:07 pm, Jo Rhett <jrh...@netconsonance.com>
wrote:
> >>> I''ve found some problems due to the extremely random
ordering puppet does.  It is necessary for some of these items to all happen
together, with no other random resources executed in between.
> >
> > On Dec 8, 2011, at 1:55 PM, jcbollinger wrote:
> >> That''s rather unusual.  I''ll discuss in a minute
how you might address
> >> this requirement, but are you sure you really need to do so?
 Would
> >> you care to satisfy my curiosity as to why?  Are you also going to
> >> have a problem if some other process (that is, not the Puppet
agent)
> >> does any work between?
> >
> > I''m not worried about normal multitasking.  I''m just
seeing problems where puppet does step 1 (which causes a small but acceptable
outage with a service) then steps aside and does dozens of other operations from
other modules, like package security checks, log handling, etc.  Then it comes
back to finish the changes to the service.  What should have been a minute
outage (5-10 seconds when they occur in order) to happen with 12-15 minutes in
between.
> 
> So, I want to be clear up front: this is the absolute opposite of a
> promise.  I want to understand what people expect here, and there is
> *nothing* resembling a hint that we are changing the model of Puppet.
> 
> So, Jo, if we wanted to solve this, would it work for you if we
> strongly prefer to process resources in the same class at the same
> time, rather than separating them over time?
> 
> Would you need that to work between multiple classes?  How would you
> like that to be specified, or just do it in the general case?
> 
> Would it work for you if we treated the relationship arrow as
> something that tried to bind as close as possible?
> 
> Is there a situation where that wouldn''t work?
Lots of debian packages restart services as the debs are upgraded. That might
de-atomic any process.
 
> Other folks, what do you think, in your environments?
I appreciate the semi-random ordering, because that shows up manifest mistakes.
I also want to know more specifics about what prompted the request.

This conversation sounds like the request for "atomicbefore" and
"atomicrequires", or at least an "atomicdefine". Any
problems would be the user''s responsibility to sort out, much like what
happens when we point a dependency forward across run stages.

(It''s obviously not a bug that we can''t do that, and adding a
bad require/before combo to something crossing the atomic boundary would
sensibly also cause problems. Like "Cannot require into an
atomicdefine" and "Cannot happen between atomically defined
resources".)
 
> Daniel
> -- 
> ⎋ Puppet Labs Developer – http://puppetlabs.com
> ♲ Made with 100 percent post-consumer electrons
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 12/12/11 19:33, Daniel Pittman wrote:
>
> So, I want to be clear up front: this is the absolute opposite of a
> promise.  I want to understand what people expect here, and there is
> *nothing* resembling a hint that we are changing the model of Puppet.
>
> So, Jo, if we wanted to solve this, would it work for you if we
> strongly prefer to process resources in the same class at the same
> time, rather than separating them over time?
>
> Would you need that to work between multiple classes?  How would you
> like that to be specified, or just do it in the general case?
>
> Would it work for you if we treated the relationship arrow as
> something that tried to bind as close as possible?
>
> Is there a situation where that wouldn''t work?
I can''t think of an example where manifests would break if you
tightened
the relationships between resources that wouldn''t have been broken 
before due to unspecified dependencies.
>
> Other folks, what do you think, in your environments?
I don''t need anything like this at the moment, but one real world 
example I can think of is dependent services. I''m thinking years back
to
Tomcat + Apache with mod_ajp - I *think* that if you restarted Tomcat 
any existing mod_ajp connections would be broken so you''d want to 
restart Apache straight after.

Another example could be forcefully clearing a Varnish cache straight 
away when pushing out new files or packages.
>
> Daniel

-- 
Luke Bigum
Information Systems
+44 (0) 20 3192 2520
Luke.Bigum@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN


The information in this e-mail and any attachment is confidential and is
intended only for the named recipient(s). The e-mail may not be disclosed or
used by any person other than the addressee, nor may it be copied in any way. If
you are not a named recipient please notify the sender immediately and delete
any copies of this message. Any unauthorized copying, disclosure or distribution
of the material in this e-mail is strictly forbidden. Any view or opinions
presented are solely those of the author and do not necessarily represent those
of the company.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.