Puppet users - Feb 2010 - Declarative style and processes

Hi,

I know puppet allow me to describe the target state of my system and 
changes the state automatically to that description.

In some situations you want to describe a process how things should go 
maybe dependent on other systems where the order is important and maybe 
you need some checks after doing a step and maybe has to wait for other 
systems. Think about a process like a network wide transition, where one 
system do something, wait for some events from other systems and continues 
the process or decide something on that events - same on other systems.

What I want is to program network wide jobs (not one job that runs on the 
all hosts - you see the difference?).

What I like in puppet is, that puppet uses its own communication channel 
besides ssh. Tools like capistrano don''t do that, so you need SSH
access
(that means you need in many situations at least a SSH gateway - I
don''t
like it cause of an extra service on an extra host open to the internet). 
And I would of course use my host definitions from puppet (what I could 
achive, if I use something like LDAP to store my host definitions and use 
it side by side with puppet instead through puppet).

Is there a tool outside where I can use puppet descriptions to program 
such jobs? Or is there a tool outside you know, that allows me to program 
network wide jobs without puppet where I could built in the usage of my 
LDAP tree?

regards
Oli

Oliver Schad wrote:
> Hi,
>
> I know puppet allow me to describe the target state of my system and 
> changes the state automatically to that description.
>
> In some situations you want to describe a process how things should go 
> maybe dependent on other systems where the order is important and maybe 
> you need some checks after doing a step and maybe has to wait for other 
> systems. Think about a process like a network wide transition, where one 
> system do something, wait for some events from other systems and continues 
> the process or decide something on that events - same on other systems.
>
> What I want is to program network wide jobs (not one job that runs on the 
> all hosts - you see the difference?).
>
> What I like in puppet is, that puppet uses its own communication channel 
> besides ssh. Tools like capistrano don''t do that, so you need SSH
access
> (that means you need in many situations at least a SSH gateway - I
don''t
> like it cause of an extra service on an extra host open to the internet). 
> And I would of course use my host definitions from puppet (what I could 
> achive, if I use something like LDAP to store my host definitions and use 
> it side by side with puppet instead through puppet).
>
> Is there a tool outside where I can use puppet descriptions to program 
> such jobs? Or is there a tool outside you know, that allows me to program 
> network wide jobs without puppet where I could built in the usage of my 
> LDAP tree?
>
>   
Hi Oliver,

There have been various things like this, though I think what I really 
would like to do is understand what the types of use cases are for these 
various things, so we can figure out how to teach puppet to do them best 
in a way that fits in well with puppet.  

For instance, this is pretty much why I created Func a while back -- for 
the one-off tasks that weren''t Puppety -- though Func has some major 
limitations (namely, a very weak module library, so it is mainly used to 
just run shell commands and replace SSH loops).   MCollective is doing 
similar things as Func, though with a message bus.   (Aside:  having 
Puppet optionally grok message buses is something we should probably 
look into).

There seems to be two different classes of things we''d like to do:

(A)   Have the orders for a recipe wait until a condition on another 
host meets some certain state... for example, installation of a service 
requires that first a database setup is fulfilled on another host.

(B)   Be able to run tasks that are not neccessarily declarative, such 
as "power off these machines now / network go into emergency off mode 
now / etc".   These kinds of things aren''t exactly what you would
store
in the recipe, but puppetd is well suited to enable these sort of tasks.

Do you have other specific examples?

I think building something like this into the core makes good sense, so 
that you don''t have to set up yet another program.

Can you share more examples of what you''d want to do in these kind of 
cases so we can talk about it?   Right now I''m familiar with the cases 
of "shut this off", or perhaps creating a build system out of the
tool,
though I''d like to know more.

I also see the case of "this can''t go forward until this other
remote
condition is true" as being something we''ll want to make a lot
easier...
I think we understand that one better and can do that fairly easily ... 
possible use cases include load balancer setup, database setup, etc.   

What do you think?

--Michael


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 2, 10:38 am, Michael DeHaan <mich...@reductivelabs.com>
wrote:
> What do you think?
Please forgive me, but I am apprehensive about some of those ideas.
One of the reasons Puppet is great is that it is organized around a
compact goal and consistent operating and design principles.   Its
goal is to achieve and maintain system state, and among its key
operating principles is "Puppet chooses how to do it."  I am concerned
that Puppet will become less reliable, harder to use, and harder to
develop and maintain if it is extended in directions that are
inconsistent with its core goals, operating principles, and design
principles.

It seems a natural extension to enable Puppet to manage the collective
state of an entire network, which I hope you find a fair
characterization of your class (A).  I think there are some
significant practical problems with doing that, however, large among
them that of assessing the collective state of a whole network in the
first place.  The framework extensions needed to support this
extension could easily dwarf all the rest of Puppet.  Looking at it
another way, a generic solution to this class of problems would
essentially be a business process orchestration framework for system
configuration.  Google "business process orchestration" if you
don''t
appreciate the size and complexity of systems that do that.

As for your class (B), to the extent Puppet doesn''t already support it
via Exec, just no.  At the risk of being pedantic, asking to "Be able
to run tasks [...]" of *any* kind via Puppet already runs counter to
the Puppet paradigm.  The Exec type notwithstanding, Puppet is not a
task runner.  I don''t think Puppet is the right tool for that job, and
I would prefer that its operating principles not be compromised to
make it the right tool.

With all that said, if you think puppetd provides a good foundation
for some of these things then nothing prevents you from building on
it.  Puppet is fairly easy to extend with new resource types, and
nothing prevents you from hooking puppetd up to a process
orchestration suite.  I am sure you would have an interested audience
for such work.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I think some really interesting things could be done around a much more
flexible require => infrastructure that would allow various things such as
global requires, requires that trigger tests in some fashion on other hosts.
 I could conceive of some kind of functionality whereby test{} exists in
modules and provides a method by which you can check some functionality is
working.  You would then be able to trigger a test{} on another host and get
back true/false with which puppet could decide if it should skip some action
on the original node.   I guess there are many more flexible and undoubtedly
better ways to do this but it would seem to be closest to the puppet way
that would allow you to test and rely on other functionality being in place.

On Tue, Feb 2, 2010 at 11:38 AM, Michael DeHaan
<michael@reductivelabs.com>wrote:
> Oliver Schad wrote:
>
>> Hi,
>>
>> I know puppet allow me to describe the target state of my system and
>> changes the state automatically to that description.
>>
>> In some situations you want to describe a process how things should go
>> maybe dependent on other systems where the order is important and maybe
you
>> need some checks after doing a step and maybe has to wait for other
systems.
>> Think about a process like a network wide transition, where one system
do
>> something, wait for some events from other systems and continues the
process
>> or decide something on that events - same on other systems.
>>
>> What I want is to program network wide jobs (not one job that runs on
the
>> all hosts - you see the difference?).
>>
>> What I like in puppet is, that puppet uses its own communication
channel
>> besides ssh. Tools like capistrano don''t do that, so you need
SSH access
>> (that means you need in many situations at least a SSH gateway - I
don''t
>> like it cause of an extra service on an extra host open to the
internet).
>> And I would of course use my host definitions from puppet (what I could
>> achive, if I use something like LDAP to store my host definitions and
use it
>> side by side with puppet instead through puppet).
>>
>> Is there a tool outside where I can use puppet descriptions to program
>> such jobs? Or is there a tool outside you know, that allows me to
program
>> network wide jobs without puppet where I could built in the usage of my
LDAP
>> tree?
>>
>>
>>
> Hi Oliver,
>
> There have been various things like this, though I think what I really
> would like to do is understand what the types of use cases are for these
> various things, so we can figure out how to teach puppet to do them best in
> a way that fits in well with puppet.
> For instance, this is pretty much why I created Func a while back -- for
> the one-off tasks that weren''t Puppety -- though Func has some
major
> limitations (namely, a very weak module library, so it is mainly used to
> just run shell commands and replace SSH loops).   MCollective is doing
> similar things as Func, though with a message bus.   (Aside:  having Puppet
> optionally grok message buses is something we should probably look into).
>
> There seems to be two different classes of things we''d like to do:
>
> (A)   Have the orders for a recipe wait until a condition on another host
> meets some certain state... for example, installation of a service requires
> that first a database setup is fulfilled on another host.
>
> (B)   Be able to run tasks that are not neccessarily declarative, such as
> "power off these machines now / network go into emergency off mode now
/
> etc".   These kinds of things aren''t exactly what you would
store in the
> recipe, but puppetd is well suited to enable these sort of tasks.
>
> Do you have other specific examples?
>
> I think building something like this into the core makes good sense, so
> that you don''t have to set up yet another program.
>
> Can you share more examples of what you''d want to do in these kind
of cases
> so we can talk about it?   Right now I''m familiar with the cases
of "shut
> this off", or perhaps creating a build system out of the tool, though
I''d
> like to know more.
>
> I also see the case of "this can''t go forward until this
other remote
> condition is true" as being something we''ll want to make a
lot easier... I
> think we understand that one better and can do that fairly easily ...
> possible use cases include load balancer setup, database setup, etc.
> What do you think?
>
> --Michael
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Ashley Penney wrote:
> I think some really interesting things could be done around a much 
> more flexible require => infrastructure that would allow various 
> things such as global requires, requires that trigger tests in some 
> fashion on other hosts.  I could conceive of some kind of 
> functionality whereby test{} exists in modules and provides a method 
> by which you can check some functionality is working.  You would then 
> be able to trigger a test{} on another host and get back true/false 
> with which puppet could decide if it should skip some action on the 
> original node.   I guess there are many more flexible and undoubtedly 
> better ways to do this but it would seem to be closest to the puppet 
> way that would allow you to test and rely on other functionality being 
> in place.
I like the idea of being able to support arbitrary tests for 
redlight-greenlight type status (could be very nice for dashboard, 
too... lightweight monitoring!).

I''m not entirely sure yet whether that requires requires [sic] though, 
or tests should cause automated behavior.
Testing could just be a separately triggered system.

Teyo has pointed out before there''s something a bit odd about using 
Puppet to test Puppet, but in this case, we''re using puppet''s
existing
(and solved) secure communications system to extend it into monitoring 
territory.

I like this.

If you''ve got more info on what tests might trigger, I''d be
interested
in hearing that.







-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger wrote:
> On Feb 2, 10:38 am, Michael DeHaan <mich...@reductivelabs.com> wrote:
>   
>> What do you think?
>>     
>
> Please forgive me, but I am apprehensive about some of those ideas.
> One of the reasons Puppet is great is that it is organized around a
> compact goal and consistent operating and design principles.   Its
> goal is to achieve and maintain system state, and among its key
> operating principles is "Puppet chooses how to do it."  I am
concerned
> that Puppet will become less reliable, harder to use, and harder to
> develop and maintain if it is extended in directions that are
> inconsistent with its core goals, operating principles, and design
> principles.
>   
*nod*

Not proposing that.   I''m just curious as to what tasks fit outside the
model, per se.

Those things are kind of the outliers, the things that a declarative 
system can''t do.  Yes, you''re right, the system is
good because it''s a system, and we don''t want to encourage
Puppet to
ever reduce things to being an SSH loop replacement.

However, occasionally... rarely... you want that.     BIOS upgrades, 
system reboots, power management, kicking off a ''koan
--replace-self'' on
a node (if you speak Cobbler), etc.   These seem like simple things to do.
> It seems a natural extension to enable Puppet to manage the collective
> state of an entire network, which I hope you find a fair
> characterization of your class (A).  I think there are some
> significant practical problems with doing that, however, large among
> them that of assessing the collective state of a whole network in the
> first place.  The framework extensions needed to support this
> extension could easily dwarf all the rest of Puppet.  Looking at it
> another way, a generic solution to this class of problems would
> essentially be a business process orchestration framework for system
> configuration.  Google "business process orchestration" if you
don''t
> appreciate the size and complexity of systems that do that.
>   
Different problem :)

Next thing you know we''ll need a rules engine and an artificial 
intelligence.

Yeah, we don''t want to go there.   But multinode deployment?  Yes, we 
want to go there.

Keeping it simple is important.
> As for your class (B), to the extent Puppet doesn''t already
support it
> via Exec, just no.  At the risk of being pedantic, asking to "Be able
> to run tasks [...]" of *any* kind via Puppet already runs counter to
> the Puppet paradigm.  The Exec type notwithstanding, Puppet is not a
> task runner.  I don''t think Puppet is the right tool for that job,
and
> I would prefer that its operating principles not be compromised to
> make it the right tool.
>   
Puppet is a tool, yes, but also think of the greater potential ecosystem.

puppetd.  puppetca.    These are installed and in-place.  With 
dashboard, we will have a nice console.
I think there is room to do more.

If you''re worried about bloatware, I assure you, it''s not in
my DNA.
It''s not in Reductive''s DNA... puppet
and it''s model will be the core around what we''ll do.  
Always.
> With all that said, if you think puppetd provides a good foundation
> for some of these things then nothing prevents you from building on
> it.  Puppet is fairly easy to extend with new resource types, and
> nothing prevents you from hooking puppetd up to a process
> orchestration suite.  I am sure you would have an interested audience
> for such work.
>
>   
Yeah, part of the reason of discussing it here.   If it proves *not* 
interesting, we''re less likely to go down those
roads.

Which makes it interesting to hear what orchestration problems, if any, 
people are having trying to do things with the existing model today.  
Push actions aren''t neccessarily the solution to those --
that''s a
seperate problem -- but probably one we also may want to touch upon.   
Multi-machine deployments are closer to the orchestration end, and I 
think there are simple logical extensions to teach Puppet about those, 
without changing the way the model works -- and we definitely like the 
way the model works :)

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Wouldnt this tie up with things like Puppet+Capistrano, Puppet+ControlTier,
and such, that was pointed to me when asking about application deployment?
------------------------------

Jesús Couto F.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Feb 3, 2010 at 9:32 AM, Jesús Couto <jesus.couto@gmail.com> wrote:
> Wouldnt this tie up with things like Puppet+Capistrano, Puppet+ControlTier,
> and such, that was pointed to me when asking about application deployment?
>

If by "tie up" you mean being able to do some of the same things,
indeed
yes.

However I still think "workflow" is different than push actions.   We
don''t
want to be like Capistrano/Fabric, we want to model the steps and the
order.    For "do things now" that''s where things like
puppetrun can come
in.

--Michael




> ------------------------------
>
> Jesús Couto F.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> Next thing you know we''ll need a rules engine and an artificial
> intelligence.
>
> Yeah, we don''t want to go there.   But multinode deployment?  Yes,
we
> want to go there.
I think that''s a great idea in principle, with several potential
pitfalls.  Do you have specific ideas about what capabilities should
be added to Puppet to support it?  For instance, are you thinking of a
new tool for this, or of adding to puppetd?  If a new tool, then how
would it interact with puppetd, if at all?
> Puppet is a tool, yes, but also think of the greater potential ecosystem.
[...]
> I think there is room to do more.
[...]
> puppet
> and it''s model will be the core around what we''ll do.  
Always.
That''s encouraging, thanks.  I guess part of what I''m trying
to
grapple with is where some of the new functionality will fit in the
Puppet ecosystem.  I am undoubtedly also worried about some issues
that really are much more Reductive''s business than mine, but I care
because I love Puppet and I want it to keep working for me.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Feb 3, 2010 at 11:27 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
> > Next thing you know we''ll need a rules engine and an
artificial
> > intelligence.
> >
> > Yeah, we don''t want to go there.   But multinode deployment? 
Yes, we
> > want to go there.
>
> I think that''s a great idea in principle, with several potential
> pitfalls.  Do you have specific ideas about what capabilities should
> be added to Puppet to support it?  For instance, are you thinking of a
> new tool for this, or of adding to puppetd?  If a new tool, then how
> would it interact with puppetd, if at all?
>

Actually I think it can be done by leveraging an improved storeconfigs and
the existing language.

Essentially it says "this step can''t continue until this remote
action,
reschedule me later".

Though that''s conjecture... it may be quite different... but multi-node
deployment and pushed-based
actions are two seperate things, it is important to realize they are
completely orthogonal.

I''m confident we can avoid pitfalls, but there will be more discussion
on
this when that starts
going :)

>
> > Puppet is a tool, yes, but also think of the greater potential
ecosystem.
> [...]
> > I think there is room to do more.
> [...]
> > puppet
> > and it''s model will be the core around what we''ll
do.   Always.
>
> That''s encouraging, thanks.  I guess part of what I''m
trying to
> grapple with is where some of the new functionality will fit in the
> Puppet ecosystem.  I am undoubtedly also worried about some issues
> that really are much more Reductive''s business than mine, but I
care
> because I love Puppet and I want it to keep working for me.
>
>
We care about keeping it working for you very much too.    I don''t want
to
see new daemons injected or the model
transmogrified to where people don''t like it... it works because it is
designed very well already, for sure.

>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Am Tuesday 02 February 2010 schrieb mir Michael DeHaan:
> Oliver Schad wrote:
> > Is there a tool outside where I can use puppet descriptions to
> > program such jobs? Or is there a tool outside you know, that allows
> > me to program network wide jobs without puppet where I could built in
> > the usage of my LDAP tree?
> 
> 
> There have been various things like this, though I think what I really
> would like to do is understand what the types of use cases are for
>  these various things, so we can figure out how to teach puppet to do
>  them best in a way that fits in well with puppet.
I had a look at mcollective and maybe this could be a way for me. Thanks 
for your pointer.
>  used to just run shell commands and replace SSH loops).   MCollective
>  is doing similar things as Func, though with a message bus.   (Aside: 
>  having Puppet optionally grok message buses is something we should
>  probably look into).
I think that messaging between systems is strongly needed for system 
automatisation.
 
> There seems to be two different classes of things we''d like to do:
> 
> (A)   Have the orders for a recipe wait until a condition on another
> host meets some certain state... for example, installation of a service
> requires that first a database setup is fulfilled on another host.
> 
> (B)   Be able to run tasks that are not neccessarily declarative, such
> as "power off these machines now / network go into emergency off mode
> now / etc".   These kinds of things aren''t exactly what you
would store
> in the recipe, but puppetd is well suited to enable these sort of
>  tasks.
> 
> Do you have other specific examples?
I see a network as one system with certain dynamics. On dynamic actions 
you need dynamic reactions, some of them you know now, some not.

Assume you have a farm with webservers and no central storage. Now you 
declare that you need those vhosts but you can''t declare on which host
you
need those virtual machine because it''s purly dynamic. The decision is 
based on the load of the machines.

If a machine has a too high load you have to move one or more vhosts to a 
free host but maybe this decision is in one hour not valid any more. 
Moving a vhost means to copy data and change a configuration on the load 
balancer and the specific host (maybe every webserver has a configuration 
for all vhosts and only the load balancer has to be reconfigured).

That is only one example. If I want to reinstall a system I want maybe to 
use a PXE server. If I want to make it secure, I have to reconfigure the 
switch so my host is alone with the PXE server in one subnet. Then install 
it and change the switch configuration again to the old state.

regards
Oli

On Feb 3, 3:08 pm, Michael DeHaan <mich...@reductivelabs.com>
wrote:
> Actually I think it can be done by leveraging an improved storeconfigs and
> the existing language.
I was wondering whether storeconfigs would come up.  The problem I see
with that is storeconfigs, as I understand them, record the managed
resource states each host *should* have, not necessarily those each
one *does* have.  What happens to my multinode deployment if at some
point in the middle, Puppet fails to apply a necessary resource state?
> Essentially it says "this step can''t continue until this
remote action,
> reschedule me later".
Wouldn''t a more Puppetly statement be "this state must not be
applied
until that remote state is reached, try again later"?  The specific
words matter, inasmuch as they shape everyone''s thinking about how the
system works.
> Though that''s conjecture... it may be quite different... but
multi-node
> deployment and pushed-based
> actions are two seperate things, it is important to realize they are
> completely orthogonal.
Absolutely so, and that connects back to your earlier comments about
message buses.  Messaging is the lifeblood of any kind of
orchestration technology, but there are a variety of message exchange
patterns, message implementations, and messaging options, many
combinations of which are viable.  Some could reasonably be
characterized as "push", but many others couldn''t.

Any way around, there is a class of messages needed for this that
Puppet does not currently have: messages from clients advertising the
individual or collective state of their managed resources.  The
absolute simplest form would be a reply to the Puppetmaster indicating
whether a catalog was successfully applied, but finer-grained state
messages would be much more flexible.  These would address the problem
of discrepancy between supposed and actual state.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Feb 4, 2010 at 9:58 AM, jcbollinger <John.Bollinger@stjude.org>
wrote:
>
>
> On Feb 3, 3:08 pm, Michael DeHaan <mich...@reductivelabs.com> wrote:
>> Actually I think it can be done by leveraging an improved storeconfigs
and
>> the existing language.
>
> I was wondering whether storeconfigs would come up.  The problem I see
> with that is storeconfigs, as I understand them, record the managed
> resource states each host *should* have, not necessarily those each
> one *does* have.  What happens to my multinode deployment if at some
> point in the middle, Puppet fails to apply a necessary resource state?
If Puppet fails applying a state, it should record the current state,
not the once and future state.

I''ll dig into the schema of what it does now, but don''t limit
your
thinking based on what storeconfigs currently does or is,
think about what it can be.


> Absolutely so, and that connects back to your earlier comments about
> message buses.  Messaging is the lifeblood of any kind of
> orchestration technology, but there are a variety of message exchange
> patterns, message implementations, and messaging options, many
> combinations of which are viable.  Some could reasonably be
> characterized as "push", but many others couldn''t.

This is an software architecture statement, and architecture is
independent of execution and concepts.

Long term, I expect to see messaging will happen and can then play a
much wider role mainly for offline delivery implications
and routing.   Though much can be hinged on present workings, whether
RESTful or RPCy (which, aren''t all that different regardless).

However, first, small steps.
>
> Any way around, there is a class of messages needed for this that
> Puppet does not currently have: messages from clients advertising the
> individual or collective state of their managed resources.  The
> absolute simplest form would be a reply to the Puppetmaster indicating
> whether a catalog was successfully applied, but finer-grained state
> messages would be much more flexible.  These would address the problem
> of discrepancy between supposed and actual state.
Eventing and reporting, and an evolved storeconfigs concept are
something we''re looking at, for sure.

Stay tuned, and of course, if you have time, patches are very much welcome.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This hits close to home with something that I would love to see added to
the puppetmaster but haven''t quite figured out how to describe until
recently.

I recently realized that, for a given environment, what you actually
have is a massive parallel program running across various nodes.
Therefore, what I would like in the puppetmaster is a set of
stored-state named semaphores.

They would work as follows:

Semaphore: DB=0 on the puppetmaster

node web_server {
  if semaphore(''semaphore_db'') > 0 {
    include "foo"
    include "bar"
  }
}

node db_server {
  include "db"
}

class db {
  service { "some_db":
    ensure => ''running'',
    require => Exec[''set_me_up'']
  }

  exec { ''set_me_up'':
    command => ''do stuff'',
    notify => semaphore(''semaphore_db'',
''>='', ''1'')
  }
}

So, what this would do, is to set the semaphore $semaphore_db to 1 only
if the exec properly executes (which means that the DB got set up
correctly) and then would only set it to 1 if it was not already >= 1 so
that multi-state processes can be handled.

Then, once the semaphore has been set, web_server would include what it
needed to properly function.

Note that this is not a locking process, the normal run of puppet would
ensue and the state change would only be noticed on the next run.

This does lag things out over a few runs of puppet, but also (I think)
keeps the entire thing running the ''puppet way'' by essentially
setting a
cross-system state fact.

I figured that the semaphore call would be an TLS REST call back to the
master, or a separate lookup process if you wanted to offload the work
to a different system.

Thanks,

Trevor

On 02/04/2010 03:42 PM, Michael DeHaan wrote:
> On Thu, Feb 4, 2010 at 9:58 AM, jcbollinger
<John.Bollinger@stjude.org> wrote:
>>
>>
>> On Feb 3, 3:08 pm, Michael DeHaan <mich...@reductivelabs.com>
wrote:
>>> Actually I think it can be done by leveraging an improved
storeconfigs and
>>> the existing language.
>>
>> I was wondering whether storeconfigs would come up.  The problem I see
>> with that is storeconfigs, as I understand them, record the managed
>> resource states each host *should* have, not necessarily those each
>> one *does* have.  What happens to my multinode deployment if at some
>> point in the middle, Puppet fails to apply a necessary resource state?
> 
> If Puppet fails applying a state, it should record the current state,
> not the once and future state.
> 
> I''ll dig into the schema of what it does now, but don''t
limit your
> thinking based on what storeconfigs currently does or is,
> think about what it can be.
> 
> 
> 
>> Absolutely so, and that connects back to your earlier comments about
>> message buses.  Messaging is the lifeblood of any kind of
>> orchestration technology, but there are a variety of message exchange
>> patterns, message implementations, and messaging options, many
>> combinations of which are viable.  Some could reasonably be
>> characterized as "push", but many others couldn''t.
> 
> 
> This is an software architecture statement, and architecture is
> independent of execution and concepts.
> 
> Long term, I expect to see messaging will happen and can then play a
> much wider role mainly for offline delivery implications
> and routing.   Though much can be hinged on present workings, whether
> RESTful or RPCy (which, aren''t all that different regardless).
> 
> However, first, small steps.
> 
>>
>> Any way around, there is a class of messages needed for this that
>> Puppet does not currently have: messages from clients advertising the
>> individual or collective state of their managed resources.  The
>> absolute simplest form would be a reply to the Puppetmaster indicating
>> whether a catalog was successfully applied, but finer-grained state
>> messages would be much more flexible.  These would address the problem
>> of discrepancy between supposed and actual state.
> 
> Eventing and reporting, and an evolved storeconfigs concept are
> something we''re looking at, for sure.
> 
> Stay tuned, and of course, if you have time, patches are very much welcome.
> 
- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaughan@onyxpoint.com
 phone: 410-541-ONYX (6699)

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAktr9CYACgkQyWMIJmxwHpR1kwCgvsOCN3x30q/INfSn2yQ0DTSR
aOwAnjztorRfAVsdCGk9UdyCxDZy3Ok6
=9cF1
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

you can do all of this stuff already today (maybe not naively).

a lot of people use functions to query or set values on some external data,
output from the function can define which resources get evaluated etc.

another option some people use is to set facter_xxx environment variables
instead of external puppet functions.

i do agree, that at the end of the day, the 30 minutes default time is
insufficient, if you want to make a lot of changes - e.g. at the end you
must wait 30 minutes between each state change (assuming you have many
clients and a few states).

so in this case, I have a gut feeling which says - use the best tool for the
job, maybe puppet is not the right choice on such cases.

Ohad

On Fri, Feb 5, 2010 at 6:34 PM, Trevor Vaughan
<tvaughan@onyxpoint.com>wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This hits close to home with something that I would love to see added to
> the puppetmaster but haven''t quite figured out how to describe
until
> recently.
>
> I recently realized that, for a given environment, what you actually
> have is a massive parallel program running across various nodes.
> Therefore, what I would like in the puppetmaster is a set of
> stored-state named semaphores.
>
> They would work as follows:
>
> Semaphore: DB=0 on the puppetmaster
>
> node web_server {
>  if semaphore(''semaphore_db'') > 0 {
>    include "foo"
>    include "bar"
>  }
> }
>
> node db_server {
>  include "db"
> }
>
> class db {
>  service { "some_db":
>    ensure => ''running'',
>    require => Exec[''set_me_up'']
>  }
>
>  exec { ''set_me_up'':
>    command => ''do stuff'',
>    notify => semaphore(''semaphore_db'',
''>='', ''1'')
>  }
> }
>
> So, what this would do, is to set the semaphore $semaphore_db to 1 only
> if the exec properly executes (which means that the DB got set up
> correctly) and then would only set it to 1 if it was not already >= 1 so
> that multi-state processes can be handled.
>
> Then, once the semaphore has been set, web_server would include what it
> needed to properly function.
>
> Note that this is not a locking process, the normal run of puppet would
> ensue and the state change would only be noticed on the next run.
>
> This does lag things out over a few runs of puppet, but also (I think)
> keeps the entire thing running the ''puppet way'' by
essentially setting a
> cross-system state fact.
>
> I figured that the semaphore call would be an TLS REST call back to the
> master, or a separate lookup process if you wanted to offload the work
> to a different system.
>
> Thanks,
>
> Trevor
>
> On 02/04/2010 03:42 PM, Michael DeHaan wrote:
> > On Thu, Feb 4, 2010 at 9:58 AM, jcbollinger
<John.Bollinger@stjude.org>
> wrote:
> >>
> >>
> >> On Feb 3, 3:08 pm, Michael DeHaan
<mich...@reductivelabs.com> wrote:
> >>> Actually I think it can be done by leveraging an improved
storeconfigs
> and
> >>> the existing language.
> >>
> >> I was wondering whether storeconfigs would come up.  The problem I
see
> >> with that is storeconfigs, as I understand them, record the
managed
> >> resource states each host *should* have, not necessarily those
each
> >> one *does* have.  What happens to my multinode deployment if at
some
> >> point in the middle, Puppet fails to apply a necessary resource
state?
> >
> > If Puppet fails applying a state, it should record the current state,
> > not the once and future state.
> >
> > I''ll dig into the schema of what it does now, but
don''t limit your
> > thinking based on what storeconfigs currently does or is,
> > think about what it can be.
> >
> >
> >
> >> Absolutely so, and that connects back to your earlier comments
about
> >> message buses.  Messaging is the lifeblood of any kind of
> >> orchestration technology, but there are a variety of message
exchange
> >> patterns, message implementations, and messaging options, many
> >> combinations of which are viable.  Some could reasonably be
> >> characterized as "push", but many others
couldn''t.
> >
> >
> > This is an software architecture statement, and architecture is
> > independent of execution and concepts.
> >
> > Long term, I expect to see messaging will happen and can then play a
> > much wider role mainly for offline delivery implications
> > and routing.   Though much can be hinged on present workings, whether
> > RESTful or RPCy (which, aren''t all that different
regardless).
> >
> > However, first, small steps.
> >
> >>
> >> Any way around, there is a class of messages needed for this that
> >> Puppet does not currently have: messages from clients advertising
the
> >> individual or collective state of their managed resources.  The
> >> absolute simplest form would be a reply to the Puppetmaster
indicating
> >> whether a catalog was successfully applied, but finer-grained
state
> >> messages would be much more flexible.  These would address the
problem
> >> of discrepancy between supposed and actual state.
> >
> > Eventing and reporting, and an evolved storeconfigs concept are
> > something we''re looking at, for sure.
> >
> > Stay tuned, and of course, if you have time, patches are very much
> welcome.
> >
>
> - --
> Trevor Vaughan
>  Vice President, Onyx Point, Inc.
>  email: tvaughan@onyxpoint.com
>  phone: 410-541-ONYX (6699)
>
> - -- This account not approved for unencrypted sensitive information --
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAktr9CYACgkQyWMIJmxwHpR1kwCgvsOCN3x30q/INfSn2yQ0DTSR
> aOwAnjztorRfAVsdCGk9UdyCxDZy3Ok6
> =9cF1
> -----END PGP SIGNATURE-----
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Feb 5, 2010 at 11:04 AM, Ohad Levy <ohadlevy@gmail.com>
wrote:
> you can do all of this stuff already today (maybe not naively).
> a lot of people use functions to query or set values on some external data,
> output from the function can define which resources get evaluated etc.
> another option some people use is to set facter_xxx environment variables
> instead of external puppet functions.
> i do agree, that at the end of the day, the 30 minutes default time is
> insufficient, if you want to make a lot of changes - e.g. at the end you
> must wait 30 minutes between each state change (assuming you have many
> clients and a few states).
> so in this case, I have a gut feeling which says - use the best tool for
the
> job, maybe puppet is not the right choice on such cases.
> Ohad
>
> On Fri, Feb 5, 2010 at 6:34 PM, Trevor Vaughan
<tvaughan@onyxpoint.com>
> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> This hits close to home with something that I would love to see added
to
>> the puppetmaster but haven''t quite figured out how to describe
until
>> recently.
>>
>> I recently realized that, for a given environment, what you actually
>> have is a massive parallel program running across various nodes.
>> Therefore, what I would like in the puppetmaster is a set of
>> stored-state named semaphores.
>>
>> They would work as follows:
>>
>> Semaphore: DB=0 on the puppetmaster
>>
>> node web_server {
>>  if semaphore(''semaphore_db'') > 0 {
>>    include "foo"
>>    include "bar"
>>  }
>> }
>>
>> node db_server {
>>  include "db"
>> }
>>
>> class db {
>>  service { "some_db":
>>    ensure => ''running'',
>>    require => Exec[''set_me_up'']
>>  }
>>
>>  exec { ''set_me_up'':
>>    command => ''do stuff'',
>>    notify => semaphore(''semaphore_db'',
''>='', ''1'')
>>  }
>> }
>>
>> So, what this would do, is to set the semaphore $semaphore_db to 1 only
>> if the exec properly executes (which means that the DB got set up
>> correctly) and then would only set it to 1 if it was not already >=
1 so
>> that multi-state processes can be handled.
>>
>> Then, once the semaphore has been set, web_server would include what it
>> needed to properly function.
>>
>> Note that this is not a locking process, the normal run of puppet would
>> ensue and the state change would only be noticed on the next run.
>>
>> This does lag things out over a few runs of puppet, but also (I think)
>> keeps the entire thing running the ''puppet way'' by
essentially setting a
>> cross-system state fact.
>>
>> I figured that the semaphore call would be an TLS REST call back to the
>> master, or a separate lookup process if you wanted to offload the work
>> to a different system.
>>
>> Thanks,
>>
>> Trevor
>>
>> On 02/04/2010 03:42 PM, Michael DeHaan wrote:
>> > On Thu, Feb 4, 2010 at 9:58 AM, jcbollinger
<John.Bollinger@stjude.org>
>> > wrote:
>> >>
>> >>
>> >> On Feb 3, 3:08 pm, Michael DeHaan
<mich...@reductivelabs.com> wrote:
>> >>> Actually I think it can be done by leveraging an improved
storeconfigs
>> >>> and
>> >>> the existing language.
>> >>
>> >> I was wondering whether storeconfigs would come up.  The
problem I see
>> >> with that is storeconfigs, as I understand them, record the
managed
>> >> resource states each host *should* have, not necessarily those
each
>> >> one *does* have.  What happens to my multinode deployment if
at some
>> >> point in the middle, Puppet fails to apply a necessary
resource state?
>> >
>> > If Puppet fails applying a state, it should record the current
state,
>> > not the once and future state.
>> >
>> > I''ll dig into the schema of what it does now, but
don''t limit your
>> > thinking based on what storeconfigs currently does or is,
>> > think about what it can be.
>> >
>> >
>> >
>> >> Absolutely so, and that connects back to your earlier comments
about
>> >> message buses.  Messaging is the lifeblood of any kind of
>> >> orchestration technology, but there are a variety of message
exchange
>> >> patterns, message implementations, and messaging options, many
>> >> combinations of which are viable.  Some could reasonably be
>> >> characterized as "push", but many others
couldn''t.
>> >
>> >
>> > This is an software architecture statement, and architecture is
>> > independent of execution and concepts.
>> >
>> > Long term, I expect to see messaging will happen and can then play
a
>> > much wider role mainly for offline delivery implications
>> > and routing.   Though much can be hinged on present workings,
whether
>> > RESTful or RPCy (which, aren''t all that different
regardless).
>> >
>> > However, first, small steps.
>> >
>> >>
>> >> Any way around, there is a class of messages needed for this
that
>> >> Puppet does not currently have: messages from clients
advertising the
>> >> individual or collective state of their managed resources.
 The
>> >> absolute simplest form would be a reply to the Puppetmaster
indicating
>> >> whether a catalog was successfully applied, but finer-grained
state
>> >> messages would be much more flexible.  These would address the
problem
>> >> of discrepancy between supposed and actual state.
>> >
>> > Eventing and reporting, and an evolved storeconfigs concept are
>> > something we''re looking at, for sure.
>> >
>> > Stay tuned, and of course, if you have time, patches are very much
>> > welcome.
>> >
>>
>> - --
>> Trevor Vaughan
>>  Vice President, Onyx Point, Inc.
>>  email: tvaughan@onyxpoint.com
>>  phone: 410-541-ONYX (6699)
>>
>> - -- This account not approved for unencrypted sensitive information --
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (GNU/Linux)
>>
>> iEYEARECAAYFAktr9CYACgkQyWMIJmxwHpR1kwCgvsOCN3x30q/INfSn2yQ0DTSR
>> aOwAnjztorRfAVsdCGk9UdyCxDZy3Ok6
>> =9cF1
>> -----END PGP SIGNATURE-----
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
Ohad, while hacks may be possible today, what I want to achieve here
are core language and out-of-the-box working system features.

I''d prefer to not perpetuate the painful workarounds if possible.

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Feb 4, 2010 at 2:58 PM, jcbollinger <John.Bollinger@stjude.org>
wrote:
>
>
> On Feb 3, 3:08 pm, Michael DeHaan <mich...@reductivelabs.com> wrote:
> I was wondering whether storeconfigs would come up.  The problem I see
> with that is storeconfigs, as I understand them, record the managed
> resource states each host *should* have, not necessarily those each
> one *does* have.  What happens to my multinode deployment if at some
> point in the middle, Puppet fails to apply a necessary resource state?
I think that hits the nail on the head.

For my part, I''d really like any solution to be able to handle
''external entities'' -
i.e. query the state of things that aren''t managed by Puppet.

The ''command and control'' layer tools (Func, mcollective,
nanite/vertebra)
seem a better tool for co-ordination tasks - trouble is they all seem a bit
disconnected (mcollective seems the closest, from a quick glance).

There has to be some way to stop them ''fighting'' with Puppet
too - I''m thinking
of things like manual failover of databases, etc.
If I had one wish, it would be an agreed API/protocol for those systems
to speak to ''Puppet'' (Foreman, an ENC, whatever) as well as
get ''live'' data.

I''m still a relative puppet n00b - I''ve put off any big
decisions
beyond straight file config
(storeconfigs, ENC, dashboards) precisely because I''m not qualified to
make that decision yet - I can certainly see the benefits of a
''sanctioned'' option,
and I don''t think that has to leave other options high and dry so long
as there''s
some sort of agreed standard.
> Any way around, there is a class of messages needed for this that
> Puppet does not currently have: messages from clients advertising the
> individual or collective state of their managed resources.
Think that explains what I was trying to say pretty neatly.
I''m really hoping someone can do an XMPP-based solution, so I
don''t have to :)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.