Puppet users - Jan 2012 - Cross-module dependencies

Since Felix seems not to have gotten around to doing this in the past
couple of days, or else was unable to do so,

On Jan 24, 3:28 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:
>
> there was a discussion in the "can we deprecate defined() in
Telly"
> thread about how we can even begin to design Forge modules without it.
>
> A recurring problem is that multiple modules rely on certain packages,
> and there is no good model (yet) to unite their resource declarations.
> Therefore it''s a common (although imho disgusting) workaround to
do
> things like
> if !defined(Package[foo]) { package { "foo": ensure =>
installed } }
>
> On 01/20/2012 11:34 PM, Cody wrote:
>
> > Defining all somewhat common packages in a central location becomes
> > unrealistic when you no longer "control" the code that is in
every
> > module you use.  If you obtain five modules from the forge and they
> > all require a specific package and so all define that package your not
> > going to convince, nor is it a good design to require everyone to move
> > the package definitions from that collection of modules.  They need to
> > function as a collection out of the box.
>
> Agreed. How can this be accomplished?

I''m not sure it can be, though I have some ideas for how we can do
better than we do now.

To start, consider the extreme case where modules have conflicting
essential requirements.  For example, they require packages that
declare conflicts with each other, or they differ about whether some
service must be enabled or disabled.  Such modules cannot *ever* work
correctly together, out of the box or otherwise, therefore it is
impossible to achieve a system that ensures that any random collection
of modules will automatically work together.

Furthermore, I think that its probably the wrong goal that even
compatible modules should always work together automatically.  The
goal should be that compatible modules work together without
modification, but it is altogether reasonable for that to require
extra code somewhere else (e.g. extra classes assigned to the node,
intermodule relationships defined, etc.).

> Perhaps there needs to be some kind of "Forge common" module that
by
> policy can only ever declare virtual resources (packages are a prominent
> example).
> A user who wishes to retain the capability of using modules from the
> Forge would be required to install this common module, and replace their
> own resource declarations with realizations of the common resources.
> For this to work, it''s definitely a plus that you can override
> attributes in collections:
> Package<| title == "apache2": |> { ensure =>
"2.2.12" }
> ...although that does bear some caveats. Does this still work in recent
> versions?
>
> If we can take this for granted, all Forge modules can adhere to that
> same standard.
>
> This is a rough sketch of how things might possibly work, and surely has
> lots of wrinkles of its own. Still, I''m quite sure we need a
proper way
> to rid ourselves of the horror that is the parse order dependent check
> for defined resources ;-)

If we must rely only on features already present in Puppet, then I
think that''s heading in the right direction.  I doubt it''s
feasible to
rely on a single "Forge Common" module, however.  Aside from the
problem of maintaining a "Common" module as other modules are created
and maintained, there is also the same essential problem I began with:
different modules may have conflicting requirements.

With respect to a given module, we need to distinguish between two
three types of resources:
1) Resources owned by that module
2) All other resources

Modules provide definitions of resources that they own.  For the most
part, those definitions should be virtual to avoid unnecessary inter-
module coupling, but some resources are reasonable to define
concretely.  Modules may realize virtual resources belonging to other
modules (without necessarily needing to know which module actually
provides the definition), but they must not override properties of
resources they do not own.

The set of resources owned by a module and the set of other resources
it depends on are parts of its external interface, and modules whose
sets of owned resources overlap are inherently incompatible.  Because
of that source of incompatibility, modules should seek to keep their
lists of owned resources small.  But how, then, can optional inter-
module dependencies be handled, or equivalently, how can a module be
made able both to interoperate and to stand on its own?  I see two
elements to this:

1) It is the site''s responsibility to ensure that all "other"
resources required by each module in use be provided.  That is the
role that Felix''s "Forge Common" module is aimed at, though I
don''t
much care for that particular solution.  Instead, I think in many
cases it will be necessary for sites to provide at least some resource
definitions via local custom modules.

2) Modules can ease the task for users by providing *optional* classes
virtually defining some or all of the "other" resources they need.
Where there are no conflicts to deal with, users could assign those
classes to nodes along with the module''s main classes, and thereby
avoid any need to define the needed resources themselves.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 26, 11:00 am, jcbollinger <John.Bollin...@stJude.org>
wrote:
> If we must rely only on features already present in Puppet, then I
> think that''s heading in the right direction.
On the other hand, if we can wish for new features to address this
area, then there may be altogether different approaches available.  In
particular, it is useful to recognize that dependencies are not just
on a particular resource generally -- rather, they are on a particular
resource having certain specific properties.  For example, a web
server module doesn''t just rely on, say,
Package[''httpd''].  Rather, it
relies on that package having an ''ensure'' parameter different
from
''absent'' and ''purged''.  Puppet DSL does not
currently have a means to
express that.

Consider, then, a new metaresource type, Constraint.  The purpose of
the Constraint resource type would be to allow multiple unrelated
classes to collaborate on defining the properties of a single
resource, and it would do so by allowing classes to limit the values
that chosen resource properties may have.

At compilation time, Puppet would collect and combine all the
constraints on any particular resource, and use the results to set
unspecified property values and validate specified ones.  Usage might
look something like this:

constraint { ''webserver-httpd_package-present'':
    resource => Package[''httpd''],
    property => ''ensure'',
    forbidden_value => [ ''absent'',
''purged'' ],
# also available: allowed_value
# maybe: default_value
}

Not only would this nicely meet the needs of different modules to
express their requirements on shared resources, it would also make it
much easier to recognize resource conflicts.  If Puppet automatically
generated empty resource definitions to constrain when it discovered
constraints on otherwise-undefined resources, then that would also
overcome the problem of deciding where to define particular resources.

I can imagine many -- perhaps most -- resource definitions being
replaced or supplemented by constraint declarations.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi John,

thanks for coming up with such elaborate ideas, your input to this group
adds a lot of meat to many discussions.

I can agree with a lot of what you wrote, barring the following remarks:

On 01/26/2012 06:00 PM, jcbollinger wrote:
> Modules provide definitions of resources that they own.  For the most
> part, those definitions should be virtual to avoid unnecessary inter-
> module coupling, but some resources are reasonable to define
> concretely.
Jeff has made a strong point against using virtual resources in modules
at all, causing me to shift my own views as well.
If I understand him correctly, one of the chief problems is the high
probability of accidental collection/realisation of such resources by
the end user''s manifest.

On 01/26/2012 06:48 PM, jcbollinger wrote:
> I can imagine many -- perhaps most -- resource definitions being
> replaced or supplemented by constraint declarations.
The model is intriguing, but gives me another usability headache.
Wouldn''t this put an end to self-contained modules?

I wrote in a latter mail (is this the same thread? Sorry, I use this
only through Thunderbird and get confused sometimes) how I see need for
explicit module dependencies and a system that can automatically
download required modules from the forge. I can see this supplementing
your idea of constraints nicely, but without it, downloading modules
could quickly become a nightmare for users.

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hello,

On Fri, Jan 27, 2012 at 15:20, Felix Frank
<felix.frank@alumni.tu-berlin.de> wrote:
> how I see need for
> explicit module dependencies and a system that can automatically
> download required modules from the forge. I can see this supplementing
> your idea of constraints nicely, but without it, downloading modules
> could quickly become a nightmare for users.
There''s something else we need to think about here. Some modules have
a soft/conditional requirement for other modules. What I mean is that
if you don''t use certain parts of a module, you don''t need the
module
that that part of the code refers to. the only decent way I can come
up with to solve that is to use what for instance in C is done with
#IFDEF. That way the module could just ignore modules that it doesn''t
_really_ require.

I for instance have modules that allow you to use different backends
for monitoring or backups. If requirements were done automatically
based on the whole module, it would need a myriad of other modules,
only one of which is ever used.

cheers,
-- 
Walter Heck

--
follow @walterheck on twitter to see what I''m up to!
--
Check out my new startup: Server Monitoring as a Service @ http://tribily.com
Follow @tribily on Twitter and/or ''Like'' our Facebook page at
http://www.facebook.com/tribily

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 26/01/12 17:48, jcbollinger wrote:
> In particular, it is useful to recognize that dependencies are not just on
a
> particular resource generally -- rather, they are on a particular resource
> having certain specific properties.
Yes.

Also: currently in Puppet one cannot say anything about a resource without a
declaration that you will "manage" it. (Unless perhaps that state
happens to be
encapsulated by a Fact, which typically isn''t and in many cases
couldn''t
feasibly be - like the case of file attributes.)

Therefore many "dependencies" are created only because of a need to
check some
state of a resource - which one may not want or need to manage.

> Consider, then, a new metaresource type, Constraint.  The purpose of the
> Constraint resource type would be to allow multiple unrelated classes to
> collaborate on defining the properties of a single resource, and it would
do
> so by allowing classes to limit the values that chosen resource properties
> may have.
> 
> At compilation time, Puppet would collect and combine all the constraints
on
> any particular resource, and use the results to set unspecified property
> values and validate specified ones.  Usage might look something like this:
> 
> constraint { ''webserver-httpd_package-present'':
>     resource => Package[''httpd''],
>     property => ''ensure'',
>     forbidden_value => [ ''absent'',
''purged'' ],
> # also available: allowed_value
> # maybe: default_value
> }
> 
> Not only would this nicely meet the needs of different modules to express
> their requirements on shared resources, it would also make it much easier
to
> recognize resource conflicts.  If Puppet automatically generated empty
> resource definitions to constrain when it discovered constraints on
> otherwise-undefined resources, then that would also overcome the problem of
> deciding where to define particular resources.
> 
> I can imagine many -- perhaps most -- resource definitions being replaced
or
> supplemented by constraint declarations.

Here''s a slightly different angle.

(Note, I''ll use capitalisation distinguish between the
"resources" that exist
outside Puppet, and the "Resource" instances inside, which model
them.)

I think there *is* a case to be made that Puppet needs a new "kind" of
Resource
declaration.  One which promises never to change the state of anything.

Immutability is, I think, the key to allowing multiple declarations of this sort
co-exist. Resources currently have to uniquely "own" a resource so
that they can
safely change it.  As I said, one doesn''t always need or want that
ownership: we
know the kind of baggage it carries.

Ideally we''d be able to separate out the aspects of a Resource which
merely
assert what *should* be the case (ensure => present etc.) from those bits
which
would then change the state of the resource if it deviates.

For the sake of discussion I''ll call that former kind of declaration an
"Assertion".

To briefly address the points from earlier email: Nan asked how would one
address merging, with respect to the following aspects of a Resource?

 a) unifying before/requires attributes
 b) unifying if/then/else constructs
 c) auditing changes back to their source
 d) unifying hash/array attributes

Well, when using Resources, yes, these make them very hard to unify in general.

However - although this needs more thought - if we could invent some way to
declare mere Assertions, they might do instead of Resources for many cases, and
it might be possible to unify them more simply, because:

 a) Problems related to ordering mostly disappear,

...since nothing is being changed. (Of course, external things which might
change things could be a problem.)

 b) If/then/else clauses can still make sense within definitions of custom
    Assertions,

...if they can compose only other Assertions, and the conditions don''t
change.

 c) Auditing is not a problem: since nothing is changed, there is nothing to
    audit.

 d) Hash/array attribute values would need to be resolved on a case-by-case
    basis,

...depending on the semantics.  When hash and arrays are semantically
representing are sets, this should be a straightforward "and" or
"or" operation.
 When the order matters, like a search path, or there is no obvious way to unify
two attributes, then this is an unresolvable contradiction and should generate
an error.  There may be cases in between.

Possibly this doesn''t fit all the use-cases which run into cross-module
dependency problems, but might significantly reduce the need to create the
dependencies in the first place.

Anyway, I need to get back to work, I''ll try to say more in a later
email.


Cheers,

N



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

On 01/27/2012 02:52 PM, Walter Heck wrote:
> There''s something else we need to think about here. Some modules
have
> a soft/conditional requirement for other modules. What I mean is that
> if you don''t use certain parts of a module, you don''t
need the module
> that that part of the code refers to. the only decent way I can come
> up with to solve that is to use what for instance in C is done with
> #IFDEF. That way the module could just ignore modules that it
doesn''t
> _really_ require.
thanks for pointing this out, but it has been covered (I think) in
another thread already:

On 01/19/2012 09:17 PM, Nick Fagerlund wrote:
> So, you can conditionally declare the rule if the defined type is
> available to the autoloader, and otherwise you don''t attempt to
manage
> the firewall and expect that the user has read the documentation and
> will make a hole for the service themselves.
>
> if defined(firewall::iptables::rule) {
>   firewall::iptables::rule {''mysql_server'':
>     ...etc. etc.
>   }
> }
>
> See? It''s just a way to peek around at what the user has
installed.
Thanks again to Nick for this quote, it keeps proving useful ;-)

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 27, 7:20 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:
> Hi John,
>
> thanks for coming up with such elaborate ideas, your input to this group
> adds a lot of meat to many discussions.
>
> I can agree with a lot of what you wrote, barring the following remarks:
>
> On 01/26/2012 06:00 PM, jcbollinger wrote:
>
> > Modules provide definitions of resources that they own.  For the most
> > part, those definitions should be virtual to avoid unnecessary inter-
> > module coupling, but some resources are reasonable to define
> > concretely.
>
> Jeff has made a strong point against using virtual resources in modules
> at all, causing me to shift my own views as well.
> If I understand him correctly, one of the chief problems is the high
> probability of accidental collection/realisation of such resources by
> the end user''s manifest.

The core problem to solve is that multiple modules may depend on the
same resources, but each resource can be defined only once.

If modules'' resource requirements were invariant then I could be
satisfied with documenting them and requiring the user to provide them
concretely; indeed, that''s not very far from what I just proposed.  On
the contrary, however, whether a particular resource is needed by some
module may depend on which of that module''s classes are used, and on
which parameters and external data are supplied.  This approach
therefore has its own inherent problem for users: to avoid declaring
unneeded resources.

Given a choice between the problems of avoiding declaring unneeded
resources and of avoiding collecting unwanted resources, I would
choose the latter.  It is easier to deal with, and failure to
completely solve it is no more costly than failure to solve the
alternative problem.

> On 01/26/2012 06:48 PM, jcbollinger wrote:
>
> > I can imagine many -- perhaps most -- resource definitions being
> > replaced or supplemented by constraint declarations.
>
> The model is intriguing, but gives me another usability headache.

From a usability perspective, I think this is a far better proposal
than anything else on the table:
(+) It enables modules to declare their requirements more precisely,
reducing the risk of modules compiling and applying correctly, but
nevertheless failing to achieve the desired configuration.
Constraints are the only proposal currently on the table that provide
that.
(+) A key aspect is that it allows Puppet to pinpoint and diagnose
conflicts at compilation time.
(+) Furthermore, it allows each module to express *only* its
requirements, ignoring properties that it doesn''t care about (though
other modules might care about them).  In fact, it rewards modules for
doing so by making them more interoperable.
(+) It does not rely on virtual resources (though it is compatible
with them), so it does not impose any risk of accidental resource
collection.
(+) It could be implemented in a manner that forestalls inter-module
parse order dependencies.

The main cost I see is:
(-) It creates a new distinction between resource properties that are
explicitly specified in resource definitions and those that are not.
For this reason (at least), such a feature could not be introduced
until Telly at the earliest.

That cost is incurred, however, only in manifest sets that use
Constraints, and then only with respect to resources that are targeted
by Constraints.  It is anyway not a usability issue, but rather an
upgrade / conversion / compatibility issue.

> Wouldn''t this put an end to self-contained modules?

No.  What makes you think it would?  I think it allows modules to be
*more* self-contained than they currently are.

> I wrote in a latter mail (is this the same thread? Sorry, I use this
> only through Thunderbird and get confused sometimes) how I see need for
> explicit module dependencies and a system that can automatically
> download required modules from the forge. I can see this supplementing
> your idea of constraints nicely, but without it, downloading modules
> could quickly become a nightmare for users.

I have considered that possibility, essentially a package management
system for Puppet modules.  Although such a thing could undoubtedly be
implemented, I don''t think it would be very suitable for addressing
the inter-module dependency problem.  Package management systems such
as yum and apt rely on the fact that software dependency graphs are
more or less trees, but I don''t think module dependency graphs tend to
be anywhere near as treelike.  I suppose module repository managers
could enforce tree-shaped dependency graphs, but that imposes the
tool''s requirements on the problem space instead of relying on a tool
that suits well in the first place.

I do not mean to say that a tool that automatically downloaded and
installed modules from the Forge would be useless -- far from it.  I
just don''t think that it would adequately address the inter-module
dependency issue, and therefore I would recommend that such a tool not
even try to do so.  It solves an altogether different problem.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

On 01/27/2012 04:22 PM, jcbollinger wrote:
> From a usability perspective, I think this is a far better proposal
> than anything else on the table:
I''ve thought of another plus. Even though the design proposal adds to
the DSL (and complexity is generally to be avoided), it does so in a
manner that will not make it necessary for novices (or any end user) to
deal with the particular featureset, but instead limits its target
audience to developers of public modules.
>> Wouldn''t this put an end to self-contained modules?
>
> No.  What makes you think it would?  I think it allows modules to be
> *more* self-contained than they currently are.
I was thinking along the lines of "currently modules will just install
the packages they need", for instance. But you''re right of course
-
currently modules can break each other because of this, so a way for the
compiler to clearly pinpoint reasons and locations of mismatches would
indeed be superior.
> I do not mean to say that a tool that automatically downloaded and
> installed modules from the Forge would be useless -- far from it.  I
> just don''t think that it would adequately address the inter-module
> dependency issue, and therefore I would recommend that such a tool not
> even try to do so.  It solves an altogether different problem.
Again, I''m inclined to concur. The issue at hand is to stop modules
from
breaking each other horribly. Once that''s off the table, a module
management system is a whole new issue in and of itself.

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 27, 8:01 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:
> Hi,
>
> On 01/27/2012 02:52 PM, Walter Heck wrote:
>
> > There''s something else we need to think about here. Some
modules have
> > a soft/conditional requirement for other modules. What I mean is that
> > if you don''t use certain parts of a module, you
don''t need the module
> > that that part of the code refers to. the only decent way I can come
> > up with to solve that is to use what for instance in C is done with
> > #IFDEF. That way the module could just ignore modules that it
doesn''t
> > _really_ require.
>
> thanks for pointing this out, but it has been covered (I think) in
> another thread already:
>
> On 01/19/2012 09:17 PM, Nick Fagerlund wrote:
[...]

And Nick''s approach seems good and useful, but I don''t think
it
addresses Walter''s concern.  Walter observes that modules''
runtime
inter-module dependencies can depend on how they are used, and he
argues, I think, that that complicates the problem of managing such
dependencies via a package management system.

For example, if module X depends on module Y only for feature
X::ySupport, and I intend never to use X::ySupport, then I might like
to avoid installing module Y.  If I furthermore want to use module Y2
that is incompatible with module Y, then I might *need* to avoid
installing Y.  There are ways to handle problems such as those, but it
all creates a lot of overhead to make everything work together
properly in such a scheme.


John
A hollow voice says "Plugh".

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 27, 7:58 am, Nick <oinksoc...@letterboxes.org>
wrote:
> On 26/01/12 17:48, jcbollinger wrote:
>
> > In particular, it is useful to recognize that dependencies are not
just on a
> > particular resource generally -- rather, they are on a particular
resource
> > having certain specific properties.
>
> Yes.
>
> Also: currently in Puppet one cannot say anything about a resource without
a
> declaration that you will "manage" it. (Unless perhaps that state
happens to be
> encapsulated by a Fact, which typically isn''t and in many cases
couldn''t
> feasibly be - like the case of file attributes.)

Of course, this is a question of scale and foreknowledge.  One can
easily create a custom fact expressing any or all of the attributes of
a specific file.  I think one could even do it for all of the files in
a directory, or even for an entire directory tree.  Before too long,
however, that''s going to become quite slow, and it would be difficult
to use on the Puppet side.  I can''t think of a persistent node state
detail that couldn''t be expressed via a custom fact, but if you need
many such details, or if you can''t be sure in advance which you will
need, then you do have a problem.

> Therefore many "dependencies" are created only because of a need
to check some
> state of a resource - which one may not want or need to manage.

It sounds like you want to perform conditional logic on the agent.  I
am not at all comfortable with that idea, though perhaps I could
become more so.  It entails a much more fundamental change to Puppet
than anything else we have been discussing.  Or did you have something
different in mind?

> Here''s a slightly different angle.
[...]
> Ideally we''d be able to separate out the aspects of a Resource
which merely
> assert what *should* be the case (ensure => present etc.) from those
bits which
> would then change the state of the resource if it deviates.
>
> For the sake of discussion I''ll call that former kind of
declaration an "Assertion".
[...]
> Possibly this doesn''t fit all the use-cases which run into
cross-module
> dependency problems, but might significantly reduce the need to create the
> dependencies in the first place.

I could see implementing a version of your Assertion idea as a check-
only Constraint, but I don''t think there is yet an infrastructure to
support that part.  However, I don''t see at all how the Assertion idea
fits into the cross-module dependency picture.  Perhaps this is
because I don''t see the purpose of declaring resources that you do not
intend to manage.

When I consider cross-module dependencies, I think the usual case is
that a module *does* want ensure that particular resource properties
are managed, but it may not care about other properties, and it cares
about who owns Resources only insofar as someone has to own them and
it affects who can manage them.  I can imagine wanting to _assert_
Resource properties only as a workaround for not being able actually
to manage them.  Are there other reasons to want such a thing?

> Anyway, I need to get back to work, I''ll try to say more in a
later email.

That would help me to determine what I think about the idea.  As it
is, I suspect I don''t quite understand what you are hoping to
accomplish with it.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I''ve been mulling this over and wanted to get opinions on an, ugly, but
completely functional approach.

We''ve been talking about people supplying metadata to describe
inter-class dependencies (inter-module dependencies really, but hear me out).
With the
advent of the parameterized class, you could simply write all ifdef dependencies
directly into a parameter like follows:

# Get this for the ''contains'' function
include ''stdlib''

class foo (
  # No requirements by default
  $reqs = ['''']
) {
  if contains($reqs,''bar'') {
    include ''bar''
  }

  ...some stuff...

  if contains($reqs, ''bar'') {
    bar::baz { ''something'': ... }
  }
}

It''s not elegant by any means, but it is functional and since (in
theory) puppet only includes a class once, then all of the various includes
would be
completely skipped.

If would be nice if, in this example, $reqs was actually a class metaparameter
and Puppet would automatically try to include the class when passed
into that variable.

Benefits:

 * Works with the current language structure
 * Does what you want it to in terms of not needing defined

Drawbacks:

 * Requires the user to have an explicit working knowledge of all modules and
namespaces
 * Adds a lot of random logic to the code (unless it becomes a metaparam of some
sort)

Thanks,

Trevor

On 01/27/2012 08:52 AM, Walter Heck wrote:
> Hello,
> 
> On Fri, Jan 27, 2012 at 15:20, Felix Frank
> <felix.frank@alumni.tu-berlin.de> wrote:
>> how I see need for
>> explicit module dependencies and a system that can automatically
>> download required modules from the forge. I can see this supplementing
>> your idea of constraints nicely, but without it, downloading modules
>> could quickly become a nightmare for users.
> There''s something else we need to think about here. Some modules
have
> a soft/conditional requirement for other modules. What I mean is that
> if you don''t use certain parts of a module, you don''t
need the module
> that that part of the code refers to. the only decent way I can come
> up with to solve that is to use what for instance in C is done with
> #IFDEF. That way the module could just ignore modules that it
doesn''t
> _really_ require.
> 
> I for instance have modules that allow you to use different backends
> for monitoring or backups. If requirements were done automatically
> based on the whole module, it would need a myriad of other modules,
> only one of which is ever used.
> 
> cheers,
- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaughan@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJPJBXGAAoJECNCGV1OLcypcXkH/3Y2nqqGgJzAKg9YVj/DjiB7
8zbtA7/nVvC8LwtIwwGi7jY+VcbietGwNC8JOoxnTdFN4dCb1xsAcTqzt8p/NXHE
HhwGIG9YGaMoZzvwtfUGc6wrOeqxLvInq2g6e0Qk5QkhBZVg7T5DV4/mvXfheZOR
n1mENjPNMoRONifb24PqxK91CbRtBmJGxEX8b6pDB529oU6aZxNQi6xSn1KSkCJM
SZjVaDoxPqHC4V9L3/J34Rq8H96tfMvTHvSjI3+/nrX80k9MRTkIw5LMIESfTktM
oHKmIXeYcf1yymepuwFmjEgvQ/hp0P5YWsXX3xhE+OCEoaby0AQ6FRHtSJq2y8E=qNAo
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I just wanted to post to this thread to primarily encourage you all to keep
brainstorming, and to make it clear that I''m paying close attention. :)



-- 
Nigel Kersten
Product Manager, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel,

It frightens me a bit that I think the "correct" solution, will be to
replicate what the distros are doing in Puppetforge. Basically turning
puppetforge into a massive cross distro metadata repo, with very strict
contribution standards and rules. This would involve strong rules for
curated modules that would require manpower to vet (and to contribute the
modules).

Maybe, we might want to extend modules so that there are two namespaces,
one for curated forge modules and another for local modules. (Making the
"local module" namespace the default for backwards compatibility.) One
example of a potential rule would be to require that official modules
support a mandatory set of popular OSes. Don''t allow more than one
official
module per package. e.g.: In the curated section of forge there will be one
MySQL module. (not to get too ahead of the cart, but I envision a time when
you can instantiate a MySQL service by just telling puppet to use the Forge
MySQL module, puppet handles downloading and installing the module and
figuring out what OS you are running and with the help of the native
package management, installs and configures the basic MySQL service.) The
official modules will be curated such that if there is a a common resource
they need to manage that resource will be split into a separate singular
dependency module, that incorporates the requirements of
all dependent forge modules. (Not a bag of common resources in a single
module, but rather a separate module for each shared resource.)

Maybe, I am overthinking this, but I think this is the "right"
solution,
that may require more resources to implement than the community has
available.

That leaves us what to do about "defined". (Or was that a different
thread?) In the case of defined, my group has only ever used it once in
almost 4 years, but it seems from the discussions that there are others
still using it. Maybe the answer is provide a real alternative first, and
then go about deprecating it? We wouldn''t miss it, but I could see the
challenges of rewriting a codebase that depends on it, and I wouldn''t
want
that rewrite enforced on me, without a solid alternative.

Cheers,
Brian

P.S. - No one is going to really love this solution, including myself, but
that doesn''t necessarily mean it shouldn''t be done.

On Sat, Jan 28, 2012 at 9:18 PM, Nigel Kersten <nigel@puppetlabs.com>
wrote:
> I just wanted to post to this thread to primarily encourage you all to
> keep brainstorming, and to make it clear that I''m paying close
attention. :)
>
>
>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
<http://aws.amazon.com/solutions/solution-providers/brandorr/>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Jan 27, 2012 at 5:20 AM, Felix Frank
<felix.frank@alumni.tu-berlin.de> wrote:
> Jeff has made a strong point against using virtual resources in modules
> at all, causing me to shift my own views as well.
> If I understand him correctly, one of the chief problems is the high
> probability of accidental collection/realisation of such resources by
> the end user''s manifest.
Yep, that''s it.

-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel,

I just wanted to add, if we do go this route, we should work to support
private "forges" (module repos) as well.

Cheers,
Brian

On Sun, Jan 29, 2012 at 1:39 AM, Brian Gupta
<brian.gupta@brandorr.com>wrote:
> Nigel,
>
> It frightens me a bit that I think the "correct" solution, will
be to
> replicate what the distros are doing in Puppetforge. Basically turning
> puppetforge into a massive cross distro metadata repo, with very strict
> contribution standards and rules. This would involve strong rules for
> curated modules that would require manpower to vet (and to contribute the
> modules).
>
> Maybe, we might want to extend modules so that there are two namespaces,
> one for curated forge modules and another for local modules. (Making the
> "local module" namespace the default for backwards
compatibility.) One
> example of a potential rule would be to require that official modules
> support a mandatory set of popular OSes. Don''t allow more than one
official
> module per package. e.g.: In the curated section of forge there will be one
> MySQL module. (not to get too ahead of the cart, but I envision a time when
> you can instantiate a MySQL service by just telling puppet to use the Forge
> MySQL module, puppet handles downloading and installing the module and
> figuring out what OS you are running and with the help of the native
> package management, installs and configures the basic MySQL service.) The
> official modules will be curated such that if there is a a common resource
> they need to manage that resource will be split into a separate singular
> dependency module, that incorporates the requirements of
> all dependent forge modules. (Not a bag of common resources in a single
> module, but rather a separate module for each shared resource.)
>
> Maybe, I am overthinking this, but I think this is the "right"
solution,
> that may require more resources to implement than the community has
> available.
>
> That leaves us what to do about "defined". (Or was that a
different
> thread?) In the case of defined, my group has only ever used it once in
> almost 4 years, but it seems from the discussions that there are others
> still using it. Maybe the answer is provide a real alternative first, and
> then go about deprecating it? We wouldn''t miss it, but I could see
the
> challenges of rewriting a codebase that depends on it, and I
wouldn''t want
> that rewrite enforced on me, without a solid alternative.
>
> Cheers,
> Brian
>
> P.S. - No one is going to really love this solution, including myself, but
> that doesn''t necessarily mean it shouldn''t be done.
>
>
> On Sat, Jan 28, 2012 at 9:18 PM, Nigel Kersten
<nigel@puppetlabs.com>wrote:
>
>> I just wanted to post to this thread to primarily encourage you all to
>> keep brainstorming, and to make it clear that I''m paying close
attention. :)
>>
>>
>>
>> --
>> Nigel Kersten
>> Product Manager, Puppet Labs
>>
>>
>>
>>  --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
>
>
> --
> <http://aws.amazon.com/solutions/solution-providers/brandorr/>
>
>

-- 
<http://aws.amazon.com/solutions/solution-providers/brandorr/>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

On 01/28/2012 04:35 PM, Trevor Vaughan wrote:
> Drawbacks:
>
>  * Requires the user to have an explicit working knowledge of all
modules and namespaces
>  * Adds a lot of random logic to the code (unless it becomes a
metaparam of some sort)

You skipped the most important drawback: Commitment to parameterized
classes. The fact that there can be only one place that includes those
classes, and that this singular place must have the whole picture of
what requirements are met, is conceivably a show stopper from my point
of view.

This will work for people that have a functional ENC, I guess, but
should that be a requirement for using Forge modules?

Furthermore, how can modules hope to ever interoperate like this? If all
module classes get parameterized, it will be outright impossible for one
module to ever include another module''s classes.
Say module A includes class B::C. As soon as a user installs module A in
addition to B, they have to clean their manifests of inclusions of B::C.

On 01/29/2012 07:39 AM, Brian Gupta wrote:
> It frightens me a bit that I think the "correct" solution, will
be to
> replicate what the distros are doing in Puppetforge. Basically turning
> puppetforge into a massive cross distro metadata repo, with very strict
> contribution standards and rules. This would involve strong rules for
> curated modules that would require manpower to vet (and to
> contribute the modules). 
I honestly don''t see the problem. Imagine CPAN was limited to downloads
of tarballs from the website (or even souce control checkouts). I
disbelieve it would be as significant today as it has become.
The same goes for Ruby Gems and all such systems.

As this seems to be a recurring theme: Am I wrong to compare these to
the Forge?

Sincerely,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Had a bunch more thoughts on this topic, and I feel I agree with
Brian: a forge with one module for every purpose, supported maybe by
library modules (like linux) would be the best. It would mean
consolidating the 17 ssh modules that exist now into one all
encompassing ssh module. In the case of SSH that is totally doable,
but consider the 300 gazillion different use cases of apache and
things get interesting. I think this would mean we''d have appointed
maintainers for each module, and if possible a puppetlabs provided
jenkins install (with community provided jenkins slaves?) that would
do the CI testing every night. That would then make it possible to
have an auto-updated overview for each module of the latest supported
OS''s for puppet, and wether that module passes tests for that OS.

This seems like a massive undertaking from where we are now, but it
would in the end make all of our lives a ton easier (one trusted
source for good high quality modules) and reduce the ''problem''
of
inter-module dependencies to a minimum. Of course it still exists for
in-house applications that are being puppetised, but it would already
mean the world if they would be able to depend on what the public
trusted modules define.

I personally like the way the drupal module projects work: anyone can
start a project, but they are all hosted on the drupal.org site within
drupal.org version control, and they have teams of code reviewers
maintaining integrity of the module base that lives on drupal.org.

cheers,

Walter

On Mon, Jan 30, 2012 at 10:51, Felix Frank
<felix.frank@alumni.tu-berlin.de> wrote:
> Hi,
>
> On 01/28/2012 04:35 PM, Trevor Vaughan wrote:
>> Drawbacks:
>>
>>  * Requires the user to have an explicit working knowledge of all
> modules and namespaces
>>  * Adds a lot of random logic to the code (unless it becomes a
> metaparam of some sort)
>
> You skipped the most important drawback: Commitment to parameterized
> classes. The fact that there can be only one place that includes those
> classes, and that this singular place must have the whole picture of
> what requirements are met, is conceivably a show stopper from my point
> of view.
>
> This will work for people that have a functional ENC, I guess, but
> should that be a requirement for using Forge modules?
>
> Furthermore, how can modules hope to ever interoperate like this? If all
> module classes get parameterized, it will be outright impossible for one
> module to ever include another module''s classes.
> Say module A includes class B::C. As soon as a user installs module A in
> addition to B, they have to clean their manifests of inclusions of B::C.
>
> On 01/29/2012 07:39 AM, Brian Gupta wrote:
>> It frightens me a bit that I think the "correct" solution,
will be to
>> replicate what the distros are doing in Puppetforge. Basically turning
>> puppetforge into a massive cross distro metadata repo, with very strict
>> contribution standards and rules. This would involve strong rules for
>> curated modules that would require manpower to vet (and to
>> contribute the modules).
>
> I honestly don''t see the problem. Imagine CPAN was limited to
downloads
> of tarballs from the website (or even souce control checkouts). I
> disbelieve it would be as significant today as it has become.
> The same goes for Ruby Gems and all such systems.
>
> As this seems to be a recurring theme: Am I wrong to compare these to
> the Forge?
>
> Sincerely,
> Felix
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Walter Heck

--
follow @walterheck on twitter to see what I''m up to!
--
Check out my new startup: Server Monitoring as a Service @ http://tribily.com
Follow @tribily on Twitter and/or ''Like'' our Facebook page at
http://www.facebook.com/tribily

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It is annoying to have everything in a single place that defines the state of
your nodes but, as you point out, this seems to be the model if you''re
using an ENC and that seems to be the recommended practice across the mailing
list for any sort of scale.

But, you don''t need a functional ENC to make this work, you simply need
to have everything defined at the top level whether it be via node, higher
level class, or ENC.

The main issue here seems to be modules that are trying to be "too
helpful" from my reading of the mailing list.

It seems that many would like this to be an anti-pattern:

class foo {

  if not defined(Class[''bar'']) {
    include ''bar''
    bar::baz { ... }
  }
}

include ''foo''

Instead, you should be less helpful, and do the following:

include ''foo''
include ''bar''

bar::baz { ... }

So, instead of doing something like, say, setting up IPTables in your module
(thus creating a cross-module dependency), you should do all of this in
one monolithic place at the node level or a higher level aggregation class
level.

While this keeps your modules clean, it seems like a lot more effort to maintain
since the module for nginx should really know what ports it''s using
and know how to set up its own firewall rules.

So, the tradeoff is an ENC vs. a large collection of cluttered classes at the
top level to make sure you don''t have cross-module dependencies.

I''m not sure if either is better (or if either is any good at all) but
they''re both functional.

The ability to tag modules as requiring other modules of a particular version
(ala CPAN, Gems, everything else....) would solve this issue as Puppet
would be able to check to make sure that you have the correct version of the
modules installed prior to compiling the catalog.

Trevor

On 01/30/2012 03:51 AM, Felix Frank wrote:
> Hi,
> 
> On 01/28/2012 04:35 PM, Trevor Vaughan wrote:
>> Drawbacks:
>>
>>  * Requires the user to have an explicit working knowledge of all
> modules and namespaces
>>  * Adds a lot of random logic to the code (unless it becomes a
> metaparam of some sort)
> 
> You skipped the most important drawback: Commitment to parameterized
> classes. The fact that there can be only one place that includes those
> classes, and that this singular place must have the whole picture of
> what requirements are met, is conceivably a show stopper from my point
> of view.
> 
> This will work for people that have a functional ENC, I guess, but
> should that be a requirement for using Forge modules?
> 
> Furthermore, how can modules hope to ever interoperate like this? If all
> module classes get parameterized, it will be outright impossible for one
> module to ever include another module''s classes.
> Say module A includes class B::C. As soon as a user installs module A in
> addition to B, they have to clean their manifests of inclusions of B::C.
> 
> On 01/29/2012 07:39 AM, Brian Gupta wrote:
>> It frightens me a bit that I think the "correct" solution,
will be to
>> replicate what the distros are doing in Puppetforge. Basically turning
>> puppetforge into a massive cross distro metadata repo, with very strict
>> contribution standards and rules. This would involve strong rules for
>> curated modules that would require manpower to vet (and to
>> contribute the modules). 
> 
> I honestly don''t see the problem. Imagine CPAN was limited to
downloads
> of tarballs from the website (or even souce control checkouts). I
> disbelieve it would be as significant today as it has become.
> The same goes for Ruby Gems and all such systems.
> 
> As this seems to be a recurring theme: Am I wrong to compare these to
> the Forge?
> 
> Sincerely,
> Felix
> 
- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaughan@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJPJoUxAAoJECNCGV1OLcyp7igH/0rroAjC8Ewc9Aw2bdE7gO0N
0KfvzYCTZtJLFTBeNIAErliWd9iR5W84H0j8KJGjPg18qcRcDHjC/hnf5+GV8lIS
6kG3EgwYwyDg8Xc0qAbWubJv7bJ29X4Fc8CCHkq13CkXFM/OqnKpUbXA6X6+o5a/
Hv5Z6WXQjPC7uCupwyqktkjj5sjwvdgniSvKsj6EK3bhGRyMsvJAzmDjucwcRNsM
vz6IG05aFJrYTUp0rZzTJf/HjIPgmD90puoXXSa/RVQnsb3WSw0AwYe1jBAHWap4
pDw+F2qrMdwc9XgQv4ZFFNp/A1OCFh21uW3B1D7XjM+U3QRpmXTKhX71lcqbX08=XRv1
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 30, 2:51 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:
> Hi,
>
> On 01/28/2012 04:35 PM, Trevor Vaughan wrote:> Drawbacks:
>
> >  * Requires the user to have an explicit working knowledge of all
>
> modules and namespaces>  * Adds a lot of random logic to the code
(unless it becomes a
>
> metaparam of some sort)
>
> You skipped the most important drawback: Commitment to parameterized
> classes. The fact that there can be only one place that includes those
> classes, and that this singular place must have the whole picture of
> what requirements are met, is conceivably a show stopper from my point
> of view.

+1

> This will work for people that have a functional ENC, I guess, but
> should that be a requirement for using Forge modules?
>
> Furthermore, how can modules hope to ever interoperate like this? If all
> module classes get parameterized, it will be outright impossible for one
> module to ever include another module''s classes.
> Say module A includes class B::C. As soon as a user installs module A in
> addition to B, they have to clean their manifests of inclusions of B::C.

Right.  The approach depends on using class parameters to pass
requirements, but it depends on classes being *un*parameterized to
apply those requirements -- catch-22.  There is a solution to that
particular problem, however: define requirements via external
[meta]data instead of via parameters.

That''s not to say I like the proposal, however. I don''t.  In
fact I
think it''s completely inside-out.  Classes should *know* what other
classes they depend upon; they should not need to be told.

Moreover, I don''t see what is gained by passing requirements as
parameters rather than, say, having the user separately declare
optional classes that carry the same information.  For example,
instead of a structure requiring this:

class { ''foo'': reqs => [''bar''] }

it should be possible to structure a module''s classes so that you can
do this instead:

include ''foo::bar''
include ''foo''

(perhaps in the opposite order).

> On 01/29/2012 07:39 AM, Brian Gupta wrote:
>
> > It frightens me a bit that I think the "correct" solution,
will be to
> > replicate what the distros are doing in Puppetforge. Basically turning
> > puppetforge into a massive cross distro metadata repo, with very
strict
> > contribution standards and rules. This would involve strong rules for
> > curated modules that would require manpower to vet (and to
> > contribute the modules).
>
> I honestly don''t see the problem. Imagine CPAN was limited to
downloads
> of tarballs from the website (or even souce control checkouts). I
> disbelieve it would be as significant today as it has become.
> The same goes for Ruby Gems and all such systems.
>
> As this seems to be a recurring theme: Am I wrong to compare these to
> the Forge?

Yes and no.

On one hand, clearly there are structural similarities between, say,
Puppet modules and Perl modules.  The Forge serves a similar role for
the former to the role CPAN serves for the latter.  I can totally see
the usefulness of tools for automatically installing and removing
Forged Puppet modules.

On the other hand, there are several problems with adopting such an
approach to solve the Puppet module dependency issue, among them:

1) The Forge doesn''t address the problem AT ALL as it now stands.  The
relevant resource that package management systems offer is not the
packages themselves (which the Forge has), but rather the metadata
database that each system relies on.  Defining, building, and
maintaining such a database would be a massive and ongoing endeavor,
quite separate from developing and managing the Puppet and module
codebases.

2) A package management system would only work around the problem, not
solve it, and then only among modules from the same repository.

3) In fact, it would not really be the package management system
itself that made anything work.  Instead, authors of Forge modules
would still need to find some way to make their modules interoperate,
and Forge curators would need somehow to ensure that they did.  All
the management software would really contribute is a dependency-aware
ability to automatically install/remove packages.  The system overall
would provide a promise that the particular modules it contains all do
work together, but that''s altogether different from a means by which
anyone -- the Forge or otherwise -- can fulfill such a promise.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sat, Jan 28, 2012 at 10:39 PM, Brian Gupta
<brian.gupta@brandorr.com>wrote:
> Nigel,
>
> It frightens me a bit that I think the "correct" solution, will
be to
> replicate what the distros are doing in Puppetforge. Basically turning
> puppetforge into a massive cross distro metadata repo, with very strict
> contribution standards and rules. This would involve strong rules for
> curated modules that would require manpower to vet (and to contribute the
> modules).
>
It''s worth pointing out here that a massive cross distro repository of
modules doesn''t necessarily require strict contribution standards and
rules.

There are other options to ensure that high quality modules are available
other than creating a high barrier to entry for the modules themselves.
Exposing community opinion on modules and success rates of people deploying
them can help us all achieve the same goal.

What kind of a repository do we want?

-- 
Nigel Kersten
Product Manager, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 27/01/12 18:27, jcbollinger wrote:
>> Anyway, I need to get back to work, I''ll try to say more in a
later email.
> 
> 
> That would help me to determine what I think about the idea.  As it
> is, I suspect I don''t quite understand what you are hoping to
> accomplish with it.
Ok, here''s a couple of examples.  Apologies for the length.


1. Packages

Let''s say I''m writing a module called foo.

It uses a package called ''libfoo''.

Therefore I want to make sure this package is installed before the
module''s
configuration is applied.  I use the pattern where I define separate classes to
handle install and config, and chain them together:

 class foo::install {
    # Make sure libfoo is installed
    package { ''libfoo'': ensure => present }
 }

 class foo::config {
    # adjusts foo.conf
    # ....
 }

 class foo {
   include foo::install
   include foo::config

   Class[foo::install] -> Class[foo::config]
 }


Bingo! Works fine.  Until I want to add another 3rd party module which needs
libfoo.  Now I have to move the package declaration out to a shared class and
update both modules to fit: maybe the other module wants a specific version,
other install_options, or whatever.

Sounds simple in a short example like this, except, that:

 - I''ve been forced to customise what should be an ''off the
shelf'' module,
 - I have to figure out what the shared class/module should say
 - and fix an arbitrary name for it
 - I''ve therefore hard-wired strong coupling in both to my shared class
 - I''ve added potential for refactoring breakage
 - and more of the same sort of problems when scaling up

Genuinely reusable modules seem nearly impossible to write as it stands. If I
want to publish my module on Puppet Forge, then the shared::libfoo module must
be published too.  Except it might not agree with other published
modules'' ideas
about what a shared::libfoo should declare, or even be called, and so it is not
typically re-usable without refactoring.

Or, I don''t publish it, leave a dangling dependency on a class called
shared::libfoo.  I am still hard-wiring a name, but not what it does. I have to
tell users the name the module they must define, and a list of requirements to
put in it on our behalf.

Or, I just don''t define anything about libfoo except in the
documentation.
Which seems the most practical thing to do, assuming that things break
intelligibly if libfoo is absent, but this really amounts to giving up and
moving on.

Or, maybe there currently are better ways than all the above - but if so
I''m
unclear what.




Now imagine we could simply assert a requirement on a package, without actually
managing it.  For the sake of this example, I''ll invent a syntax for
"Assertions" similar to that used for Virtual  Resources which use an
''@'' sigil
prefix. I''ll arbitrarily use a ''+'' instead:

  +package { ''libfoo'': ensure => present }


This just means "ensure libfoo is installed".  It changes nothing
about the
target system. It does not mean "go and install the
''libfoo'' package".  It does
not mean "I own the ''libfoo'' package resource, and
I''ll fight anyone who tries
to say otherwise".

Therefore, this type of assertion can be repeated multiple times in different
modules.  Possibly in slightly different ways - with extra attributes, etc.
Puppet should just check they are all consistent, and fail if they
aren''t, or if
the net requirements are not met.  I don''t know enough about Puppet
internals to
say for sure, but as described in my previous email:  because the Assertion
changes nothing, I hope this would be relatively easy to implement.

Now I can write my module using an Assertion instead:

 class foo::install {
    # Make sure libfoo is installed
    +package { ''libfoo'': ensure => present }

    # ...
 }

...and I no longer have to find the common ground between modules which use
libfoo, and/or modify the modules to use the shared declaration.

Also, we have lost an explicit dependency on a shared module arbitrarily called
''shared::libfoo'' which merely declared:

   package { ''libfoo'': ensure => present }

So I no longer need to publish this shared module and either dictate to, or
negotiate with with potential users of my module about the intersection of our
requirements.  Nor do I need to omit this requirement entirely (which might be
the only practical alternative).

Yet I am still checking the prerequisites are there.

Of course, I may still have to create a package which actually does the
appropriate package install. Or maybe not? Perhaps my provisioning system does
that for me, and I can skip that step?  Either way, the knowledge that my system
is still checking the prerequisites are there.

If my prerequisites are missing, I would hope Puppet would give helpful errors
showing what needed what, and I can add a declaration to install the right
packages in a top-level "glue" class. But means we can avoid
hard-wiring
arbitrary module names into the component modules.

In summary, this would be simpler and more effective than any existing Puppet
pattern I know about.




2. Creating user accounts

Another example, which was the topic of my earlier post "constraint
checking".
Say I want to create a custom resource which sets up user accounts for me in a
manner of my choice.


  define user_account(
    $name,
    $home,
    $shell,
    $passwd,
    $uid,
    $gid,
    $groups,
  ) {
    # I want to validate $home, $shell, $groups exist and are usable...

    # This is a classic case where one is tempted to use this anti-pattern
    # to define something usable if it doesn''t exist:
    if !define(Group[$gid]) {
       group { $gid: ensure => present }
    }

    # If I can''t do that, perhaps I can just depend and hope
it''s picked up
    # elsewhere?
    require File[$shell]

    # ... except this can''t say anything about $shell, like
    # "it must be executable".


    # .... do other stuff here ....

    # Now define the user resource which creates the user
    # (but in my tests, does not seem to check the requirements
    # exist.)
    user {$name:
      ensure => present,
      home => $home,
      shell => $shell,
      passwd =>
      uid => $uid,
      gid => $gid,
      groups => $groups,
    }
  }



Imagine we could use Assertions as described above.  Validating the parameters
is now straightforward:

  define user_account(
    $name,
    $home,
    $shell,
    $passwd,
    $uid,
    $gid,
    $groups,
  ) {
    # Ensure $home, $shell, gid, $groups exist

    # We want to own this one
    group { $gid: ensure => present }

    # These are shared
    +group { $groups:  ensure => present }

    # This is shared
    +file { $shell:
      ensure => present,
      mode => ''0744'', # ...if only I could merely say
"a+x" here
    }

    # ... do other stuff ...

    # Now define the resource:
    user {$name:
      ensure => present,
      home => $home,
      shell => $shell,
      passwd =>
      uid => $uid,
      gid => $gid,
      groups => $groups,
    }
  }

And as above:

 - only User[$name] or Group[$gid] will ever conflict (which is what we want)
 - there are no shared module dependencies forced into existence
 - we are not risking silly mistakes like shell =>
''/bin/bqsh'' or
   groups => [''weel'']


There''s more I could say, but I hope this gives the basic idea.


Cheers,

N

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

thanks for your elaborate design sketch.

Sorry for limiting my quote severely.

On 01/30/2012 06:28 PM, Nick wrote:
>   +package { ''libfoo'': ensure => present }
Is this different from John''s "constraint" proposal?

To me this didn''t become clear: Does the manifest still need to declare
an actual package { "libfoo" } somewhere, or is this implied by at
least
one assertion regarding any of its parameters?

Of the latter is the case, then this is not different from just allowing
puppet to consume multiple declarations of the same resource, along with
all the oft-discussed difficulties.

If instead, there still is that one central resource declaration
somewhere, I''m pretty sure this is the same as constraints.

Which is probably a really neat idea.

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 30/01/12 17:48, Felix Frank wrote:
> Hi,
> 
> thanks for your elaborate design sketch.
> 
> Sorry for limiting my quote severely.
> 
> On 01/30/2012 06:28 PM, Nick wrote:
>>   +package { ''libfoo'': ensure => present }
> 
> Is this different from John''s "constraint" proposal?
It did sound similar, yes - but unless I misunderstand it, not identical.  For
example, I don''t understand how Constraints would avoid the problems
with
unifying resources that Nan mentioned.

John''s example appeared to be wrapping an existing Resource with
something which
puts constraints on it, i.e. a layer on top of "Resources". It implies
a regular
Resource to wrap somewhere.

Whereas what I had in mind was something which in principle at least, was more
basic than a Resource.  With an "Assertion" there is nothing being
managed, or
mutated, by definition. It defines conditions on a resource (lower-case
''r'')
which can be checked, and merged,  but doesn''t imply that any Resource
(upper-case ''R'') need to be declared.  It''s quite
possible that one wouldn''t
bother, if you don''t need to manage or mutate anything.

So Resources (upper case ''R'') could be thought of as
extensions to Assertions
which also supply rules to mutate a system''s state, should the
conditions of the
Assertion not be met, so that the conditions *are* met.
> To me this didn''t become clear: Does the manifest still need to
declare
> an actual package { "libfoo" } somewhere, or is this implied by
at least
> one assertion regarding any of its parameters?
To be explicit: if an Assertion "+package { libfoo: }" is declared, it
just
means "libfoo must be installed for this manifest to work". I
don''t think it
needs to mandate a declaration of a full-blown "package { libfoo: }"
somewhere.

In fact, I can probably imagine circumstances when something might be invoked
which indirectly takes care of the "libfoo" package (or file, or
whatever) - and
then being forced to manage the "libfoo" package in Puppet just
because you want
to assert its presence could be a liability.


N

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

On 01/30/2012 10:28 PM, Nick wrote:
> It did sound similar, yes - but unless I misunderstand it, not identical. 
For
> example, I don''t understand how Constraints would avoid the
problems with
> unifying resources that Nan mentioned.
as far as I understand, there is no need to merge anything. The catalog
will or will not contain a certain resource. If existing, the resource
will have a certain set of parameteres and metaparameters. Each
constraint can be individually compared to this state. If a constrained
resource does not exist, or any of its (meta)parameters deviate from
what the constraint defines, the catalog is no longer valid.

The beauty of this design is that the language is very expressive and
all validation can be done by the master.

Err, right, John? :-)
> John''s example appeared to be wrapping an existing Resource with
something which
> puts constraints on it, i.e. a layer on top of "Resources". It
implies a regular
> Resource to wrap somewhere.
> 
> Whereas what I had in mind was something which in principle at least, was
more
> basic than a Resource.  With an "Assertion" there is nothing
being managed, or
> mutated, by definition. It defines conditions on a resource (lower-case
''r'')
> which can be checked, and merged,  but doesn''t imply that any
Resource
> (upper-case ''R'') need to be declared.  It''s
quite possible that one wouldn''t
> bother, if you don''t need to manage or mutate anything.
Ah, so you''d have the agent verify if all assertions (which need to
appear as first-class citizens in the catalog) hold true, and otherwise
fail the catalog?

That strikes me as very elegant indeed.

How will situations be handled where assertions won''t hold true until
parts of the catalog have been applied?
> So Resources (upper case ''R'') could be thought of as
extensions to Assertions
> which also supply rules to mutate a system''s state, should the
conditions of the
> Assertion not be met, so that the conditions *are* met.
Let''s not alienate the community by declassing the proven and beloved
Resources ;-) but I''ve got to say, this idea does hold merit.

So does the constraint idea. Something tells me that both might be of
benefit, but I''m afraid of years of user confusion to come when
everyone
is tasked with understanding the difference between the two and to
decide when to use which.

If we need to take a pick, there''s two things I''d have to say
for
constraints:
1. They''re more closely tailored to the problem at hand.
2. They''re stronger in chorus with what puppet is today.

Assertions would probably widen the borders of what''s possible with
puppet (and how easy it is), and they would allow/require us to part
with some paradigms. I''m torn whether we want this sooner than seeing
the multiple declaration problem sorted out in a less intrusive way.

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 31/01/12 09:01, Felix Frank wrote:
> Ah, so you''d have the agent verify if all assertions (which need
to
> appear as first-class citizens in the catalog) hold true, and otherwise
> fail the catalog?
> 
> That strikes me as very elegant indeed.
> 
> How will situations be handled where assertions won''t hold true
until
> parts of the catalog have been applied?
I''m not familiar enough with Puppet''s internals to answer that
very confidently.
My guess is that one might be able to express requires/before relationships
between Assertions and Resources in order to enforce things like this.

The main implication of that would be to restrict the freedom to assume that
assertions can be applied in any order, because the agent''s application
of the
catalog would need to be split into a sequence of "assertion" and
"mutation"
steps.  An Assertion must then not be moved outside the part of the sequence
assigned to it.

N

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 31, 3:01 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:
> Hi,
>
> On 01/30/2012 10:28 PM, Nick wrote:
>
> > It did sound similar, yes - but unless I misunderstand it, not
identical.  For
> > example, I don''t understand how Constraints would avoid the
problems with
> > unifying resources that Nan mentioned.
>
> as far as I understand, there is no need to merge anything. The catalog
> will or will not contain a certain resource.

That is correct.  Constraints do not have their own separate Resource
instances.  Instead, they use resource type and title, or perhaps a
resource reference, to identify a resource to which they apply.
Traditional Puppet rules are in effect -- there is never more than one
Resource per catalog for each (type, title) pair.

> If existing, the resource
> will have a certain set of parameteres and metaparameters. Each
> constraint can be individually compared to this state. If a constrained
> resource does not exist, or any of its (meta)parameters deviate from
> what the constraint defines, the catalog is no longer valid.

And that''s getting around to the interesting part.  Constraints could
leave it there, and that would be a good and useful solution.
Something like that could even be implemented in 2.7, as it introduces
no incompatibility whatever.

I am proposing that constraints might go even further, however, in one
or both of these ways:

1) Where the resource definition to which one or more constraints
apply does not explicitly specify a value for a constrained parameter
or property, applicable constraints are used to *choose* a value
(rather than allowing that property to take a default value).  This
would help with the problem of writing compatible resource definitions
in the first place.  Example:

Somewhere:
----
package { ''foo'':
  provider => ''yum''
}


Somewhere else:
----
constraint { ''my-package-foo-ensure'':
   resource => Package[''foo''],
   property => ''ensure'',
   forbidden_values => [''absent'',
''purged''],
   preferred_values => ''latest''
}


Yet another other place:
----
constraint { ''your-package-foo-provider'':
   resource => Package[''foo''],
   property => ''provider'',
   allowed_values => ''yum'',
}


The definition of Package[''foo''] explicitly specifies the
''provider''
property, so Constraint[''your-package-foo-provider''] only
validates
it, failing catalog compilation if the value does not satisfy the
constraint.  The definition does not explicitly specify a value for
the ''ensure'' property, however, so Puppet *chooses* one that
is
mutually agreeable to the relevant constraints.  In this case that
would work out to ''latest''.  If there is no mutually agreeable
value
then catalog compilation fails.


2) Extending (1), if constraints are declared for a resource that is
not otherwise itself declared for a node, then Puppet synthesizes a
declaration, using the applicable contraints to set values for all
constrained properties.  This could be considered a create_resources()
on steroids.


These would make Constraints much more useful and powerful because
they lessen the burden on module users to write suitable resource
declarations.  Instead of merely testing after the fact whether an
appropriate Resource was declared, item (1) would allow modules to
*collaborate* to ensure that suitable Resources are declared, without
requiring those modules to have any knowledge of one another.  Item
(2) would further reduce the burden on module users by alleviating any
need for them to explicitly declare modules'' needed resources
anywhere, though they could provide such declarations if they wished,
subject to the same rules that apply already.

> The beauty of this design is that the language is very expressive and
> all validation can be done by the master.
>
> Err, right, John? :-)

Indeed, the language is not only expressive, but also completely
compliant with current DSL syntax.  I''m not sure how magical the
implementation of the Constraint [meta]resource type would need to be,
but at least in principle I think it fits very cleanly into Puppet''s
model.  Furthermore, because all the work is done on the master,
*nothing* about the agent or any provider needs to change.  Probably
no resource type implementation needs to change, either.

> > John''s example appeared to be wrapping an existing Resource
with something which
> > puts constraints on it, i.e. a layer on top of "Resources".
It implies a regular
> > Resource to wrap somewhere.
>
> > Whereas what I had in mind was something which in principle at least,
was more
> > basic than a Resource.  With an "Assertion" there is nothing
being managed, or
> > mutated, by definition. It defines conditions on a resource
(lower-case ''r'')
> > which can be checked, and merged,  but doesn''t imply that any
Resource
> > (upper-case ''R'') need to be declared.  It''s
quite possible that one wouldn''t
> > bother, if you don''t need to manage or mutate anything.
>
> Ah, so you''d have the agent verify if all assertions (which need
to
> appear as first-class citizens in the catalog) hold true, and otherwise
> fail the catalog?

And here you have hit on some of my main issues with Assertions, as I
understand that proposal:

1) Assertions are tested by the agent (or else they do nothing that
constraints do not also do), so they need to be represented in the
catalog.  This makes them no lighter-weight than ordinary Resources as
far as I can tell.

2) Supposing that Assertions are indeed tested by the agent, they
defer catalog failures to the agent, which makes module interoperation
(the issue we''re trying do deal with) harder to test.

3) Assertions leave the burden on module users to provide resource
declarations that are agreeable to all modules.

> That strikes me as very elegant indeed.

I''m not persuaded that it is a broadly useful capability.  I
acknowledge that there may be cases where one wants the agent to
perform a test without changing anything.  At present, one would
probably use an Exec for the purpose.  I tend to think, however, that
such cases usually reflect module design shortcomings.

The Puppet paradigm is to tell the agent how a node should look, then
let the agent make it so.  Why, then, would it be desirable to give
the agent contradictory node descriptions?  Why would it be desirable
to give the agent a node description, but not allow it to make the
node meet the description?  How would either capability further the
goal of module interoperability?  Especially, why should an
interoperability solution that relies on such behaviors be preferred
to one that does not?

Assertions would be an interoperability aid for one of the same
reasons that Constraints would be: they would allow modules to express
resource requirements without actually declaring resources.  This is a
better solution than merging multiple Resource declarations would be
because modules'' requirements are expressed precisely, and also
because there is no change to the invariants of Puppet''s data model.
In these respects, however, Assertions offer nothing that Constraints
do not also offer.

> How will situations be handled where assertions won''t hold true
until
> parts of the catalog have been applied?

That, too.  Assertions would need their own support for resource
relationships to adequately address that issue.  At this point they
don''t look lightweight at all to me; rather they look like a full-
blown new [meta]resource type (as Constraints also would be).
Implementation of Assertions separate from Resources, as Nick
suggests, would be tricky.  The Assertion provider would need to
somehow hook into all other providers, including custom ones, to test
whether assertions were satisfied.

> > So Resources (upper case ''R'') could be thought of as
extensions to Assertions
> > which also supply rules to mutate a system''s state, should
the conditions of the
> > Assertion not be met, so that the conditions *are* met.
>
> Let''s not alienate the community by declassing the proven and
beloved
> Resources ;-) but I''ve got to say, this idea does hold merit.

I think the best chance of integrating something like Assertions into
Puppet is to flip that around: assertion nature could be an attribute
of (some) Resource instances.  Consider its similarities to the
effects of the ''audit'' metaparameter, for instance.  Only in
that way
can I reconcile Nick''s characterization of Assertions as
"lightweight"
and "low-level" with the apparent design and implementation
requirements.

> So does the constraint idea. Something tells me that both might be of
> benefit, but I''m afraid of years of user confusion to come when
everyone
> is tasked with understanding the difference between the two and to
> decide when to use which.
>
> If we need to take a pick, there''s two things I''d have to
say for
> constraints:
> 1. They''re more closely tailored to the problem at hand.
> 2. They''re stronger in chorus with what puppet is today.
>
> Assertions would probably widen the borders of what''s possible
with
> puppet (and how easy it is), and they would allow/require us to part
> with some paradigms. I''m torn whether we want this sooner than
seeing
> the multiple declaration problem sorted out in a less intrusive way.

There may well be a better solution for module interoperability than
Constraints, but Assertions (as I understand them) are not it.  I can,
however, imagine using Assertions similarly to how I use the like-
named features of C and Java -- to validate assumptions and internal
consistency in catalogs.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.