Puppet users - Jun 2013 - defined() implicitly requiring resource reference

I''m testing a ''cleanup'' stage which runs after
Stage[main] and removes a bunch of package resources.

To do this, I tried a simple check of defined(Package[<foo>]) combined
with a custom facter fact (called ''app_packages''):
> class app::package::cleaner {
> 
>   define check_and_remove {
>     if !defined(Package[$title]) {
>       package { $title:
>         ensure => absent
>       }
>     }
>   }
> 
>   $apps = split($::app_packages, '','')
>   check_and_remove { $apps: }
> 
> }
> 
> node ''foo'' {
>   class { ''app::package::cleaner'': stage =>
''cleanup'' }
> }
Unfortunately, this results in a dependency cycle.  It appears that putting the
Package[$title] resource reference in defined() actually invokes an implicit
dependency between my cleanup helper resource in the cleanup stage and the
original Package resource in the main stage.
> Augeas[redacted] => Service[iptables] => Class[Iptables] =>
Stage[main] => Stage[cleanup] => Class[App::Package::Cleaner] =>
App::Package::Cleaner::Check_and_remove[package-434] => Package[package-434]
=> Exec[app-graceful-restart] => Class[App] => Stage[main]
Is this implicit dependency expected behaviour or am I doing something Bad(tm)?

Tom

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Tuesday, June 4, 2013 1:22:08 AM UTC-5, Tom Lanyon
wrote:
>
> I''m testing a ''cleanup'' stage which runs after
Stage[main] and removes a
> bunch of package resources. 
>
> To do this, I tried a simple check of defined(Package[<foo>])
combined
> with a custom facter fact (called ''app_packages''): 
>
> > class app::package::cleaner { 
> > 
> >   define check_and_remove { 
> >     if !defined(Package[$title]) { 
> >       package { $title: 
> >         ensure => absent 
> >       } 
> >     } 
> >   } 
> > 
> >   $apps = split($::app_packages, '','') 
> >   check_and_remove { $apps: } 
> > 
> > } 
> > 
> > node ''foo'' { 
> >   class { ''app::package::cleaner'': stage =>
''cleanup'' }
>
> > } 
>
> Unfortunately, this results in a dependency cycle.  It appears that 
> putting the Package[$title] resource reference in defined() actually 
> invokes an implicit dependency between my cleanup helper resource in the 
> cleanup stage and the original Package resource in the main stage. 
>
> > Augeas[redacted] => Service[iptables] => Class[Iptables] =>
Stage[main]
> => Stage[cleanup] => Class[App::Package::Cleaner] => 
> App::Package::Cleaner::Check_and_remove[package-434] => 
> Package[package-434] => Exec[app-graceful-restart] => Class[App]
=>
> Stage[main] 
>
>
Does it do that when Package[package-434] is already declared elsewhere, or 
only when it is not?

 
> Is this implicit dependency expected behaviour or am I doing something 
> Bad(tm)? 
>
>
Both.

Supposing that the target package is not declared elsewhere (so that the 
!defined() condition is true) the definition will declare the package 
itself to ensure it absent, and in that case you would expect a 
relationship between the defined-type instance and the resource declared by 
it.  If elsewhere you have specific references to that package, applicable 
resource parameter defaults, or collectors that will match that package, 
then you can get relationships with it that are not evident from the 
defined type body.

On the other hand, defined() is evil.  Do not use it.  Ever.  I usually 
attribute its malignancy to the parse-order dependency it inherently 
creates -- which is indeed a serious problem -- but in this case I think 
trying to use it to approach your problem it has also obfuscated your 
manifests enough to confuse you about the scope and nature of some of your 
other declarations.

Instead of using defined(), you can apply logic farther upstream to make 
the correct declaration in the first (one) place or to apply resource 
parameter overrides to the correct resources.  Alternatively, you can 
simply determine by other means what packages need to be ensured absent, 
such as by filtering a list of possible packages against a list of packages 
that are supposed to be installed.  Some of those options may still 
susceptible to the problem you observed, however, if relevant relationships 
spring from declarations elsewhere, as I described they may do.

For the record, however, no order-of-application relationship should be 
implied by the reference itself.  Therefore, when the referenced Package is 
declared elsewhere (so that the !defined() condition is false), there 
should be a relationship between 
App::Package::Cleaner::Check_and_remove[foo] and Package[foo] only if that 
relationship is declared somewhere else.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi John,

Thanks for the reply.

On 05/06/2013, at 12:33 AM, jcbollinger <John.Bollinger@stJude.org>
wrote:
> On Tuesday, June 4, 2013 1:22:08 AM UTC-5, Tom Lanyon wrote:
> I''m testing a ''cleanup'' stage which runs after
Stage[main] and removes a bunch of package resources.
> 
> To do this, I tried a simple check of defined(Package[<foo>])
combined with a custom facter fact (called ''app_packages''):
> 
> > class app::package::cleaner { 
> > 
> >   define check_and_remove { 
> >     if !defined(Package[$title]) { 
> >       package { $title: 
> >         ensure => absent 
> >       } 
> >     } 
> >   } 
> > 
> >   $apps = split($::app_packages, '','') 
> >   check_and_remove { $apps: } 
> > 
> > } 
> > 
> > node ''foo'' { 
> >   class { ''app::package::cleaner'': stage =>
''cleanup'' }
> 
> > } 
> 
> Unfortunately, this results in a dependency cycle.  It appears that putting
the Package[$title] resource reference in defined() actually invokes an implicit
dependency between my cleanup helper resource in the cleanup stage and the
original Package resource in the main stage.
> 
> > Augeas[redacted] => Service[iptables] => Class[Iptables] =>
Stage[main] => Stage[cleanup] => Class[App::Package::Cleaner] =>
App::Package::Cleaner::Check_and_remove[package-434] => Package[package-434]
=> Exec[app-graceful-restart] => Class[App] => Stage[main]
> 
>> Does it do that when Package[package-434] is already declared
elsewhere, or only when it is not?
Sorry, I should have been clearer that this occurs when Package[package-434] IS
declared elsewhere. "!defined(Package[package-434])" therefore is
false, so just by referencing the existing declaration within the defined() call
it seems to incite an implicit dependency.

>> Is this implicit dependency expected behaviour or am I doing something
Bad(tm)?
> 
> Both.
> 
> Supposing that the target package is not declared elsewhere (so that the
!defined() condition is true) the definition will declare the package itself to
ensure it absent, and in that case you would expect a relationship between the
defined-type instance and the resource declared by it.  If elsewhere you have
specific references to that package, applicable resource parameter defaults, or
collectors that will match that package, then you can get relationships with it
that are not evident from the defined type body.
> 
> On the other hand, defined() is evil.  Do not use it.  Ever.
I had this discussion with someone on #puppet IRC earlier and they ended up with
"Oh, in your case, defined() is probably actually what you want."
>  I usually attribute its malignancy to the parse-order dependency it
inherently creates -- which is indeed a serious problem -- but in this case I
think trying to use it to approach your problem it has also obfuscated your
manifests enough to confuse you about the scope and nature of some of your other
declarations.
> 
> Instead of using defined(), you can apply logic farther upstream to make
the correct declaration in the first (one) place or to apply resource parameter
overrides to the correct resources.  Alternatively, you can simply determine by
other means what packages need to be ensured absent, such as by filtering a list
of possible packages against a list of packages that are supposed to be
installed.  Some of those options may still susceptible to the problem you
observed, however, if relevant relationships spring from declarations elsewhere,
as I described they may do.
I''ve tried this other ways, but here''s an example of why
farther upstream logic doesn''t work:

define myapp ($requested_package){

  package { $requested_package:
    ensure => present
  }

  define package_cleanup {
    $installed_package = $title

    if $installed_package != $requested_package {
      package { $installed_package:
        ensure => purged
      }
    }
  }

  # assuming a facter fact named ''installed_packages''
  package_cleanup { split($::installed_packages, '',''): }
}

# now in the case of:
#   $::installed_packages = ''one,two,three''
# with:
myapp { ''oneA'': requested_package =>
''one'' }
myapp { ''twoA'': requested_package =>
''two'' }
myapp { ''oneB'': requested_package =>
''one'' }

# we''d end up with package conflicts because
# Myapp[oneA] will define Package[one] (present)
# then define Package[two], Package[three] (absent),
# and Myapp[twoA] will try and define Package[two]
# (present) and fail with a non-uniqueness error.

I don''t see how this is doable without defined() or some other check of
the catalog to see what packages are "needed" elsewhere.  Do you have
any suggestions?

> For the record, however, no order-of-application relationship should be
implied by the reference itself.  Therefore, when the referenced Package is
declared elsewhere (so that the !defined() condition is false), there should be
a relationship between App::Package::Cleaner::Check_and_remove[foo] and
Package[foo] only if that relationship is declared somewhere else.
I''d hoped that using a Stage to run after everything else would sort
this all out, but I''m now not sure this is correct and also see your
earlier post about never using stages [with this and "defined() is
evil", which pieces of Puppet /can/ we use safely?].

Thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Tuesday, June 4, 2013 6:01:58 PM UTC-5, Tom Lanyon
wrote:
>
> Hi John, 
>
> Thanks for the reply. 
>
> On 05/06/2013, at 12:33 AM, jcbollinger wrote: 
> > On Tuesday, June 4, 2013 1:22:08 AM UTC-5, Tom Lanyon wrote: 
> > 
> > Unfortunately, this results in a dependency cycle.  It appears that 
> putting the Package[$title] resource reference in defined() actually 
> invokes an implicit dependency between my cleanup helper resource in the 
> cleanup stage and the original Package resource in the main stage. 
> > 
> > > Augeas[redacted] => Service[iptables] => Class[Iptables]
=>
> Stage[main] => Stage[cleanup] => Class[App::Package::Cleaner] => 
> App::Package::Cleaner::Check_and_remove[package-434] => 
> Package[package-434] => Exec[app-graceful-restart] => Class[App]
=>
> Stage[main] 
> > 
> >> Does it do that when Package[package-434] is already declared 
> elsewhere, or only when it is not? 
>
> Sorry, I should have been clearer that this occurs when 
> Package[package-434] IS declared elsewhere. 
> "!defined(Package[package-434])" therefore is false, so just by
referencing
> the existing declaration within the defined() call it seems to incite an 
> implicit dependency. 
>
>
If that''s really what''s happening then you should be able to
create a
simple test case that demonstrates it.  That would be a worthy subject for 
a bug report.

 
>
> >> Is this implicit dependency expected behaviour or am I doing
something
> Bad(tm)? 
> > 
> > Both. 
> > 
> > Supposing that the target package is not declared elsewhere (so that
the
> !defined() condition is true) the definition will declare the package 
> itself to ensure it absent, and in that case you would expect a 
> relationship between the defined-type instance and the resource declared by
> it.  If elsewhere you have specific references to that package, applicable 
> resource parameter defaults, or collectors that will match that package, 
> then you can get relationships with it that are not evident from the 
> defined type body. 
> > 
> > On the other hand, defined() is evil.  Do not use it.  Ever. 
>
> I had this discussion with someone on #puppet IRC earlier and they ended 
> up with "Oh, in your case, defined() is probably actually what you
want."
>

No.  defined() is never what you want.  It may at times seem expedient, but 
it''s bad news every time.

 
>
> >  I usually attribute its malignancy to the parse-order dependency it 
> inherently creates -- which is indeed a serious problem -- but in this case
> I think trying to use it to approach your problem it has also obfuscated 
> your manifests enough to confuse you about the scope and nature of some of 
> your other declarations. 
> > 
> > Instead of using defined(), you can apply logic farther upstream to
make
> the correct declaration in the first (one) place or to apply resource 
> parameter overrides to the correct resources.  Alternatively, you can 
> simply determine by other means what packages need to be ensured absent, 
> such as by filtering a list of possible packages against a list of packages
> that are supposed to be installed.  Some of those options may still 
> susceptible to the problem you observed, however, if relevant relationships
> spring from declarations elsewhere, as I described they may do. 
>
> I''ve tried this other ways, but here''s an example of why
farther upstream
> logic doesn''t work: 
>
> define myapp ($requested_package){ 
>
>   package { $requested_package: 
>     ensure => present 
>   } 
>
>   define package_cleanup { 
>     $installed_package = $title 
>
>     if $installed_package != $requested_package { 
>       package { $installed_package: 
>         ensure => purged 
>       } 
>     } 
>   } 
>
>   # assuming a facter fact named ''installed_packages'' 
>   package_cleanup { split($::installed_packages, '',''): } 
> } 
>

I don''t much like that general approach in the first place on account
of
the $requested_package parameter.  That you encounter difficulty when you 
try something a bit dodgy should not be surprising.

 
>
> # now in the case of: 
> #   $::installed_packages = ''one,two,three'' 
> # with: 
> myapp { ''oneA'': requested_package =>
''one'' }
> myapp { ''twoA'': requested_package =>
''two'' }
> myapp { ''oneB'': requested_package =>
''one'' }
>
> # we''d end up with package conflicts because 
> # Myapp[oneA] will define Package[one] (present) 
> # then define Package[two], Package[three] (absent), 
> # and Myapp[twoA] will try and define Package[two] 
> # (present) and fail with a non-uniqueness error. 
>
> I don''t see how this is doable without defined() or some other
check of
> the catalog to see what packages are "needed" elsewhere.  Do you
have any
> suggestions? 
>
>
In fact, despite my dissatisfaction with your approach, you can indeed do 
this without defined(), and without even disrupting your current structure 
very much.  Here''s one way I think would work:

# This class ensures all known app packages are
# by default purged
class app::packages {
    $apps = split($::app_packages, '','') 
    package { $apps:
        ensure => ''purged''
    }
}

# Overrides the requested package to be declared
# present instead of purged.
define app::myapp($requested_package) {
    include ''app::packages''
    Package<| title == $requested_package |> {
        ensure => ''present''
    }
}

# no separate package_cleanup required

> > For the record, however, no order-of-application relationship should
be
> implied by the reference itself.  Therefore, when the referenced Package is
> declared elsewhere (so that the !defined() condition is false), there 
> should be a relationship between 
> App::Package::Cleaner::Check_and_remove[foo] and Package[foo] only if that 
> relationship is declared somewhere else. 
>
> I''d hoped that using a Stage to run after everything else would
sort this
> all out, but I''m now not sure this is correct and also see your
earlier
> post about never using stages [with this and "defined() is evil",
which
> pieces of Puppet /can/ we use safely?]. 
>
>
It''s not uncommon for stages to cause cycles, but they will not resolve
cycles that are already present without them, because stages only add 
relationships to those that are otherwise there.  They are not evil in the 
same way that defined() is, however.  Although I disfavor stages myself and 
generally do not recommend them to folks, I do not consider a manifest set 
that successfully uses them to be inherently flawed (unlike one that relies 
on defined()).

Puppet is broad in scope and rich in features, so it should not be 
surprising that a few of its features are of questionable value.  Pretty 
much any system of comparable size has such.  To Puppetlabs''s credit,
they
do deprecate and eventually remove features that they conclude are obsolete 
or irretrievably broken.  They have floated the idea of doing this with 
defined(), in fact, and that may eventually happen.

Other things I recommend you avoid:

   - most uses of ''import''.  Bringing node definitions into
site.pp is
   about the only appropriate use of ''import'' that comes to
mind.  Most other
   potential uses are better served by the autoloader.
   - Lexical nesting of classes and/or defined types.  This is partly for 
   cooperating with the autoloader and partly just a matter of style, but it 
   is widely considered a best practice to put each class and definition in 
   its own file.
   - parameterized-style class declarations.  (That is, declarations of the 
   form "class { ''foo'': param1 => bar }".)  This
one is controversial.  I
   have written on it frequently in the past, so I will omit the recap unless 
   you insist.  Parameterized classes themselves are ok in Puppet 3, provided 
   that you rely on hiera data bindings or maybe an ENC when you need to 
   customize their parameters.

John

"Parameterized classes: the best thing since decaffeinated coffee"

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi John,

Sorry for the delayed reply.

On 05/06/2013, at 11:51 PM, jcbollinger <John.Bollinger@stJude.org>
wrote:
> 
>> Sorry, I should have been clearer that this occurs when
Package[package-434] IS declared elsewhere.
"!defined(Package[package-434])" therefore is false, so just by
referencing the existing declaration within the defined() call it seems to
incite an implicit dependency.
>> 
> If that''s really what''s happening then you should be able
to create a simple test case that demonstrates it.  That would be a worthy
subject for a bug report.
I''ll see what I can do.
>>>> Is this implicit dependency expected behaviour or am I doing
something Bad(tm)?
>>> 
>>> Both. 
>>> 
>>> Supposing that the target package is not declared elsewhere (so
that the !defined() condition is true) the definition will declare the package
itself to ensure it absent, and in that case you would expect a relationship
between the defined-type instance and the resource declared by it.  If elsewhere
you have specific references to that package, applicable resource parameter
defaults, or collectors that will match that package, then you can get
relationships with it that are not evident from the defined type body.
>>> 
>>> On the other hand, defined() is evil.  Do not use it.  Ever. 
>> 
>> I had this discussion with someone on #puppet IRC earlier and they
ended up with "Oh, in your case, defined() is probably actually what you
want."
> 
> No.  defined() is never what you want.  It may at times seem expedient, but
it''s bad news every time.
OK, understood.
>> define myapp ($requested_package){ 
>> 
>>  package { $requested_package: 
>>    ensure => present 
>>  } 
>> 
>>  define package_cleanup { 
>>    $installed_package = $title 
>> 
>>    if $installed_package != $requested_package { 
>>      package { $installed_package: 
>>        ensure => purged 
>>      } 
>>    } 
>>  } 
>> 
>>  # assuming a facter fact named ''installed_packages'' 
>>  package_cleanup { split($::installed_packages, '',''):
}
>> } 
> 
> I don''t much like that general approach in the first place on
account of the $requested_package parameter.  That you encounter difficulty when
you try something a bit dodgy should not be surprising.
Can you explain this further so I can understand the issue?


> In fact, despite my dissatisfaction with your approach, you can indeed do
this without defined(), and without even disrupting your current structure very
much.  Here''s one way I think would work:
> 
> # This class ensures all known app packages are
> # by default purged
> class app::packages {
>    $apps = split($::app_packages, '','') 
>    package { $apps:
>        ensure => ''purged''
>    }
> }
> 
> # Overrides the requested package to be declared
> # present instead of purged.
> define app::myapp($requested_package) {
>    include ''app::packages''
>    Package<| title == $requested_package |> {
>        ensure => ''present''
>    }
> }
> 
> # no separate package_cleanup required

OK, I wondered whether we could do something like this however - forgive my
naivety - I still can''t see how this could be a complete solution
without something like defined().

As an example... your above snippet works fine to ensure already installed
packages remain installed, but what if we wanted to install a brand new version
of app::myapp?  Because a ''package'' resource with title
$requested_package does not yet exist, the Package<||> collector matches
no resources and the new package is not installed.  The only solution that I can
come up with is to check whether such a resource is already defined and, if not,
define one.

Your guidance is appreciated.

Tom

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Wednesday, June 12, 2013 9:15:22 AM UTC-5, Tom Lanyon
wrote:
>
> On 05/06/2013, at 11:51 PM, jcbollinger [...] wrote: 
> > I don''t much like that general approach in the first place on
account of
> the $requested_package parameter.  That you encounter difficulty when you 
> try something a bit dodgy should not be surprising. 
>
> Can you explain this further so I can understand the issue? 
>
>
Initially, it was mostly a gut feeling.  After having had time to step back 
from the issue and return to it fresh, I think it''s a combination of 
things, mostly revolving around what you''re actually modeling, and how 
you''re modeling it.

Basically, the ''myapp'' definition represents one package
chosen from a list
of mutually exclusive packages.  If that''s all it is, then its name is 
misleading -- it should be more generic -- and it should probably take the 
exclusive list as a second parameter.  On the other hand, if it is indeed 
supposed to be something specific, then it doesn''t take much advantage
of
that.  In particular -- and here''s where my previous comment came from
--
if it supposed to represent something specific to your application, then 
why doesn''t it know anything about the application''s package
names?

Also, if the point is supposed to be that only one version of the 
application can be installed at a time, and the definition is specific to 
that application, then it really ought to be a class instead.
 

> > In fact, despite my dissatisfaction with your approach, you can indeed
> do this without defined(), and without even disrupting your current 
> structure very much.  Here''s one way I think would work: 
> > 
> > # This class ensures all known app packages are 
> > # by default purged 
> > class app::packages { 
> >    $apps = split($::app_packages, '','') 
> >    package { $apps: 
> >        ensure => ''purged'' 
> >    } 
> > } 
> > 
> > # Overrides the requested package to be declared 
> > # present instead of purged. 
> > define app::myapp($requested_package) { 
> >    include ''app::packages'' 
> >    Package<| title == $requested_package |> { 
> >        ensure => ''present'' 
> >    } 
> > } 
> > 
> > # no separate package_cleanup required 
>
>
> OK, I wondered whether we could do something like this however - forgive 
> my naivety - I still can''t see how this could be a complete
solution
> without something like defined(). 
>
> As an example... your above snippet works fine to ensure already installed 
> packages remain installed, but what if we wanted to install a brand new 
> version of app::myapp?  Because a ''package'' resource with
title
> $requested_package does not yet exist, the Package<||> collector
matches no
> resources and the new package is not installed.  The only solution that I 
> can come up with is to check whether such a resource is already defined 
> and, if not, define one. 
>
>
You appear to have a serious misunderstanding.  Resource collectors have no 
direct relationship with or dependency on which resources are already 
installed on the target system.  They work exclusively with resource *
declarations* in your manifests, and they do so at catalog compilation 
time.  Moreover, they are independent of parse order (though the example 
anyway ensures a parse order that would work if collectors were parse-order 
dependent).

Explanation of the example:

   - class app::packages declares all of the possible application packages, 
   specifying the intended state for each one as ''purged''.  If
that is the
   only thing applied to the target node then it will cause the removal of 
   each and every one of those packages that is installed. 
(''purged'' is
   stronger than ''absent''.  The former is more sure to remove
the specified
   package, but the latter takes care to avoid causing any other packages to 
   be removed, and therefore fails if any other package depends on the target 
   package.)  It is necessary that the list of possible packages include every 
   one that you may want to have installed, so it needs to be updated whenever 
   you introduce a new one that you want to manage.  That was already a 
   requirement for you, however, whether you recognized it or not.
   - Resources of defined type app::myapp ensure that class app::packages 
   is declared on the target node (by declaring it itself)
   - Resources of defined type app::myapp override the declaration of the 
   target package (made by class app::packages) so that its target state is 
   ''present'' instead of ''purged''.  This will
cause it to be installed if it is
   not already present, and will avoid removing it if it is present.
   - To use the example, simply declare one or more instances of app::myapp 
   for the target node, either in a node block or in some class assigned to 
   the node.  You may also declare class app::packages directly for any node, 
   whether or not they declare any app::myapp instances.  That is useful if 
   you have nodes on which you want to ensure that no version of the 
   application is installed.
   

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi John

On 14/06/2013, at 12:06 AM, jcbollinger <John.Bollinger@stJude.org>
wrote:
>> 
>> On Wednesday, June 12, 2013 9:15:22 AM UTC-5, Tom Lanyon wrote:
>> On 05/06/2013, at 11:51 PM, jcbollinger [...] wrote: 
>> > I don''t much like that general approach in the first
place on account of the $requested_package parameter.  That you encounter
difficulty when you try something a bit dodgy should not be surprising.
>> 
>> Can you explain this further so I can understand the issue? 
> 
> Initially, it was mostly a gut feeling.  After having had time to step back
from the issue and return to it fresh, I think it''s a combination of
things, mostly revolving around what you''re actually modeling, and how
you''re modeling it.
> 
> Basically, the ''myapp'' definition represents one package
chosen from a list of mutually exclusive packages.  If that''s all it
is, then its name is misleading -- it should be more generic -- and it should
probably take the exclusive list as a second parameter.  On the other hand, if
it is indeed supposed to be something specific, then it doesn''t take
much advantage of that.  In particular -- and here''s where my previous
comment came from -- if it supposed to represent something specific to your
application, then why doesn''t it know anything about the
application''s package names?
> 
> Also, if the point is supposed to be that only one version of the
application can be installed at a time, and the definition is specific to that
application, then it really ought to be a class instead.
Alas, the whole intention of this is that multiple versions of the package must
be installed at the same time.  This is where the problem lies, in that we have
no way to clean up old unused versions once they''re no longer needed.

We have instances of an application MyApp, which utilise a shared Package in
many to one relationship:

    Package[one] ___ MyApp[app1]

    Package[two] ___ MyApp[app2]
                \___ MyApp[app3]

    Package[three] _ MyApp[app4]
                  \_ MyApp[app5]
                  \_ MyApp[app6]

In the above example, if we upgraded MyApp[app2] and MyApp[app3] to
Package[three], we''d still have Package[two] installed on all of the
hosts but no MyApp instances would be using it.  This is the unused Package
we''re trying to clean up with Puppet (there''s no longer a
MyApp resource definition which references Package[two]).

There is no mutually exclusive packages or any other such conflicts.

>> > In fact, despite my dissatisfaction with your approach, you can
indeed do this without defined(), and without even disrupting your current
structure very much.  Here''s one way I think would work:
>> > 
>> > # This class ensures all known app packages are 
>> > # by default purged 
>> > class app::packages { 
>> >    $apps = split($::app_packages, '','') 
>> >    package { $apps: 
>> >        ensure => ''purged'' 
>> >    } 
>> > } 
>> > 
>> > # Overrides the requested package to be declared 
>> > # present instead of purged. 
>> > define app::myapp($requested_package) { 
>> >    include ''app::packages'' 
>> >    Package<| title == $requested_package |> { 
>> >        ensure => ''present'' 
>> >    } 
>> > } 
>> > 
>> > # no separate package_cleanup required 
>> 
>> 
>> OK, I wondered whether we could do something like this however -
forgive my naivety - I still can''t see how this could be a complete
solution without something like defined().
>> 
>> As an example... your above snippet works fine to ensure already
installed packages remain installed, but what if we wanted to install a brand
new version of app::myapp?  Because a ''package'' resource with
title $requested_package does not yet exist, the Package<||> collector
matches no resources and the new package is not installed.  The only solution
that I can come up with is to check whether such a resource is already defined
and, if not, define one.
> 
> You appear to have a serious misunderstanding.  Resource collectors have no
direct relationship with or dependency on which resources are already installed
on the target system.  They work exclusively with resource declarations in your
manifests, and they do so at catalog compilation time.  Moreover, they are
independent of parse order (though the example anyway ensures a parse order that
would work if collectors were parse-order dependent).
No, I understand this.
> Explanation of the example:
> 	• class app::packages declares all of the possible application packages,
specifying the intended state for each one as ''purged''.  If
that is the only thing applied to the target node then it will cause the removal
of each and every one of those packages that is installed. 
(''purged'' is stronger than ''absent''.  The
former is more sure to remove the specified package, but the latter takes care
to avoid causing any other packages to be removed, and therefore fails if any
other package depends on the target package.)  It is necessary that the list of
possible packages include every one that you may want to have installed, so it
needs to be updated whenever you introduce a new one that you want to manage. 
That was already a requirement for you, however, whether you recognized it or
not.
This is where it fell apart for me before, I didn''t realise that you
assumed $::app_packages was every possible package;  in my first post
I''d specified that $::app_packages was a fact generated on each node
from the list of *currently installed* packages, so it contains all of the
packages which have been previously been installed but not packages which may be
installed in the future.

I think it''s going to be a managerial nightmare to update some such
list of all possible packages, so might look for an alternate solution.  I
don''t think Puppet really meets our needs here, unless there''s
a way we can programmatically parse out all of the versions of the package
specified in our manifests to include them in the $::app_packages (so they get
defined as ''purged'') before building the rest of the catalog
which would override them as ''present''.  This feels like
something that might be possible with the use of some combination of defined()
or stages, but I''m keen to not run up against any caveats
you''ve mentioned with those features.
> 	• Resources of defined type app::myapp ensure that class app::packages is
declared on the target node (by declaring it itself)
> 	• Resources of defined type app::myapp override the declaration of the
target package (made by class app::packages) so that its target state is
''present'' instead of ''purged''.  This will
cause it to be installed if it is not already present, and will avoid removing
it if it is present.
> 	• To use the example, simply declare one or more instances of app::myapp
for the target node, either in a node block or in some class assigned to the
node.  You may also declare class app::packages directly for any node, whether
or not they declare any app::myapp instances.  That is useful if you have nodes
on which you want to ensure that no version of the application is installed.
> 
> John
Thanks for your help.

Cheers,
Tom

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Friday, June 14, 2013 2:01:50 AM UTC-5, Tom Lanyon
wrote:
>
> Hi John 
>
> On 14/06/2013, at 12:06 AM, jcbollinger [...] wrote: 
>
> This is where it fell apart for me before, I didn''t realise that
you
> assumed $::app_packages was every possible package;  in my first post
I''d
> specified that $::app_packages was a fact generated on each node from the 
> list of *currently installed* packages, so it contains all of the packages 
> which have been previously been installed but not packages which may be 
> installed in the future. 
>

My apologies for the confusion.

 
>
> I think it''s going to be a managerial nightmare to update some
such list
> of all possible packages, so might look for an alternate solution.  I
don''t
> think Puppet really meets our needs here, unless there''s a way we
can
> programmatically parse out all of the versions of the package specified in 
> our manifests to include them in the $::app_packages (so they get defined 
> as ''purged'') before building the rest of the catalog
which would override
> them as ''present''.


I think you''re looking at it the wrong way around.  It is a fundamental
problem for you if you don''t know what all the packages you (may) want
to
manage are.  This is not information that ought to be gleaned from your 
manifest set, nor even information that you ought to be collecting 
dynamically from clients, but rather information which you ought to have a 
priori.  How you store it and keep it updated is a separate matter that I 
have heretofore avoided, but Puppet does have a capable external data 
interface, Hiera, that could reasonably be applied to the problem.

Nevertheless, if it is not feasible for you to determine up front all the 
packages your apps may be using or may want to use, then you can get by 
with a smaller list, consisting of just the union of those packages 
currently installed and those you want to be installed.  You already have 
those data at hand, and it is possible to merge them into the needed list.  
For example, rewriting my app::packages this way should help:

class app::packages {
    $apps = unique(concat(split($::app_packages, '',''),
<wanted_packages>))
    package { $apps:
        ensure => ''purged''
    }
}

That might be paired with app::myapp declarations there or elsewhere of the 
form

app::myapp { <wanted_packages>: }

The <wanted_packages> represents an array of the names of the packages you
want installed, which could be a variable (including a parameter) of some 
class in the ''app'' module (even app::packages itself), or
might be a
function call (e.g. to an external data service) or maybe something else.  
The unique() and concat() functions are available from PuppetLabs''s
popular
"stdlib" add-in module.  


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Friday, June 14, 2013 9:14:07 AM UTC-5, jcbollinger
wrote:
>
> Nevertheless, if it is not feasible for you to determine up front all the 
> packages your apps may be using or may want to use, then you can get by 
> with a smaller list, consisting of just the union of those packages 
> currently installed and those you want to be installed.
>

Do note, by the way, that this general approach of relying on a dynamic, 
client-supplied list of packages that may need to be removed is inherently 
more risky than one based on a centrally managed, static list of packages.  
If your custom fact malfunctions (or is subverted) then you could end up 
removing packages you did not intend to remove, including packages 
essential to the client''s operation.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.