Puppet users - Aug 2013 - node definition too big in mutualized hosting context

Hi,

I use puppet to manage servers in a mutualized hosting context and I get 
some trouble concerning the node definitions.

For example on dns nodes I have to declare many `bind::zone` resources like 
this:

bind::zone { [''website1.com'']:
  expire => 604800,
  minimum => 3600,
  ttl => 38400,
  …
  records => [
    ''www        IN    A    xxx.xxx.xxx.xxx''
    …
        ]
}

bind::zone { [''website2.com'']:
  expire => 604800,
  minimum => 3600,
  ttl => 38400,
  …
  records => [
    ''www        IN    A    xxx.xxx.xxx.xxx''
    …
        ]
}

… and hundred others like this.

Finally, the file containing node definition became very big and difficult 
to maintain.
I have the same issue for a webserver with apache::vhost resources for 
example.

For the moment I use a workaround like this:

import vhosts/*.pp

And in each files I put a condition to make a restriction for a particular 
nodes like this:

if $::fqdn =~ /^web(\d*)\.example\.net$/ {
    apache::vhost { ''website2'':
    …
    }
}

I suppose that many puppet users use puppet to manage nodes in mutualized 
hosting context and I would ask if someone have a better solution for this 
issue?

Regards,

Thomas

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Assuming all the dns servers are getting the same config, We break big chunks
like that out into their own modules. So:

node dns1.myco.com {
  include myco::dns
}

node web1.myco.com {
  include myco::web
}

If the servers have different configs, you can make myco::dns::common,
myco::dns::this myco::dns::that myco::dns::other.

We also use a "role" fact instead of a regex on the fqdn for case
statements, but even those are fragile. I prefer to explicitly include things in
the node definitions. It''s a bit verbose, but its also clear for anyone
else that has to maintain the code.

— Mason Turner

On Aug 7, 2013, at 9:55 AM, mezcalito38@gmail.com wrote:
> Hi,
> 
> I use puppet to manage servers in a mutualized hosting context and I get
some trouble concerning the node definitions.
> 
> For example on dns nodes I have to declare many `bind::zone` resources like
this:
> 
> bind::zone { [''website1.com'']:
>   expire => 604800,
>   minimum => 3600,
>   ttl => 38400,
>   …
>   records => [
>     ''www        IN    A    xxx.xxx.xxx.xxx''
>     …
>         ]
> }
> 
> bind::zone { [''website2.com'']:
>   expire => 604800,
>   minimum => 3600,
>   ttl => 38400,
>   …
>   records => [
>     ''www        IN    A    xxx.xxx.xxx.xxx''
>     …
>         ]
> }
> 
> … and hundred others like this.
> 
> Finally, the file containing node definition became very big and difficult
to maintain.
> I have the same issue for a webserver with apache::vhost resources for
example.
> 
> For the moment I use a workaround like this:
> 
> import vhosts/*.pp
> 
> And in each files I put a condition to make a restriction for a particular
nodes like this:
> 
> if $::fqdn =~ /^web(\d*)\.example\.net$/ {
>     apache::vhost { ''website2'':
>     …
>     }
> }
> 
> I suppose that many puppet users use puppet to manage nodes in mutualized
hosting context and I would ask if someone have a better solution for this
issue?
> 
> Regards,
> 
> Thomas
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 8/7/2013 6:55 AM, mezcalito38@gmail.com wrote:
> Hi,
>
> I use puppet to manage servers in a mutualized hosting context and I get
> some trouble concerning the node definitions.
>
> For example on dns nodes I have to declare many `bind::zone` resources
> like this:
>
> bind::zone { [''website1.com'']:
>    expire => 604800,
>    minimum => 3600,
>    ttl => 38400,
>    …
>    records => [
>      ''www        IN    A    xxx.xxx.xxx.xxx''
>      …
>          ]
> }
>
> bind::zone { [''website2.com'']:
>    expire => 604800,
>    minimum => 3600,
>    ttl => 38400,
>    …
>    records => [
>      ''www        IN    A    xxx.xxx.xxx.xxx''
>      …
>          ]
> }
>
> … and hundred others like this.
>
> Finally, the file containing node definition became very big and
> difficult to maintain.
> I have the same issue for a webserver with apache::vhost resources for
> example.
>
> For the moment I use a workaround like this:
>
> import vhosts/*.pp
>
> And in each files I put a condition to make a restriction for a
> particular nodes like this:
>
> if $::fqdn =~ /^web(\d*)\.example\.net$/ {
>      apache::vhost { ''website2'':
>      …
>      }
> }
>
> I suppose that many puppet users use puppet to manage nodes in
> mutualized hosting context and I would ask if someone have a better
> solution for this issue?

You might consider using create_resources with hiera. Here''s a simple 
example.

yaml data
---
apache::a2mods:
   expires: {}
   gnutls:  {}
   headers: {}
   rewrite: {}
apache::vhosts:
     statsstage.example.com:
         priority:   ''99''
         a_template: ''apache/vhosts/stats.example.com.erb''
     stage.example.com:
         priority:   ''00''

And the class the queries Hiera and pumps the data through various defines.

class profile::apache {

   include ::apache
   include logrotate::apache

   $mymods = hiera(''apache::a2mods'', {})
   create_resources(''apache::a2mod'', $mymods)

   $myvhosts = hiera(''apache::vhosts'', {})
   create_resources(''apache::vhost'', $myvhosts)

}

Ramin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Le mercredi 7 août 2013 16:13:26 UTC+2, Mason a écrit :
>
> Assuming all the dns servers are getting the same config, We break big 
> chunks like that out into their own modules. So:
>
> node dns1.myco.com {
>   include myco::dns
> }
>
> node web1.myco.com {
>   include myco::web
> }
>
> If the servers have different configs, you can make myco::dns::common, 
> myco::dns::this myco::dns::that myco::dns::other.
>
> We also use a "role" fact instead of a regex on the fqdn for case
> statements, but even those are fragile. I prefer to explicitly include 
> things in the node definitions. It''s a bit verbose, but its also
clear for
> anyone else that has to maintain the code.
>
> — Mason Turner
>
 
Thanks for your answer.

In your solution you put resources declarations in an external class and 
the node definition stay small but your class which contain all resources 
declarations should be very big.
Finally you just moved the issue in an other place or maybe I miss 
something?

Regards,

Thomas

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

> You might consider using create_resources with hiera. Here''s a
simple
> example. 
>
> yaml data 
> --- 
> apache::a2mods: 
>    expires: {} 
>    gnutls:  {} 
>    headers: {} 
>    rewrite: {} 
> apache::vhosts: 
>      statsstage.example.com: 
>          priority:   ''99'' 
>          a_template:
''apache/vhosts/stats.example.com.erb''
>      stage.example.com: 
>          priority:   ''00'' 
>
> And the class the queries Hiera and pumps the data through various 
> defines. 
>
> class profile::apache { 
>
>    include ::apache 
>    include logrotate::apache 
>
>    $mymods = hiera(''apache::a2mods'', {}) 
>    create_resources(''apache::a2mod'', $mymods) 
>
>    $myvhosts = hiera(''apache::vhosts'', {}) 
>    create_resources(''apache::vhost'', $myvhosts) 
>
> } 
>
> Ramin 
>
Thanks for your answer.

In your example using hiera, you put all vhosts config data in one yaml 
file. This solution seems to be maintainable in this case but if I use this 
solution for bind::zone datas the yaml file still be very big and difficult 
to maintain.
Is it possible with hiera to split this config data in multiple files which 
contain each bind::zone resource for example?

Regards,

Thomas

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, August 8, 2013 4:48:02 AM UTC-5, mezca...@gmail.com
wrote:
>
> In your example using hiera, you put all vhosts config data in one yaml 
> file. This solution seems to be maintainable in this case but if I use this
> solution for bind::zone datas the yaml file still be very big and difficult
> to maintain.
> Is it possible with hiera to split this config data in multiple files 
> which contain each bind::zone resource for example?
>
>
Hiera does not come with built-in support for what I think you want.  It 
does, however, support pluggable data source back ends, and it ought to be 
fairly straightforward to build a custom one that handles whatever data 
format you prefer.

Still, I think you''re being a bit inconsistent here.  I don''t
see a big
difference between splitting your zone data among multiple classes or 
modules as Mason suggested on one hand, and splitting them among multiple 
Hiera data files on the other.

Your core problem is that you have big data, and nothing we can suggest 
will change that.  So, what criteria are important to you in comparing 
potential solutions?  Without a better idea of what you''re after,
we''re
just shooting in the dark.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

>
> Your core problem is that you have big data, and nothing we can suggest 
> will change that.  So, what criteria are important to you in comparing 
> potential solutions?  Without a better idea of what you''re after,
we''re
> just shooting in the dark.
>
>
I have big data and I understand I can''t make it disappear but I just
want
to know if there is any way to not put all these datas in the same file.
Maybe I misunderstood but the solution of Ramin or Mason seem to just 
"move" all these datas in a other place/file.

In short, I have to declare few hundred of resources on a node and each 
resources potentialy have specific param/datas.
And my criteria is that I don''t want to put all these declarations in a
single file because it is too difficult to maintain.

Thomas

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, August 8, 2013 11:40:45 AM UTC-5, mezca...@gmail.com
wrote:
>
>
>
>> Your core problem is that you have big data, and nothing we can suggest
>> will change that.  So, what criteria are important to you in comparing 
>> potential solutions?  Without a better idea of what you''re
after, we''re
>> just shooting in the dark.
>>
>>
> I have big data and I understand I can''t make it disappear but I
just
> want to know if there is any way to not put all these datas in the same 
> file.
> Maybe I misunderstood but the solution of Ramin or Mason seem to just 
> "move" all these datas in a other place/file.
>

Yes, you have misunderstood especially Mason''s response.
 
>
> In short, I have to declare few hundred of resources on a node and each 
> resources potentialy have specific param/datas.
> And my criteria is that I don''t want to put all these declarations
in a
> single file because it is too difficult to maintain.
>
>
You can divide your resource declarations among multiple classes and have 
each node declare (only) the classes it needs.  This is part of what Mason 
was trying to tell you.  Each class should be recorded in a separate file 
anyway, so that achieves splitting up your data among multiple files.  Each 
class may contain as few or as many resource declarations as you like.  
Also, if there is any overlap between the resources needed by different 
nodes then you can reduce the overall volume of data by merging the common 
bits into classes used by multiple nodes:

modules/zones/manifests/website1_com.pp:
----
class zones::website1_com {
  bind::zone { ''website1.com'':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,
    …
    records => [
      ''www        IN    A    xxx.xxx.xxx.xxx''
      …
        ]
  }
}

modules/zones/manifests/website2_com.pp:
----
class zones::website2_com {
  bind::zone { ''website2.com'':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,
    …
    records => [
      ''www        IN    A    yyy.yyy.yyy.yyy''
      …
          ]
  }
}

manifests/site.pp (or wherever you store your node definitions):
----
node /.*\.website1\.com$/ {
  include zones::website1_com
}

node /.*\.website2\.com$/ {
  include zones::website2_com
}

You can do the same thing with vhosts, of course.

Alternatively, if individual nodes are not unreasonably large by 
themselves, then it might suit you to just split your node declarations 
into separate files.  That''s actually pretty common:

manifests/site.pp:
----
import ''nodes/*.pp''

manifests/nodes/website1_com.pp
----
node /.*\.website1\.com$/ {
  bind::zone { ''website1.com'':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,
    …
    records => [
      ''www        IN    A    xxx.xxx.xxx.xxx''
      …
        ]
  }

  apache::vhost { ... }

  ...
}

manifests/nodes/website2_com.pp
----
node /.*\.website2\.com$/ {
  bind::zone { ''website2.com'':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,
    …
    records => [
      ''www        IN    A    yyy.yyy.yyy.yyy''
      …
        ]
  }

  apache::vhost { ... }

  ...
}

That''s similar to what you''re doing now, but it''s
safer.  Of course, you
can combine those approaches, too.

Do note that Puppet''s ''include'' function is not
analogous to
similarly-named directives in some programming languages.  It does not 
interpolate files; rather, it declares that the named class should be 
included in the target node''s catalog.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.