Puppet users - Sep 2012 - Using Exported Resources in nginx loadbalancer conf

Hi all,

I''m attempting to use exported resources to dynamically generate an
nginx
config file. What I''d like to do if have all my web servers export
their
hostname inside a file resource, and have these collected by my 
loadbalancer and used in the config file, restarting nginx if anything has 
changed.

There is an example of something very similar in Pro Puppet on pages 
145-146 using Apache. In the example the individual hostnames are gathered 
by apache inside the balancer block using an include statement, however it 
doesn''t seem possible to do this in an upstream block in nginx.

So I think I need to somehow gather the hostnames into an array which I 
then use inside my conf file erb template to create the upstream server 
entries. My problem is how to do this. If I create a custom fact this will 
get evaluated before (I think) the resources are exported and gathered. If 
I use a custom function or some ruby inside the erb template that''s 
evaluated on the master, not the agent. I do have a puppet agent running on 
the master server so I could collect the exported resources there and use a 
function, that''s the best solution I can think of at the moment, just 
wondering if anyone else out there has had any success doing something like 
this?

Thanks,

Jon
 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/aSxa1uTDCAcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 25 September 2012 12:03, Jon Ward <jghward@gmail.com>
wrote:
> Hi all,
>
> I''m attempting to use exported resources to dynamically generate
an nginx
> config file. What I''d like to do if have all my web servers export
their
> hostname inside a file resource, and have these collected by my
loadbalancer
> and used in the config file, restarting nginx if anything has changed.
Nginx can do include directives, right? Can you do something like:

<upstream>
include conf.d/upstream/*.conf
</upstream>

and then have each host export a file in
conf.d/upstream/$hostname.conf containing $hostname:$port (or whatever
is needed).

Not sure if that will work, my nginx is a little rusty...

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Greg,

Yeah that''s what I''d hoped - my balancer.conf.erb template had
this:

upstream <%= balancer_name %>_balancer {
  include /etc/nginx/conf.d.upstream_servers/*.conf;
}

However when I try to start nginx I get:

Starting nginx: [emerg]: "include" directive is not allowed here

A bit of googling (e.g. 
http://stackoverflow.com/questions/4165116/can-include-directive-be-used-within-upstream-block-of-nginx)
seemed to suggest it''s not possible.

On Tuesday, September 25, 2012 12:47:37 PM UTC+1, Greg Sutcliffe
wrote:
>
> On 25 September 2012 12:03, Jon Ward <jgh...@gmail.com
<javascript:>>
> wrote: 
> > Hi all, 
> > 
> > I''m attempting to use exported resources to dynamically
generate an
> nginx 
> > config file. What I''d like to do if have all my web servers
export their
> > hostname inside a file resource, and have these collected by my 
> loadbalancer 
> > and used in the config file, restarting nginx if anything has changed.
>
> Nginx can do include directives, right? Can you do something like: 
>
> <upstream> 
> include conf.d/upstream/*.conf 
> </upstream> 
>
> and then have each host export a file in 
> conf.d/upstream/$hostname.conf containing $hostname:$port (or whatever 
> is needed). 
>
> Not sure if that will work, my nginx is a little rusty... 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/j8WLZrkYOSgJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Jon,

I have seen something similar done with Puppet Labs'' haproxy module:

https://github.com/puppetlabs/puppetlabs-haproxy

Essentially the balancer members export information about themselves
(hostname, port, listening service, etc) which is collected by a
balancer server. The configuration is then pieced together using the
concat module.

I believe that is what you are trying to accomplish here, and hopefully
you find it useful to see how it is done in another module.

Regards,
Bill

On 12-09-25 07:03 AM, Jon Ward wrote:
> Hi all,
> 
> I''m attempting to use exported resources to dynamically generate
an nginx
> config file. What I''d like to do if have all my web servers export
their
> hostname inside a file resource, and have these collected by my 
> loadbalancer and used in the config file, restarting nginx if anything has 
> changed.
> 
> There is an example of something very similar in Pro Puppet on pages 
> 145-146 using Apache. In the example the individual hostnames are gathered 
> by apache inside the balancer block using an include statement, however it 
> doesn''t seem possible to do this in an upstream block in nginx.
> 
> So I think I need to somehow gather the hostnames into an array which I 
> then use inside my conf file erb template to create the upstream server 
> entries. My problem is how to do this. If I create a custom fact this will 
> get evaluated before (I think) the resources are exported and gathered. If 
> I use a custom function or some ruby inside the erb template
that''s
> evaluated on the master, not the agent. I do have a puppet agent running on
> the master server so I could collect the exported resources there and use a
> function, that''s the best solution I can think of at the moment,
just
> wondering if anyone else out there has had any success doing something like
> this?
> 
> Thanks,
> 
> Jon
>  
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks Bill - looks like concat provides a potential solution, will give 
that a go.

Jon

On Tuesday, September 25, 2012 1:50:39 PM UTC+1, Bill Fraser
wrote:
>
> Hi Jon, 
>
> I have seen something similar done with Puppet Labs'' haproxy
module:
>
> https://github.com/puppetlabs/puppetlabs-haproxy 
>
> Essentially the balancer members export information about themselves 
> (hostname, port, listening service, etc) which is collected by a 
> balancer server. The configuration is then pieced together using the 
> concat module. 
>
> I believe that is what you are trying to accomplish here, and hopefully 
> you find it useful to see how it is done in another module. 
>
> Regards, 
> Bill 
>
> On 12-09-25 07:03 AM, Jon Ward wrote: 
> > Hi all, 
> > 
> > I''m attempting to use exported resources to dynamically
generate an
> nginx 
> > config file. What I''d like to do if have all my web servers
export their
> > hostname inside a file resource, and have these collected by my 
> > loadbalancer and used in the config file, restarting nginx if anything
> has 
> > changed. 
> > 
> > There is an example of something very similar in Pro Puppet on pages 
> > 145-146 using Apache. In the example the individual hostnames are 
> gathered 
> > by apache inside the balancer block using an include statement,
however
> it 
> > doesn''t seem possible to do this in an upstream block in
nginx.
> > 
> > So I think I need to somehow gather the hostnames into an array which
I
> > then use inside my conf file erb template to create the upstream
server
> > entries. My problem is how to do this. If I create a custom fact this 
> will 
> > get evaluated before (I think) the resources are exported and
gathered.
> If 
> > I use a custom function or some ruby inside the erb template
that''s
> > evaluated on the master, not the agent. I do have a puppet agent
running
> on 
> > the master server so I could collect the exported resources there and 
> use a 
> > function, that''s the best solution I can think of at the
moment, just
> > wondering if anyone else out there has had any success doing something
> like 
> > this? 
> > 
> > Thanks, 
> > 
> > Jon 
> >   
> > 
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/CI4CJG3-hBoJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.