Puppet users - Feb 2010 - Plugins in modules with environments and the puppetmaster libdir

Unless I''m missing something, we kind of have a fatal flaw with using
plugins in modules with environments when it comes to making those
same plugins available to the puppetmaster itself.

If you''re deploying a custom type/provider with the plugins in modules
model, you still need to make these plugins discoverable in the
puppetmaster libdir.

ie, if I''m deploying:
<modulepath>/base/lib/puppet/type/foo.rb
<modulepath>/base/lib/puppet/provider/foo.rb

Then I need to ensure that these plugins are available at:
$libdir/puppet/{type,provider}/foo.rb
on the puppetmaster itself.

The problem with this is that it makes it difficult to release
different versions of the same plugin to different clients, as you
have to pick one version and one version only of the plugin to be in
the puppetmaster libdir, and this version may conflict with the
version you''re supplying to clients in terms of valid syntax and
acceptable parameters.

How are other people dealing with this?


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Feb 4, 2010 at 11:49 AM, Nigel Kersten <nigelk@google.com>
wrote:
> Unless I''m missing something, we kind of have a fatal flaw with
using
> plugins in modules with environments when it comes to making those
> same plugins available to the puppetmaster itself.
>
> If you''re deploying a custom type/provider with the plugins in
modules
> model, you still need to make these plugins discoverable in the
> puppetmaster libdir.
>
> ie, if I''m deploying:
> <modulepath>/base/lib/puppet/type/foo.rb
> <modulepath>/base/lib/puppet/provider/foo.rb
>
> Then I need to ensure that these plugins are available at:
> $libdir/puppet/{type,provider}/foo.rb
> on the puppetmaster itself.
>
> The problem with this is that it makes it difficult to release
> different versions of the same plugin to different clients, as you
> have to pick one version and one version only of the plugin to be in
> the puppetmaster libdir, and this version may conflict with the
> version you''re supplying to clients in terms of valid syntax and
> acceptable parameters.
>
> How are other people dealing with this?
>
We created a module called plugins and put our code there. For example, we have:

modules/plugins/files/facter/hugepagesize.rb

That gets replicated across environments (and we can have different
hugepagesize.rb files per environment if we need to).

Matt

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Feb 4, 2010 at 9:08 AM, Matthew Hyclak <hyclak@gmail.com>
wrote:
> On Thu, Feb 4, 2010 at 11:49 AM, Nigel Kersten <nigelk@google.com>
wrote:
>> Unless I''m missing something, we kind of have a fatal flaw
with using
>> plugins in modules with environments when it comes to making those
>> same plugins available to the puppetmaster itself.
>>
>> If you''re deploying a custom type/provider with the plugins in
modules
>> model, you still need to make these plugins discoverable in the
>> puppetmaster libdir.
>>
>> ie, if I''m deploying:
>> <modulepath>/base/lib/puppet/type/foo.rb
>> <modulepath>/base/lib/puppet/provider/foo.rb
>>
>> Then I need to ensure that these plugins are available at:
>> $libdir/puppet/{type,provider}/foo.rb
>> on the puppetmaster itself.
>>
>> The problem with this is that it makes it difficult to release
>> different versions of the same plugin to different clients, as you
>> have to pick one version and one version only of the plugin to be in
>> the puppetmaster libdir, and this version may conflict with the
>> version you''re supplying to clients in terms of valid syntax
and
>> acceptable parameters.
>>
>> How are other people dealing with this?
>>
>
> We created a module called plugins and put our code there. For example, we
have:
>
> modules/plugins/files/facter/hugepagesize.rb
>
> That gets replicated across environments (and we can have different
> hugepagesize.rb files per environment if we need to).

So facter plugins are kind of different, as they''re not actually
required to be in the puppetmaster libdir.

Say this was a type/provider, and you wanted to add a new parameter,
but only roll it out to your testing environments.

What do you do then?

If the version in the puppetmaster libdir doesn''t accept that
parameter, the manifest compilation will fail for clients who *are*
getting a version of the plugin that supports that parameter.


>
> Matt
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel Kersten wrote:
> So facter plugins are kind of different, as they''re not actually
> required to be in the puppetmaster libdir.
> 
> Say this was a type/provider, and you wanted to add a new parameter,
> but only roll it out to your testing environments.
Functions also have this limitation, by the way.
> What do you do then?
> 
> If the version in the puppetmaster libdir doesn''t accept that
> parameter, the manifest compilation will fail for clients who *are*
> getting a version of the plugin that supports that parameter.
You lose...

Well, there are a couple of things you can do to work around this
limitation.  For one thing, you could run a second puppetmaster
process, perhaps on another port, or listening on another IP address,
or perhaps even on another machine.

Another thing you can do, is to run puppet with local manifests,
instead of puppetd connecting to puppetmasterd, when developing
the new version of the plugin.  That''s what I do.  It does get a
bit iffy if your manifests want to fetch files from the puppetmaster
(i.e. that aren''t in the "modules" namespace) though, or
otherwise
need to access files that are only available on the puppetmaster.


Good news, though, is that the upcoming "Rowlf" version of Puppet is
supposed to solve this for at least resource types.  At least Luke has
posted patches to the devel list that I gather will do that.  But it
will probably still be a problem for custom functions (I have somewhat
volunteered to take a look at it, but I''m unlikely to find the time
before Rowlf is supposed to be out).


	/Bellman

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Feb 5, 2010 at 3:29 AM, Thomas Bellman <bellman@nsc.liu.se>
wrote:
> Nigel Kersten wrote:
>
>> So facter plugins are kind of different, as they''re not
actually
>> required to be in the puppetmaster libdir.
>>
>> Say this was a type/provider, and you wanted to add a new parameter,
>> but only roll it out to your testing environments.
>
> Functions also have this limitation, by the way.
>
>> What do you do then?
>>
>> If the version in the puppetmaster libdir doesn''t accept that
>> parameter, the manifest compilation will fail for clients who *are*
>> getting a version of the plugin that supports that parameter.
>
> You lose...
>
> Well, there are a couple of things you can do to work around this
> limitation.  For one thing, you could run a second puppetmaster
> process, perhaps on another port, or listening on another IP address,
> or perhaps even on another machine.
>
> Another thing you can do, is to run puppet with local manifests,
> instead of puppetd connecting to puppetmasterd, when developing
> the new version of the plugin.  That''s what I do.  It does get a
> bit iffy if your manifests want to fetch files from the puppetmaster
> (i.e. that aren''t in the "modules" namespace) though, or
otherwise
> need to access files that are only available on the puppetmaster.
>
>
> Good news, though, is that the upcoming "Rowlf" version of Puppet
is
> supposed to solve this for at least resource types.  At least Luke has
> posted patches to the devel list that I gather will do that.  But it
> will probably still be a problem for custom functions (I have somewhat
> volunteered to take a look at it, but I''m unlikely to find the
time
> before Rowlf is supposed to be out).
+ puppet-dev

Luke, how is this going to be solved in practice? The puppetmasterd
will automatically add modules/*/lib dirs to it''s own libdir in the
context/environment of the current client?


I''m thinking that in the meantime I may need to encode plugin versions
in their names, so when a client''s manifest contains a given plugin,
it''s always going to refer to that version, and I script something
that collates all the lib/... directories into the puppetmasterd
libdir.

I feel dirty.

>
>
>        /Bellman
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 5, 2010, at 8:47 AM, Nigel Kersten wrote:
> On Fri, Feb 5, 2010 at 3:29 AM, Thomas Bellman <bellman@nsc.liu.se>  
> wrote:
>> Nigel Kersten wrote:
>>
>>> So facter plugins are kind of different, as they''re not
actually
>>> required to be in the puppetmaster libdir.
>>>
>>> Say this was a type/provider, and you wanted to add a new
parameter,
>>> but only roll it out to your testing environments.
>>
>> Functions also have this limitation, by the way.
>>
>>> What do you do then?
>>>
>>> If the version in the puppetmaster libdir doesn''t accept
that
>>> parameter, the manifest compilation will fail for clients who *are*
>>> getting a version of the plugin that supports that parameter.
>>
>> You lose...
>>
>> Well, there are a couple of things you can do to work around this
>> limitation.  For one thing, you could run a second puppetmaster
>> process, perhaps on another port, or listening on another IP address,
>> or perhaps even on another machine.
>>
>> Another thing you can do, is to run puppet with local manifests,
>> instead of puppetd connecting to puppetmasterd, when developing
>> the new version of the plugin.  That''s what I do.  It does get
a
>> bit iffy if your manifests want to fetch files from the puppetmaster
>> (i.e. that aren''t in the "modules" namespace)
though, or otherwise
>> need to access files that are only available on the puppetmaster.
>>
>>
>> Good news, though, is that the upcoming "Rowlf" version of
Puppet is
>> supposed to solve this for at least resource types.  At least Luke  
>> has
>> posted patches to the devel list that I gather will do that.  But it
>> will probably still be a problem for custom functions (I have  
>> somewhat
>> volunteered to take a look at it, but I''m unlikely to find the
time
>> before Rowlf is supposed to be out).
>
> + puppet-dev
>
> Luke, how is this going to be solved in practice? The puppetmasterd
> will automatically add modules/*/lib dirs to it''s own libdir in
the
> context/environment of the current client?
Hmm, not quite true yet.

At this point, rowlf has a bunch of refactoring around just the  
language stuff, not the pure-ruby stuff.

We''re on the path to converging the language resource types and the  
pure ruby Puppet::Type classes, but we''re still a ways away.  Part of  
that convergence will be fixing this problem, but I don''t expect to  
see a real long-term fix until at least the release after rowlf -  
mostly just because we don''t want to delay rowlf too much further.
> I''m thinking that in the meantime I may need to encode plugin
versions
> in their names, so when a client''s manifest contains a given
plugin,
> it''s always going to refer to that version, and I script something
> that collates all the lib/... directories into the puppetmasterd
> libdir.
>
> I feel dirty.
Yeah, that''s ugly.  I think it''d be way easier to run
environment-
specific masters on a different port or server, wouldn''t it?

BTW, if this is a big priority to someone, I think it''s approachable  
today - far more than six months ago, with recent refactoring - and  
I''m happy to work with someone who''s committed to getting it
fixed.

And I know this has been coming up a lot recently, so it''s getting  
pushed up on my priority list.

-- 
Men never do evil so completely and cheerfully as when they do it from a
religious conviction. --Blaise Pascal
---------------------------------------------------------------------
Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Feb 8, 2010 at 2:45 PM, Luke Kanies <luke@reductivelabs.com>
wrote:
> On Feb 5, 2010, at 8:47 AM, Nigel Kersten wrote:
>
>  On Fri, Feb 5, 2010 at 3:29 AM, Thomas Bellman <bellman@nsc.liu.se>
>> wrote:
>>
>>> Nigel Kersten wrote:
>>>
>>>  So facter plugins are kind of different, as they''re not
actually
>>>> required to be in the puppetmaster libdir.
>>>>
>>>> Say this was a type/provider, and you wanted to add a new
parameter,
>>>> but only roll it out to your testing environments.
>>>>
>>>
>>> Functions also have this limitation, by the way.
>>>
>>>  What do you do then?
>>>>
>>>> If the version in the puppetmaster libdir doesn''t
accept that
>>>> parameter, the manifest compilation will fail for clients who
*are*
>>>> getting a version of the plugin that supports that parameter.
>>>>
>>>
>>> You lose...
>>>
>>> Well, there are a couple of things you can do to work around this
>>> limitation.  For one thing, you could run a second puppetmaster
>>> process, perhaps on another port, or listening on another IP
address,
>>> or perhaps even on another machine.
>>>
>>> Another thing you can do, is to run puppet with local manifests,
>>> instead of puppetd connecting to puppetmasterd, when developing
>>> the new version of the plugin.  That''s what I do.  It does
get a
>>> bit iffy if your manifests want to fetch files from the
puppetmaster
>>> (i.e. that aren''t in the "modules" namespace)
though, or otherwise
>>> need to access files that are only available on the puppetmaster.
>>>
>>>
>>> Good news, though, is that the upcoming "Rowlf" version
of Puppet is
>>> supposed to solve this for at least resource types.  At least Luke
has
>>> posted patches to the devel list that I gather will do that.  But
it
>>> will probably still be a problem for custom functions (I have
somewhat
>>> volunteered to take a look at it, but I''m unlikely to find
the time
>>> before Rowlf is supposed to be out).
>>>
>>
>> + puppet-dev
>>
>> Luke, how is this going to be solved in practice? The puppetmasterd
>> will automatically add modules/*/lib dirs to it''s own libdir
in the
>> context/environment of the current client?
>>
>
> Hmm, not quite true yet.
>
> At this point, rowlf has a bunch of refactoring around just the language
> stuff, not the pure-ruby stuff.
>
> We''re on the path to converging the language resource types and
the pure
> ruby Puppet::Type classes, but we''re still a ways away.  Part of
that
> convergence will be fixing this problem, but I don''t expect to see
a real
> long-term fix until at least the release after rowlf - mostly just because
> we don''t want to delay rowlf too much further.

Can you give us an overview of the way this is intended to work?

>
>
>  I''m thinking that in the meantime I may need to encode plugin
versions
>> in their names, so when a client''s manifest contains a given
plugin,
>> it''s always going to refer to that version, and I script
something
>> that collates all the lib/... directories into the puppetmasterd
>> libdir.
>>
>> I feel dirty.
>>
>
> Yeah, that''s ugly.  I think it''d be way easier to run
environment-specific
> masters on a different port or server, wouldn''t it?
>
Not for the number of environments we have...

>
> BTW, if this is a big priority to someone, I think it''s
approachable today
> - far more than six months ago, with recent refactoring - and I''m
happy to
> work with someone who''s committed to getting it fixed.
>
> And I know this has been coming up a lot recently, so it''s getting
pushed
> up on my priority list.
>
What would you need from those of us who are interested in working on this
with you Luke?

>
> --
> Men never do evil so completely and cheerfully as when they do it from a
> religious conviction. --Blaise Pascal
> ---------------------------------------------------------------------
> Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppet-dev@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-dev+unsubscribe@googlegroups.com<puppet-dev%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.
>
>

-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 9, 2010, at 6:55 AM, Nigel Kersten wrote:
>
>
> On Mon, Feb 8, 2010 at 2:45 PM, Luke Kanies <luke@reductivelabs.com>
> wrote:
> On Feb 5, 2010, at 8:47 AM, Nigel Kersten wrote:
>
> On Fri, Feb 5, 2010 at 3:29 AM, Thomas Bellman <bellman@nsc.liu.se>  
> wrote:
> Nigel Kersten wrote:
>
> So facter plugins are kind of different, as they''re not actually
> required to be in the puppetmaster libdir.
>
> Say this was a type/provider, and you wanted to add a new parameter,
> but only roll it out to your testing environments.
>
> Functions also have this limitation, by the way.
>
> What do you do then?
>
> If the version in the puppetmaster libdir doesn''t accept that
> parameter, the manifest compilation will fail for clients who *are*
> getting a version of the plugin that supports that parameter.
>
> You lose...
>
> Well, there are a couple of things you can do to work around this
> limitation.  For one thing, you could run a second puppetmaster
> process, perhaps on another port, or listening on another IP address,
> or perhaps even on another machine.
>
> Another thing you can do, is to run puppet with local manifests,
> instead of puppetd connecting to puppetmasterd, when developing
> the new version of the plugin.  That''s what I do.  It does get a
> bit iffy if your manifests want to fetch files from the puppetmaster
> (i.e. that aren''t in the "modules" namespace) though, or
otherwise
> need to access files that are only available on the puppetmaster.
>
>
> Good news, though, is that the upcoming "Rowlf" version of Puppet
is
> supposed to solve this for at least resource types.  At least Luke has
> posted patches to the devel list that I gather will do that.  But it
> will probably still be a problem for custom functions (I have somewhat
> volunteered to take a look at it, but I''m unlikely to find the
time
> before Rowlf is supposed to be out).
>
> + puppet-dev
>
> Luke, how is this going to be solved in practice? The puppetmasterd
> will automatically add modules/*/lib dirs to it''s own libdir in
the
> context/environment of the current client?
>
> Hmm, not quite true yet.
>
> At this point, rowlf has a bunch of refactoring around just the  
> language stuff, not the pure-ruby stuff.
>
> We''re on the path to converging the language resource types and
the
> pure ruby Puppet::Type classes, but we''re still a ways away.  Part
> of that convergence will be fixing this problem, but I don''t
expect
> to see a real long-term fix until at least the release after rowlf -  
> mostly just because we don''t want to delay rowlf too much further.
>
> Can you give us an overview of the way this is intended to work?
Not sure exactly how to answer this question, since there''s a lot
there.

The convergence of the two classes (Puppet::Type and  
Puppet::Resource::Type) is complex to describe.  Basically, current  
resource types are subclasses of Puppet::Type, and resources are  
instances of those subclasses.  We''ll be changing things so that  
resource types are instances of Puppet::Resource::Type, and resources  
are instances of Puppet::Resource.  In terms of observable  
differences, other than the constants involved there won''t really be  
any - it''ll just make contribution, maintenance, and custom  
development much easier.

However, it''ll also enable us to do a lot of things we currently  
can''t, like maintain non-global (i.e., environment-specific) resource  
types.  The refactoring I did to enable the pure-ruby DSL was the main  
step toward this:  When you use ''define'' in the language,
you''re
creating an environment-specific resource type, it''s just not as  
functional as a builtin resource type.  I think the next major release  
will normalize the functionality between the two, and hopefully also  
make it possible to use existing providers with new-style resource  
types.  Then it''s just a question of porting over the existing types  
to use the new class structure.
> I''m thinking that in the meantime I may need to encode plugin
versions
> in their names, so when a client''s manifest contains a given
plugin,
> it''s always going to refer to that version, and I script something
> that collates all the lib/... directories into the puppetmasterd
> libdir.
>
> I feel dirty.
>
> Yeah, that''s ugly.  I think it''d be way easier to run
environment-
> specific masters on a different port or server, wouldn''t it?
>
> Not for the number of environments we have...
Ah.
>
>
> BTW, if this is a big priority to someone, I think it''s
approachable
> today - far more than six months ago, with recent refactoring - and  
> I''m happy to work with someone who''s committed to getting
it fixed.
>
> And I know this has been coming up a lot recently, so it''s getting
> pushed up on my priority list.
>
> What would you need from those of us who are interested in working  
> on this with you Luke?

Code? :)

-- 
Finn''s Law:
     Uncertainty is the final test of innovation.
---------------------------------------------------------------------
Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.