hello, i wondered if it would not be interesting to have some filters function in puppet like filter_integer(), filter_prelregex, filter_text, filter_email or such so we can validate and make our defines a little more foolproof :) i particulary see filter_regex as usefull in quite a lot of exec that manipulates text files. What do you think about this ? -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Mon, Apr 30, 2007 at 06:17:45PM +0200, ADNET Ghislain wrote:> i wondered if it would not be interesting to have some filters > function in puppet like > > filter_integer(), filter_prelregex, filter_text, filter_email or such > so we can validate and make our defines a little more foolproof :) > i particulary see filter_regex as usefull in quite a lot of exec that > manipulates text files. What do you think about this ?Can you give concrete examples of usage? - Matt -- Judging by this particular thread, many people in this group spent their school years taking illogical, pointless orders from morons and having their will to live systematically crushed. And people say school doesn''t prepare kids for the real world. -- Rayner, in the Monastery
On Apr 30, 2007, at 3:14 PM, Matthew Palmer wrote:> On Mon, Apr 30, 2007 at 06:17:45PM +0200, ADNET Ghislain wrote: >> i wondered if it would not be interesting to have some filters >> function in puppet like >> >> filter_integer(), filter_prelregex, filter_text, filter_email or >> such >> so we can validate and make our defines a little more foolproof :) >> i particulary see filter_regex as usefull in quite a lot of exec that >> manipulates text files. What do you think about this ? > > Can you give concrete examples of usage?I also would like to see examples, but I''ll follow up and say I don''t generally add parser functions that a customer isn''t paying me for or that I personally don''t see a need for. These parser functions are a great place to start learning ruby, since they''re easy to write and provide benefit immediately, just like facts. -- The chief lesson I have learned in a long life is that the only way to make a man trustworthy is to trust him; and the surest way to make him untrustworthy is to distrust him and show your distrust. -- Henry L. Stimson --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Mon, Apr 30, 2007 at 04:52:00PM -0500, Luke Kanies wrote:> On Apr 30, 2007, at 3:14 PM, Matthew Palmer wrote: > > > On Mon, Apr 30, 2007 at 06:17:45PM +0200, ADNET Ghislain wrote: > >> i wondered if it would not be interesting to have some filters > >> function in puppet like > >> > >> filter_integer(), filter_prelregex, filter_text, filter_email or > >> such > >> so we can validate and make our defines a little more foolproof :) > >> i particulary see filter_regex as usefull in quite a lot of exec that > >> manipulates text files. What do you think about this ? > > > > Can you give concrete examples of usage? > > I also would like to see examples, but I''ll follow up and say I don''t > generally add parser functions that a customer isn''t paying me for or > that I personally don''t see a need for. These parser functions are a > great place to start learning ruby, since they''re easy to write and > provide benefit immediately, just like facts.And the process for doing so is documented! - Matt -- MySQL seems to be the Windows of the database world. Broken, underspecced, and mainly only popular due to inertia and people who don''t really know what they''re doing. -- Peter Corlett, in the Monastery
Matthew Palmer a écrit :> On Mon, Apr 30, 2007 at 06:17:45PM +0200, ADNET Ghislain wrote: > >> i wondered if it would not be interesting to have some filters >> function in puppet like >> >> filter_integer(), filter_prelregex, filter_text, filter_email or such >> so we can validate and make our defines a little more foolproof :) >> i particulary see filter_regex as usefull in quite a lot of exec that >> manipulates text files. What do you think about this ? >> > > Can you give concrete examples of usage? > > - Matt > >yes for exemple any ressources like this: exec { "ajoute_ou_remplace-$ligne-$fichier": path => "/sbin:/bin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/local/bin", command => "perl -pi -e ''s/$regex/$ligne/'' $fichier", unless => "grep -i -q -P ''$regex'' $fichier" } will fail if your regex has any ''or / in it in it it will fail, same thing if you use a exec tosend an email it would be handy to be able to filter the email and be sure it is an email and not something odd that will crash your exec because of a weird thing in it. As most of the time you will send the exec variables you will not be able to escape the content manualy (for exemple you put a line like: $name $input $output and want to replace it based on the regex: ^$name.* here you will prefer to make $regexfiltered= filter_perlregex($name) and use ^$regexfiltered.* as a regular expression so it would work in all cases. -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
>> I also would like to see examples, but I''ll follow up and say I don''t >> generally add parser functions that a customer isn''t paying me for or >> that I personally don''t see a need for. These parser functions are a >> great place to start learning ruby, since they''re easy to write and >> provide benefit immediately, just like facts. >> > > And the process for doing so is documented! > > - Matt > >yes i see, i will see how i can add one myself :) Perhaps i will be able to make some little part in ruby after all. Basicaly i throw ideas out that seems helpufull to me, input validation is something that could be usefull in some case i work on. -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
> And the process for doing so is documented! > > - Matt > >and the minute_from_address is very usefull ! :) it does not sems possible to specify the function directory. Do you know if the gem packages and other rpm deb etc.. package handle the upgrade by keeping functions we defined or do they just drop them ? -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Wed, May 02, 2007 at 04:56:49PM +0200, ADNET Ghislain wrote:> it does not sems possible to specify the function directory. > > Do you know if the gem packages and other rpm deb etc.. package handle > the upgrade by keeping functions we defined or do they just drop them ?The deb packages, at least, do not currently put anything into the functions directory, however if we start shipping more functions we might start to get namespace clashes, in which case the packaged files will (unfortunately) overwrite any files you''ve placed in the same location (with the same filename). I expect that the solution to this is going to be the modules concept, which I believe (or at least hope) has a location for extra functions. - Matt -- You have a 16-bit quantity, but 5 bits of it are here and 2 bits of it are there... and 2 bits of it are back here and 3 bits of it are up there. The C code to extract useful data had so many >> and << operators in it that it looked like the C++ version of "hello world". -- Matt Roberds, ASR
> The deb packages, at least, do not currently put anything into the functions > directory, however if we start shipping more functions we might start to get > namespace clashes, in which case the packaged files will (unfortunately) > overwrite any files you''ve placed in the same location (with the same > filename). >but if they find a function directory do they keep it ? i am more woried about the gem packages as they use a different directory for each release. Luke, do the gem update of a puppet package copy the function directory to keep all the functions in the new version ?> I expect that the solution to this is going to be the modules concept, which > I believe (or at least hope) has a location for extra functions. >i have to have a look at this module thing then :) -- Cordialement, Ghislain ADNET. AQUEOS. Attention ! Toute demande de support ou commande de domaine par email sera refusée, pour cela utilisez https://support.aqueos.net. Pour tout contact nos coordonnées : http://www.aqueos.com/aqueos-services-informatiques-societe.php Fax: 01.72.70.32.66 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Thu, May 03, 2007 at 09:32:49AM +0200, ADNET Ghislain wrote:> > >The deb packages, at least, do not currently put anything into the > >functions > >directory, however if we start shipping more functions we might start to > >get > >namespace clashes, in which case the packaged files will (unfortunately) > >overwrite any files you''ve placed in the same location (with the same > >filename). > > > but if they find a function directory do they keep it ?Yes, of course. Debian packages don''t (as a rule) fiddle with files and directories they don''t know about. - Matt
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 03 May 2007, Matthew Palmer wrote:> On Thu, May 03, 2007 at 09:32:49AM +0200, ADNET Ghislain wrote: > > >The deb packages, at least, do not currently put anything into the > > >functions > > >directory, however if we start shipping more functions we might start to > > >get > > >namespace clashes, in which case the packaged files will (unfortunately) > > >overwrite any files you''ve placed in the same location (with the same > > >filename). > > > > but if they find a function directory do they keep it ? > > Yes, of course. Debian packages don''t (as a rule) fiddle with files and > directories they don''t know about.Yeah, that''s why you should put you''re custom .rb stuff to Facter[''rubysitedir''] which is -- in Debian -- under /usr/local. The FHS[1] is very clear about distributions-may-not-touch-anything-in-/usr/local. Regards, David [1] http://www.pathname.com/fhs/ - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGOasa/Pp1N6Uzh0URAikpAJ4iCOLRVPMsGrgBp2A0gvUN1Br9ogCfcaNa Yspijbcn39s/WQshdwSYqYY=k351 -----END PGP SIGNATURE-----
On May 3, 2007, at 2:32 AM, ADNET Ghislain wrote:> but if they find a function directory do they keep it ? i am more > woried about the gem packages as they use a different directory for > each release. Luke, do the gem update of a puppet package copy the > function directory to keep all the functions in the new version ?As mentioned, use the standard ruby library location, not gem locations.>> I expect that the solution to this is going to be the modules >> concept, which >> I believe (or at least hope) has a location for extra functions. >> > i have to have a look at this module thing then :)Modules have no means of storing functions, but they could be added, I suppose. Hmm. Hadn''t thought about that. -- The Chico, California, City Council enacted a ban on nuclear weapons, setting a $500 fine for anyone detonating one within city limits. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, May 03, 2007 at 11:23:30AM -0500, Luke Kanies wrote:> >Matt Palmer: > >> I expect that the solution to this is going to be the modules > >> concept, which > >> I believe (or at least hope) has a location for extra functions. > >> > > i have to have a look at this module thing then :) > > Modules have no means of storing functions, but they could be added, > I suppose. Hmm. Hadn''t thought about that.Jaw.floor() I''m pretty certain that a number of modules will benefit from functions. Hell, if we think of modules as being "anything that extends Puppet", we''re also going to need some way to add new types as well. It all gets very tricky, though, since some of this code gets run on the server, and some on the client. We might end up with the server having to say "here, you might need this" in the manifest glom -- although I suppose the existing client sync concept could presumably be extended to modules. - Matt -- "You keep using that word. I do not think it means what you think it means." -- Inigo, The Princess Bride
On May 3, 2007, at 2:48 PM, Matthew Palmer wrote:> > Jaw.floor() > > I''m pretty certain that a number of modules will benefit from > functions. > Hell, if we think of modules as being "anything that extends > Puppet", we''re > also going to need some way to add new types as well. It all gets > very > tricky, though, since some of this code gets run on the server, and > some on > the client. We might end up with the server having to say "here, > you might > need this" in the manifest glom -- although I suppose the existing > client > sync concept could presumably be extended to modules.Hmm. I''ve been thinking of modules as entirely server-side, but I suppose it''s not entirely unreasonable to think of sticking a ruby lib directory in there for syncing facts, providers, etc. That really starts to blur the line between system package and Puppet module, though. Why is your jaw hitting the floor, btw? Because this is such a great idea, or because I''m such a ninny for not having thought of it? -- I love deadlines. I like the whooshing sound they make as they fly by. --Douglas Adams --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, May 03, 2007 at 03:41:30PM -0500, Luke Kanies wrote:> On May 3, 2007, at 2:48 PM, Matthew Palmer wrote: > > > > Jaw.floor() > > > > I''m pretty certain that a number of modules will benefit from > > functions. > > Hell, if we think of modules as being "anything that extends > > Puppet", we''re > > also going to need some way to add new types as well. It all gets > > very > > tricky, though, since some of this code gets run on the server, and > > some on > > the client. We might end up with the server having to say "here, > > you might > > need this" in the manifest glom -- although I suppose the existing > > client > > sync concept could presumably be extended to modules. > > Hmm. I''ve been thinking of modules as entirely server-side, but I > suppose it''s not entirely unreasonable to think of sticking a ruby > lib directory in there for syncing facts, providers, etc.Weeeellllll... functions *are* entirely server-side, so it seems like a natural extension of shipping .pp files and template config files.> That really starts to blur the line between system package and Puppet > module, though.EXPN system package To my way of thinking, there should be three separate sections to a Puppet installation: Puppet itself (master and client), modules (sourced and managed primarily through PRM, though with some locally-written ones too, no doubt) and finally the node mapping to lay it all out. I think functions *definitely* fit into the module concept, and it''s not such a huge stretch to think about custom types and providers in there as well. For instance, consider the canonical example: a Nagios config module. Here we''ll probably have a bunch of types to represent different Nagios types, as well as template config files and whatever else. The types need to be distributed, and being able to push them out from the Puppetmaster is a whole hell of a lot tidier than having to copy them out of band. At least you get dynamic reloading for free in Ruby, which would have been a lot trickier in most other languages.> Why is your jaw hitting the floor, btw? Because this is such a great > idea, or because I''m such a ninny for not having thought of it?Because I''m rarely the first person to have thought of an idea. It was a shock because I managed to get in first for once. <grin> - Matt
On May 3, 2007, at 4:56 PM, Matthew Palmer wrote:> > Weeeellllll... functions *are* entirely server-side, so it seems > like a > natural extension of shipping .pp files and template config files.That I agree with; I was only questioning throwing client-side .rb files in there. We''ve already discussed also throwing generators (arguments to generate()) in modules.>> That really starts to blur the line between system package and Puppet >> module, though. > > EXPN system package"Native" packages? Whatever the packaging system used by, um, the system is.> To my way of thinking, there should be three separate sections to a > Puppet > installation: Puppet itself (master and client), modules (sourced and > managed primarily through PRM, though with some locally-written > ones too, no > doubt) and finally the node mapping to lay it all out. I think > functions > *definitely* fit into the module concept, and it''s not such a huge > stretch > to think about custom types and providers in there as well. For > instance, > consider the canonical example: a Nagios config module. Here we''ll > probably > have a bunch of types to represent different Nagios types, as well as > template config files and whatever else. The types need to be > distributed, > and being able to push them out from the Puppetmaster is a whole > hell of a > lot tidier than having to copy them out of band. At least you get > dynamic > reloading for free in Ruby, which would have been a lot trickier in > most > other languages.Note that it''s not entirely free -- it''s moderately difficult to track whether a loaded file has been modified. We''d probably need to just automatically ''load'' any files changed, and have all of our loading systems automatically remove any previously loaded classes or whatnot. I''ve done some of this with the factsync and pluginsync support, but we clearly can''t scale this with a new download mechanism for every file type. I''m glad to have it, but I wouldn''t describe it as "free". Note that is already filed as an enhancement request: https://reductivelabs.com/trac/puppet/ticket/534> Because I''m rarely the first person to have thought of an idea. It > was a > shock because I managed to get in first for once. <grin>Not likely, Mr. "Let''s use the Rails support as an asset manager". -- The chief lesson I have learned in a long life is that the only way to make a man trustworthy is to trust him; and the surest way to make him untrustworthy is to distrust him and show your distrust. -- Henry L. Stimson --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Fri, 2007-05-04 at 05:48 +1000, Matthew Palmer wrote:> I''m pretty certain that a number of modules will benefit from functions. > Hell, if we think of modules as being "anything that extends Puppet", we''re > also going to need some way to add new types as well.There are a few things that make me a little queasy about extending modules that way: 1. you are going to get into much subtler and harder API stability problems if you add functions or general plugins to modules, since they rely on puppet''s internals to a much bigger extent than current modules 2. I would much prefer keeping using puppet (current modules) and extending puppet separate; shouldn''t the goal for custom types and functions be that they find their way into puppet''s core ? 3. it feels like there''s a slippery slope here where puppet will grow its own package management system Having said all that, bundling modules, plugins and functions into one easily installable thing does sound very attractive - maybe it would take some examples of what that would enable to clear things up. David
On Thu, May 03, 2007 at 03:54:24PM -0700, David Lutterkort wrote:> On Fri, 2007-05-04 at 05:48 +1000, Matthew Palmer wrote: > > I''m pretty certain that a number of modules will benefit from functions. > > Hell, if we think of modules as being "anything that extends Puppet", we''re > > also going to need some way to add new types as well. > > There are a few things that make me a little queasy about extending > modules that way: > 1. you are going to get into much subtler and harder API stability > problems if you add functions or general plugins to modules, > since they rely on puppet''s internals to a much bigger extent > than current modulesYep, it''s a massive potential problem, but one that''s inherent in any extensible system.> 2. I would much prefer keeping using puppet (current modules) and > extending puppet separate;As far as I''m concerned, most non-trivial uses of Puppet are, in some way, extending Puppet. Apart from the language used and the degree, writing a generally useful .pp file like, say, replace_line.pp is little-to-no different to writing a Ruby class to provide a ''nagios_service'' type.> shouldn''t the goal for custom types > and functions be that they find their way into puppet''s core ?That''d be nice, but I''m almost certain it''s not going to be practical. Sometimes, modules (as currently defined, ie .pps and templates) will rely on new types; waiting until those new types get into core would be unpleasant. Some types/modules will really only be useful to a small subset of people -- not-so-useful to put into core. Coupling the release of new versions of types to core releases will, I am fairly certain, become untenable. Basically, think of all the reasons why Apache, Eclipse, and Rails don''t bundle every possible plugin with their core releases, and you''ve just got my argument against bundling all possible types in core.> 3. it feels like there''s a slippery slope here where puppet will > grow its own package management systemWhat is PRM if not some form of package management? (Package management by direct manipulation of revision control is still package management).> Having said all that, bundling modules, plugins and functions into one > easily installable thing does sound very attractive - maybe it would > take some examples of what that would enable to clear things up.A big "Nagios Config" module is the canonical example. New types, some templates, some .pp files for classes, and probably even a function or two. Having seen the speed with which people have come up with new classes and defines of their own and published them on the wiki, I suspect that a "build it[1] and they will come" attitude will work wonders. - Matt [1] And Document It -- When I was a kid I used to pray every night for a new bicycle. Then I realised that the Lord doesn''t work that way so I stole one and asked Him to forgive me. -- Emo Philips.
On May 3, 2007, at 8:38 PM, Matt Palmer wrote:> > Yep, it''s a massive potential problem, but one that''s inherent in any > extensible system.Not *my* system. I''ll magically find a way to avoid allowing plugins to destroy Puppet. :)> As far as I''m concerned, most non-trivial uses of Puppet are, in > some way, > extending Puppet. Apart from the language used and the degree, > writing a > generally useful .pp file like, say, replace_line.pp is little-to-no > different to writing a Ruby class to provide a ''nagios_service'' type.That''s what I thought, but we certainly see a lot more Puppet code than Ruby code.>> shouldn''t the goal for custom types >> and functions be that they find their way into puppet''s >> core ? > > That''d be nice, but I''m almost certain it''s not going to be practical.I agree with this -- people will always have their own custom stuff, and at some point (hopefully) the number of types will be so great that we''ll need a stdlib and then additional plugins.> Sometimes, modules (as currently defined, ie .pps and templates) > will rely > on new types; waiting until those new types get into core would be > unpleasant. Some types/modules will really only be useful to a small > subset of people -- not-so-useful to put into core. Coupling the > release of > new versions of types to core releases will, I am fairly certain, > become > untenable. > > Basically, think of all the reasons why Apache, Eclipse, and Rails > don''t > bundle every possible plugin with their core releases, and you''ve > just got > my argument against bundling all possible types in core.Yep. Nagios is a perfect example in this space.>> 3. it feels like there''s a slippery slope here where puppet will >> grow its own package management system > > What is PRM if not some form of package management? (Package > management by > direct manipulation of revision control is still package management).Indeed. Scary but true.>> Having said all that, bundling modules, plugins and functions into >> one >> easily installable thing does sound very attractive - maybe it would >> take some examples of what that would enable to clear things up. > > A big "Nagios Config" module is the canonical example. New types, > some > templates, some .pp files for classes, and probably even a function > or two. > Having seen the speed with which people have come up with new > classes and > defines of their own and published them on the wiki, I suspect that > a "build > it[1] and they will come" attitude will work wonders.On a similar note, while I was creating the Fact storage stool for the first step in the Node Classification process, I realized that I''m starting to get a lot of auto-loading directories scattered all over. I''m thinking of creating a single ''plugins'' directory in lib/ puppet, and then extending autoload.rb to look in the specified path (e.g., ''puppet/type'') plus the plugins directory (''puppet/plugins/ type'', in this case). This way people would have a clean way to separate what they''ve added to Puppet vs. what came with the system. Alternatively, I could just move everything that autoloads under that directory, so that people could always tell what they could load vs. what was part of the "core". This is certainly clean -- you could cd into lib/puppet/plugins and see that you can autoload functions, reports, types, providers, and much more, all in one place, and you''d see all of the possible load targets, too. Comments? -- The people who are regarded as moral luminaries are those who forego ordinary pleasures themselves and find compensation in interfering with the pleasures of others. -- Bertrand Russell --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Matthew Palmer
2007-May-04 22:32 UTC
Re: modules, modules, modules! [was: escape regex code]
On Fri, May 04, 2007 at 04:12:12PM -0500, Luke Kanies wrote:> On May 3, 2007, at 8:38 PM, Matt Palmer wrote: > > > > Yep, it''s a massive potential problem, but one that''s inherent in any > > extensible system. > > Not *my* system. I''ll magically find a way to avoid allowing plugins > to destroy Puppet. :)(We were talking about API changes breaking plugins, actually) The way to solve this is to be really, really strict about defining plugin APIs, having lots of test code to flag if the API starts to shift, and an up-front way for plugin authors to know whether APIs might change (a special announcements mailing list, perhaps). Versioning APIs and keeping the old ones around for a while for backwards compatibility is useful, but can be a real drag on core dev resources unless there''s a clear statement of "you''ve got 6 months to update your types and then this old version of the API will go away". Otherwise people won''t update their external code. Regular changes in API just discourages people from writing code in the first place.> > As far as I''m concerned, most non-trivial uses of Puppet are, in > > some way, > > extending Puppet. Apart from the language used and the degree, > > writing a > > generally useful .pp file like, say, replace_line.pp is little-to-no > > different to writing a Ruby class to provide a ''nagios_service'' type. > > That''s what I thought, but we certainly see a lot more Puppet code > than Ruby code.There''s a number of reasons for that: * Everyone who works with Puppet needs to get a pretty good handle on the manifest language, while knowledge of Ruby is a (useful) optional extra. * The Ruby-level interfaces aren''t nearly as well (or as prominently, at least) documented, so the barrier to entry is quite a bit higher there, too. * You can do an awful lot of things in the manifest language, so the drive to learn the deeper secrets is diminished. (So it''s all your fault, Luke, for creating such a useful and expressive language). * The deployment costs for manifest code is much lower than for Ruby code. At least, those are *my* reasons (with the exception of the first point) why I write most everything in .pp files instead of extending the core in Ruby. There''s several things I want to rewrite, though (the file editing primitives, and some internal stuff I''ve got for manipulating Postfix configurations).> > Basically, think of all the reasons why Apache, Eclipse, and Rails > > don''t > > bundle every possible plugin with their core releases, and you''ve > > just got > > my argument against bundling all possible types in core. > > Yep. Nagios is a perfect example in this space.I thought I''d been banging the Nagios drum a bit hard, so I dug for some other examples. But yes, Nagios is a good example of the benefits (and downsides) of making plugins easy to write and widely distributed.> On a similar note, while I was creating the Fact storage stool for > the first step in the Node Classification process, I realized that > I''m starting to get a lot of auto-loading directories scattered all > over. I''m thinking of creating a single ''plugins'' directory in lib/ > puppet, and then extending autoload.rb to look in the specified path > (e.g., ''puppet/type'') plus the plugins directory (''puppet/plugins/ > type'', in this case). > > This way people would have a clean way to separate what they''ve added > to Puppet vs. what came with the system. > > Alternatively, I could just move everything that autoloads under that > directory, so that people could always tell what they could load vs. > what was part of the "core". This is certainly clean -- you could cd > into lib/puppet/plugins and see that you can autoload functions, > reports, types, providers, and much more, all in one place, and you''d > see all of the possible load targets, too. > > Comments?Centralising all plugins into one "here''s where you put your own code" location is a fantastic idea. I''m still convinced that providing all that code via modules, via the Puppetmaster, is the way to go longer term[1], but that''s a big chunk of work, and I think there''s definite benefits in getting people into the habit of dropping their own work in $LOAD_PATH/puppet/plugins/<whatever> in the meantime. It''ll at least make people more confident that their own code isn''t going to go *foof* on upgrade... - Matt [1] It just makes distribution of custom code *so* much easier if it can all be pushed via the Puppetmaster, but as we all know, the usual file copying just won''t work due to chicken-and-egg issues. Hence the need for specialist Puppetmaster-to-client mechanisms for pushing custom in-client code. -- How many Apple Newton users does it take to change a lightbulb? Foux. There to eat lemons, axe gravy soup. -- Seen on the ''net
On May 4, 2007, at 5:32 PM, Matthew Palmer wrote:> > (We were talking about API changes breaking plugins, actually)Whatever; my point was that Puppet will magically be immune to those kinds of problems. :P> The way to solve this is to be really, really strict about defining > plugin > APIs, having lots of test code to flag if the API starts to shift, > and an > up-front way for plugin authors to know whether APIs might change > (a special > announcements mailing list, perhaps). Versioning APIs and keeping > the old > ones around for a while for backwards compatibility is useful, but > can be a > real drag on core dev resources unless there''s a clear statement of > "you''ve > got 6 months to update your types and then this old version of the > API will > go away". Otherwise people won''t update their external code. Regular > changes in API just discourages people from writing code in the > first place.I''ve certainly been relatively ad-hoc in this area so far -- I do my best to keep backwards-compatible, but I know I''ll end up breaking things eventually. I''ve been expecting to get more rigourous as I found people doing more extension. I know Allen Ballman is about to merge a bunch of code that significantly changes and/or eliminates the usage of @is and @should in the system. I believe he''s done it in a way that largely keeps backwards compatibility, but it''s just the first step to even more significant rewriting of how the main resource abstraction layer works. So, if you''ve got a custom type that I don''t know about, maybe you should make sure I''m aware of it, and please, if you''ve got a Ruby extension to Puppet, *please* test it against the source when I announce a release candidate. The only way I can find these backward- compatibility bugs before release is with your help.> There''s a number of reasons for that: > > * Everyone who works with Puppet needs to get a pretty good handle > on the > manifest language, while knowledge of Ruby is a (useful) optional > extra.Yep.> * The Ruby-level interfaces aren''t nearly as well (or as > prominently, at > least) documented, so the barrier to entry is quite a bit higher > there, > too.This is definitely a problem; and in some cases it''s actually easier to hack up a quick shell script to get something done than to write the Ruby module for it (although I''ve made a good amount of progress on this). I''m working on the documentation problem. I''ve set myself the goal of fixing one high-priority bug every couple of days right now, and that''s probably my next one.> * You can do an awful lot of things in the manifest language, so > the drive > to learn the deeper secrets is diminished. (So it''s all your > fault, Luke, > for creating such a useful and expressive language).Everything is always my fault. :/> * The deployment costs for manifest code is much lower than for > Ruby code.That''s not true for types and facts -- you can use pluginsync and factsync. Soon (another high priority bug -- #534) you''ll be able to do that with any file (and I''ll correct the terminology problem of using "plugin" to mean "custom type"). Is anyone using --pluginsync right now?> At least, those are *my* reasons (with the exception of the first > point) why > I write most everything in .pp files instead of extending the core > in Ruby. > There''s several things I want to rewrite, though (the file editing > primitives, and some internal stuff I''ve got for manipulating Postfix > configurations).Ok.> I thought I''d been banging the Nagios drum a bit hard, so I dug for > some > other examples. But yes, Nagios is a good example of the benefits > (and > downsides) of making plugins easy to write and widely distributed.Downsides? :)> Centralising all plugins into one "here''s where you put your own code" > location is a fantastic idea. I''m still convinced that providing > all that > code via modules, via the Puppetmaster, is the way to go longer term > [1], but > that''s a big chunk of work, and I think there''s definite benefits > in getting > people into the habit of dropping their own work in > $LOAD_PATH/puppet/plugins/<whatever> in the meantime. It''ll at > least make > people more confident that their own code isn''t going to go *foof* on > upgrade...s/$LOAD_PATH/$RUBYLIB/ Maybe that''s why someone on IRC thought that was how to set Ruby''s search path the other day... I''ll plan on adding this ASAP (#621). Should I put all loadable code there, or just non-standard-lib code? For example, Puppet ships with three transaction report types -- should those be in plugins/reports, or should they stay in reports/ and just custom stuff go in plugins/reports? Should the differentiation be between that code that''s required (files, users, groups, a few others) and those that are optional (solaris zones, reports, etc.)? Or just, if it''s part of the distribution it goes in the normal tree, everything else goes in plugins?> - Matt > > [1] It just makes distribution of custom code *so* much easier if > it can all > be pushed via the Puppetmaster, but as we all know, the usual file > copying > just won''t work due to chicken-and-egg issues. Hence the need for > specialist Puppetmaster-to-client mechanisms for pushing custom in- > client > code.Yep, that''s why it already exists; it''s just not as flexible as it needs to be yet. -- The time to repair the roof is when the sun is shining. -- John F. Kennedy --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
C.M. Connelly
2007-May-04 23:22 UTC
Re: modules, modules, modules! [was: escape regex code]
"MP" == Matthew Palmer <mpalmer@hezmatt.org>
MP> Centralising all plugins into one "here''s where you put
MP> your own code" location is a fantastic idea. I''m still
MP> convinced that providing all that code via modules, via
MP> the Puppetmaster, is the way to go longer term[1], but
MP> that''s a big chunk of work, and I think there''s
definite
MP> benefits in getting people into the habit of dropping
MP> their own work in $LOAD_PATH/puppet/plugins/<whatever> in
MP> the meantime. It''ll at least make people more confident
MP> that their own code isn''t going to go *foof* on upgrade...
It would also be a win for packagers, as they''d have one true home
to install plug-in modules when packaging.
On that note, though, it might be worth thinking about a couple of
things before making it happen. The first thing to consider is
whether you''d want a cascade of directories, so that, say, local
code could override system code, and user code overrides local
code.
The second is the location of the various directories. In the
ideal world they''d all be configurable, but it would probably be a
good thing to have some sane defaults to start with.
Probably a good place for ``system code'''' would be
/usr/lib/puppet/plugins if they''re platform-specific or
/usr/share/puppet/plugins if they''re platform-neutral. Local code
might then go in equivalent directories under /usr/local, and user
code would probably be up in the air.
Claire
(Sorry if some of this stuff is already known and set; I''m just
getting started with Puppet but have been packaging software for
years, and dealing with similar sorts of cascades of code with my
work with teTeX.)
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
Claire Connelly cmc@math.hmc.edu
Systems Administrator (909) 621-8754
Department of Mathematics Harvey Mudd College
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users
Matthew Palmer
2007-May-05 00:47 UTC
Re: modules, modules, modules! [was: escape regex code]
On Fri, May 04, 2007 at 06:02:37PM -0500, Luke Kanies wrote:> On May 4, 2007, at 5:32 PM, Matthew Palmer wrote: > > * The deployment costs for manifest code is much lower than for > > Ruby code. > > That''s not true for types and facts -- you can use pluginsync and > factsync.It''s still easier to deploy manifest code -- stick your .pp file in an imported directory. I''ve not used pluginsync and factsync because they''re different, and hence something extra to worry about. <grin>> > I thought I''d been banging the Nagios drum a bit hard, so I dug for > > some > > other examples. But yes, Nagios is a good example of the benefits > > (and > > downsides) of making plugins easy to write and widely distributed. > > Downsides? :)Many different plugins that do approximately (but not quite exactly) the same job; complete unknown level of code quality and adherence to standards; difficulty sometimes in finding something that does what you want it to do; once you do find something (or usually a whole bunch) which one do you choose; and so on.> > Centralising all plugins into one "here''s where you put your own code" > > location is a fantastic idea. I''m still convinced that providing > > all that > > code via modules, via the Puppetmaster, is the way to go longer term > > [1], but > > that''s a big chunk of work, and I think there''s definite benefits > > in getting > > people into the habit of dropping their own work in > > $LOAD_PATH/puppet/plugins/<whatever> in the meantime. It''ll at > > least make > > people more confident that their own code isn''t going to go *foof* on > > upgrade... > > s/$LOAD_PATH/$RUBYLIB/irb(main):001:0> $RUBYLIB => nil irb(main):002:0> $LOAD_PATH => ["/usr/local/lib/site_ruby/1.8", "/usr/local/lib/site_ruby/1.8/i486-linux", "/usr/local/lib/site_ruby/1.8/i386-linux", "/usr/local/lib/site_ruby", "/usr/lib/ruby/1.8", "/usr/lib/ruby/1.8/i486-linux", "/usr/lib/ruby/1.8/i386-linux", "."]> Should I put all loadable code there, or just non-standard-lib code? > For example, Puppet ships with three transaction report types -- > should those be in plugins/reports, or should they stay in reports/ > and just custom stuff go in plugins/reports? > > Should the differentiation be between that code that''s required > (files, users, groups, a few others) and those that are optional > (solaris zones, reports, etc.)? Or just, if it''s part of the > distribution it goes in the normal tree, everything else goes in > plugins?It''s a tricky one. I see several different classifications: * Core. Must be loaded for Puppet to work at all. * Core Plugins. Runtime-loadable, but comes in the Puppet distribution tarball. * Extra Plugins. User-supplied code; presumably runtime-loaded. Core obviously lives in $LOAD_PATH/puppet, and Extra Plugins is obviously $LOAD_PATH/puppet/plugins. Core Plugins is the middle ground, and really could go either way. If you think the primary purpose for doing this split is to differentiate between Core and non-core, then Core Plugins should live with Extra Plugins -- because they''re all "plugins" in the sense that Puppet doesn''t really /need/ that code to start (do-something-useful is a whole other thing). This might be the case if you were interested in doing a ''Nagios vs Nagios Plugins'' style split. Conversely, if the most important thing is to give users somewhere to stash their stuff where they can be confident that no addition to core is going to run into their stuff, then Core Plugins should live with Core, and Extra Plugins is the off-to-the-side cousin. I know that isn''t a decision, but maybe it illuminates the issues in a different way to help you make a decision. - Matt -- I killed init(8) and all I got was this lousy kernel panic. -- Graham Reed, in the Monastery
On May 4, 2007, at 7:47 PM, Matthew Palmer wrote:> > It''s still easier to deploy manifest code -- stick your .pp file in an > imported directory. I''ve not used pluginsync and factsync because > they''re > different, and hence something extra to worry about. <grin>Eh. I stick a fact in the factdir and it gets deployed and loaded with no extra work (not even an import!).>> Downsides? :) > > Many different plugins that do approximately (but not quite > exactly) the > same job; complete unknown level of code quality and adherence to > standards; > difficulty sometimes in finding something that does what you want > it to do; > once you do find something (or usually a whole bunch) which one do you > choose; and so on.Geesh. It was a joke -- I''m painfully familiar with the downsides of making this kind of thing easy.>> >> s/$LOAD_PATH/$RUBYLIB/ > > irb(main):001:0> $RUBYLIB > => nil > irb(main):002:0> $LOAD_PATH > => ["/usr/local/lib/site_ruby/1.8", > "/usr/local/lib/site_ruby/1.8/i486-linux", > "/usr/local/lib/site_ruby/1.8/i386-linux", "/usr/local/lib/site_ruby", > "/usr/lib/ruby/1.8", "/usr/lib/ruby/1.8/i486-linux", > "/usr/lib/ruby/1.8/i386-linux", "."]Ah; you''re speaking from within Ruby, rather than without. I just know someone on IRC was trying to set LOADPATH in order to get Ruby to load new programs, and while that might work in Ruby you have to use RUBYLIB in the shell.> It''s a tricky one. I see several different classifications: > > * Core. Must be loaded for Puppet to work at all. > > * Core Plugins. Runtime-loadable, but comes in the Puppet > distribution > tarball. > > * Extra Plugins. User-supplied code; presumably runtime-loaded. > > Core obviously lives in $LOAD_PATH/puppet, and Extra Plugins is > obviously > $LOAD_PATH/puppet/plugins. Core Plugins is the middle ground, and > really > could go either way. > > If you think the primary purpose for doing this split is to > differentiate > between Core and non-core, then Core Plugins should live with Extra > Plugins > -- because they''re all "plugins" in the sense that Puppet doesn''t > really > /need/ that code to start (do-something-useful is a whole other > thing). > This might be the case if you were interested in doing a ''Nagios vs > Nagios > Plugins'' style split. > > Conversely, if the most important thing is to give users somewhere > to stash > their stuff where they can be confident that no addition to core is > going to > run into their stuff, then Core Plugins should live with Core, and > Extra > Plugins is the off-to-the-side cousin. > > I know that isn''t a decision, but maybe it illuminates the issues in a > different way to help you make a decision.I can add another reason -- if someone wants to replace a "core plugin", then if we stick them in ''plugins'' then they have to modify or replace core code, but if we stick them in ''puppet'', then they can just drop in a replacement and in with the right search path (plugins then puppet), their code will get caught before ours. The only downside is that it means there''s no easy way to list available plugins, but I could add some kind of reporting function to provide that info. Maybe I should extend puppetdoc -- there are a few different doc- style reports I''ve been envisioning, and maybe it''s the right place to put them (e.g., a report on which providers work on the given platform). I''d like to get the reference headers out of puppetdoc, but I can''t decide on the best place to put them. -- Humphrey''s Law of the Efficacy of Prayer: In a dangerous world there will always be more people around whose prayers for their own safety have been answered than those whose prayers have not. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On May 4, 2007, at 6:22 PM, C.M. Connelly wrote:> > It would also be a win for packagers, as they''d have one true home > to install plug-in modules when packaging.That''s a good point.> On that note, though, it might be worth thinking about a couple of > things before making it happen. The first thing to consider is > whether you''d want a cascade of directories, so that, say, local > code could override system code, and user code overrides local > code. > > The second is the location of the various directories. In the > ideal world they''d all be configurable, but it would probably be a > good thing to have some sane defaults to start with. > > Probably a good place for ``system code'''' would be > /usr/lib/puppet/plugins if they''re platform-specific or > /usr/share/puppet/plugins if they''re platform-neutral. Local code > might then go in equivalent directories under /usr/local, and user > code would probably be up in the air.Certainly it shouldn''t be too hard to have a ''plugin path''. I ''load'' files directly, rather than using Ruby''s ''require'', so the paths don''t need to be in $LOAD_PATH, and it''s no harder for the Autoload module to look through 4 directories vs. two. We''d just have ''pluginpath'' as a configurable parameter, so people could override it to be whatever they wanted. Are the directories you mentioned the LSB standards? If not, what should we use? -- "They called me mad, and I called them mad, and damn them, they outvoted me." -- Nathaniel Lee, on being consigned to a mental institution, circa 17th c. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Matthew Palmer
2007-May-08 04:12 UTC
Re: modules, modules, modules! [was: escape regex code]
On Mon, May 07, 2007 at 11:18:22AM -0500, Luke Kanies wrote:> On May 4, 2007, at 6:22 PM, C.M. Connelly wrote: > > Probably a good place for ``system code'''' would be > > /usr/lib/puppet/plugins if they''re platform-specific or > > /usr/share/puppet/plugins if they''re platform-neutral. Local code > > might then go in equivalent directories under /usr/local, and user > > code would probably be up in the air. > > Are the directories you mentioned the LSB standards? If not, what > should we use?Yes, they''re LSB-compliant. - Matt -- When the revolution comes, they won''t be able to FIND the wall. -- Brian Kantor, in the Monastery