Hi,
I was talking to Luke today about managing manifests and related
files, and he suggested that I post to the list..
What would be nice (and what I''ve use in in-house systems I''ve
used
previously) is to have a way to dynamically include manifest fragments
at runtime.
Lets assume that you''re setting up bugzilla using puppet. In the
''main'' manifest file, site.pp, you have:
node bugzilla {
svnserve { ''/opt/bugzilla'':
module => ''bugzilla'',
rev => ''release_20070117'',
}
}
Then within the bugzilla module, theres a manifest.pp file that has:
class XXX { # Syntax???
# Needs a webserver
include webserver
# Package
package { bugzilla: }
# Optional module we want, not pulled in by RPM dependancies
package { ''perl-Chart'': }
# Custom template
file {
"/usr/share/bugzilla/template/en/custom/pages/using-bugzilla.html.tmpl":
# $_path expands to /opt/bugzilla, because thats where the module
was installed.
source => "$_path/templates/using-bugzilla.html.tmpl",
}
# Apache config file
file { "/etc/httpd/conf/bugzilla.conf":
#NB - this needs local template expansion, since the file isn''t
accessible on the server
source => template("$_path/conf/bugzilla-httpd.conf"),
notify => Service[httpd],
}
}
The advantage of this is that all the files and config stays with the
module. If a newer bugzilla version comes out that needs a different
http config file, and some extra modules, then the whole lot can be
changed and tagged together, and released by just changing the
''rev''
tag in the main file. Test systems can just pull a different tag from
the main file, so a test->prod release just becomes a matter of
changing the release string.
Its a bit tricky if we let this happen recursively, with the module''s
manifest file able to pull other modules which need manifests
included. There are also ordering issues, eg if one of the bits here
has a |before| tag for something in a differnent manifest file.
Does this sound sensible/reasonable/a good idea?
Thanks,
Bradley
On Wed, 2007-01-17 at 22:53 +1100, Bradley Baetz wrote:> I was talking to Luke today about managing manifests and related > files, and he suggested that I post to the list.. > > What would be nice (and what I''ve use in in-house systems I''ve used > previously) is to have a way to dynamically include manifest fragments > at runtime.I think the issue of managing puppet manifests is pretty important; I am not convinced though that puppet itself should know about that, much less that that should be expressed in the manifest. The puppet manifest and all supporting files (like customized config files etc.) definitely need to go into a SCM; but should that be one source repo for the complete site config or one for each ''module'' ? I don''t think that the added flexibility of one repo per module buys you that much, seeing how puppet''s language already does a good job of modularizing and organizing configs; it''ll make testing much harder and increase the risk of working off the wrong checkout. What''s wrong with leaving puppet SCM-agnostic, and running puppetmaster off a checkout from your favorite SCM, with decent processes around how you make a ''release'' of your site''s config, including tagging, moving from testing to prod etc. ? There''s definitely room for tooling around that .. essentially, we''d define puppet''s job as enforcing one specific state for a site, and leave moving from state to state to another tool (and I would love to hear what that other tool should be and do) David
David Lutterkort wrote:> I think the issue of managing puppet manifests is pretty important; I am > not convinced though that puppet itself should know about that, much > less that that should be expressed in the manifest. > > The puppet manifest and all supporting files (like customized config > files etc.) definitely need to go into a SCM; but should that be one > source repo for the complete site config or one for each ''module'' ? I > don''t think that the added flexibility of one repo per module buys you > that much, seeing how puppet''s language already does a good job of > modularizing and organizing configs; it''ll make testing much harder and > increase the risk of working off the wrong checkout.Sure. I was using svn as an example, and as I said I''m not sure what the syntax should be. A |localmanifest| type, maybe? The idea is that the actual application code and its config are then linked, because they''re in the same module. So if you have some internal webapp, the actual app, any config files that need to be installed somehwere else (eg apache config files, stuff in logrotate.d, whatever), required modules, cron jobs, and so on. Then theres *less* risk of stuff getting out of sync, simply because its all together.> What''s wrong with leaving puppet SCM-agnostic, and running puppetmaster > off a checkout from your favorite SCM, with decent processes around how > you make a ''release'' of your site''s config, including tagging, moving > from testing to prod etc. ?If you have more than a few services, though, stuff will go into test and prod at different time, potentially with different development groups involved. Even if each module has its own file (included into the main manifest file via globbing), its still a lot easier to have a process that says "To release an app into prod, change its tag to whats in test, and to rollback just revert that change", rather than "take the data from the app_test class, and make it match whats in the app_prod class, then make sure that the source templates/files are made available to the new machine as well". It also makes it easier to go back through SCM history if its not full of gobs of config stuff being moved from one class definition to another. > There''s definitely room for tooling around> that .. essentially, we''d define puppet''s job as enforcing one specific > state for a site, and leave moving from state to state to another tool > (and I would love to hear what that other tool should be and do)Sure, and I''m not suggesting changing that. It would certainly be possible to change puppet to have an ''environment'' tag, which was auto-prefixed to the filebucket, or something, and then check out different tags to test/app and prod/app using regular methods. But thats a bit clunky, and still doesn''t cover what to do with the class definition has to change too (extra packages/cron jobs/etc) Bradley
On Jan 17, 2007, at 5:53 AM, Bradley Baetz wrote:> Hi, > > I was talking to Luke today about managing manifests and related > files, and he suggested that I post to the list.. > > What would be nice (and what I''ve use in in-house systems I''ve used > previously) is to have a way to dynamically include manifest fragments > at runtime. > > Lets assume that you''re setting up bugzilla using puppet. In the > ''main'' manifest file, site.pp, you have: > > node bugzilla { > svnserve { ''/opt/bugzilla'': > module => ''bugzilla'', > rev => ''release_20070117'', > } > } > > Then within the bugzilla module, theres a manifest.pp file that has: > > class XXX { # Syntax??? > # Needs a webserver > include webserver > > # Package > package { bugzilla: } > > # Optional module we want, not pulled in by RPM dependancies > package { ''perl-Chart'': } > > # Custom template > file { "/usr/share/bugzilla/template/en/custom/pages/using- > bugzilla.html.tmpl": > # $_path expands to /opt/bugzilla, because thats where the module > was installed. > source => "$_path/templates/using-bugzilla.html.tmpl", > } > > # Apache config file > file { "/etc/httpd/conf/bugzilla.conf": > #NB - this needs local template expansion, since the file isn''t > accessible on the server > source => template("$_path/conf/bugzilla-httpd.conf"), > notify => Service[httpd], > } > }So, to be clear, the only thing you''re positing here is that there be a way to group files so that they all share a path? That is, for a given manifest, Puppet would assume that file is in a ''manifests'' directory, and its parent directory could also have a ''templates'' and maybe a ''files'' directory? Is the stuff in /opt/bugzilla the Puppet manifests and templates, or the bugzilla code? I''m kinda confused. Hmm. At the least, David is right that you could do this relatively easily right now using variable expansion. I could do a few things to make it even easier, I expect, but overall I''d say this belongs in the manifest, not in Puppet''s internal code.> The advantage of this is that all the files and config stays with the > module. If a newer bugzilla version comes out that needs a different > http config file, and some extra modules, then the whole lot can be > changed and tagged together, and released by just changing the ''rev'' > tag in the main file. Test systems can just pull a different tag from > the main file, so a test->prod release just becomes a matter of > changing the release string.You can accomplish this right now, though, basically, just by modifying the svnserve type to support the ''rev'' parameter. Puppet should probably support some way to modify the default search path for manifests and templates, or maybe a chdir-like operation that will always load files relative to the current file.> Its a bit tricky if we let this happen recursively, with the module''s > manifest file able to pull other modules which need manifests > included. There are also ordering issues, eg if one of the bits here > has a |before| tag for something in a differnent manifest file.The lines between the client and server are clean enough that this shouldn''t be a problem, I''d think, but it might also mean that you can''t actually get what you want.> Does this sound sensible/reasonable/a good idea?Something like this definitely needs to happen, but I think it should sit at a layer above Puppet, not inside the tool itself. -- Good judgment comes from experience, and experience comes from bad judgment. --Barry LePatner --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 18, 2007, at 3:47 AM, Bradley Baetz wrote:> > Sure. I was using svn as an example, and as I said I''m not sure > what the > syntax should be. A |localmanifest| type, maybe? > > The idea is that the actual application code and its config are then > linked, because they''re in the same module. So if you have some > internal > webapp, the actual app, any config files that need to be installed > somehwere else (eg apache config files, stuff in logrotate.d, > whatever), > required modules, cron jobs, and so on. > > Then theres *less* risk of stuff getting out of sync, simply > because its > all together.I do like the idea of organizing files by their semantic relationship -- i.e., manifests, templates, and data for a given app -- but that seems to be only a small part of what you''re talking about. I''m quite willing to do something that supports that, although I expect I''m looking for a smaller change than you are. It should have some kind of integration with PRM, which needs attention anyway.>> What''s wrong with leaving puppet SCM-agnostic, and running >> puppetmaster >> off a checkout from your favorite SCM, with decent processes >> around how >> you make a ''release'' of your site''s config, including tagging, moving >> from testing to prod etc. ? > > If you have more than a few services, though, stuff will go into test > and prod at different time, potentially with different development > groups involved. Even if each module has its own file (included > into the > main manifest file via globbing), its still a lot easier to have a > process that says "To release an app into prod, change its tag to > whats > in test, and to rollback just revert that change", rather than > "take the > data from the app_test class, and make it match whats in the app_prod > class, then make sure that the source templates/files are made > available > to the new machine as well".This is pretty standard SCM operation -- you''d just merge changes from test to production, for instance. So it sounds like you''re talking about at least two separate abilities: One for implicitly associating an app''s manifests, templates, and data, and another for explicitly integrating "releases" into Puppet. I like both ideas, but they''re orthogonal and should be treated as such. The first is relatively straightforward and could be done with another keyword akin to require or something. The second is something that we''ve been thinking and talking about for ages, and I''m pretty hesitant to make much of a commitment here until we''ve better characterized what we''re looking for. And, as I''ve mentioned, I think the second part -- giving Puppet the ability to talk about releases -- is a higher-level problem. I''m thinking something along the lines of adding a dimension to every class in Puppet, with that extra dimension being a release tag. E.g., you could do something like ''require myclass/test'' or something (no, I''m not committing anything resembling that syntax). I do kind of like Debian''s idea of ''releases'', so all of your machines would run at a specific release-level, and Puppet would only look up classes at that release level when compiling a configuration. Of course, it''s just about as easy to just run a puppetmaster for each release level, backed against that tagged release in your SCM. So, once again the KISS principle probably looks to using process to do this, rather than anything in Puppet itself.> It also makes it easier to go back through SCM history if its not full > of gobs of config stuff being moved from one class definition to > another.It''s all about the merge. :) -- Sabbagh''s Second Law: The biggest problem with communication is the illusion that it has occurred. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday 23 January 2007 02:06, Luke Kanies wrote: [...]> So, once again the KISS principle probably looks to using process to > do this, rather than anything in Puppet itself.I would think that the documented best practices around /usr/bin/puppet* are as much part of Puppet (capital P) as everything else. To provide my solution for the different releases: I have a variable $dv which contains the release name and most source parameters are of the form "puppet://puppet/$dv/..." and some places decide on this variable which packages to install for example. The variable is currently set on the node level, but could be retrieved by facter. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFtgo4/Pp1N6Uzh0URAoAEAJ9w1VUNd/rJdf03ZhNyCdoC7zyOnQCffxLG OZWNcJrKNGLXo0OabkrG2Qs=/CCA -----END PGP SIGNATURE-----
On Jan 23, 2007, at 7:14 AM, David Schmitt wrote:> > I would think that the documented best practices around /usr/bin/ > puppet* are > as much part of Puppet (capital P) as everything else.Yes, sorry; I didn''t mean to imply that it shouldn''t be discussed and documented, just that it probably shouldn''t be added to existing tools but should instead be a separate tool or process.> To provide my solution for the different releases: > > I have a variable $dv which contains the release name and most source > parameters are of the form "puppet://puppet/$dv/..." and some > places decide > on this variable which packages to install for example. > > The variable is currently set on the node level, but could be > retrieved by > facter.This is exactly the kind of thing people are looking for. I expect others would quite appreciate it if you''d be willing to document this on the wiki. -- Ninety-eight percent of the adults in this country are decent, hard-working, honest Americans. It''s the other lousy two percent that get all the publicity. But then--we elected them. --Lily Tomlin --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday 24 January 2007 19:17, Luke Kanies wrote:> > To provide my solution for the different releases: > > > > I have a variable $dv which contains the release name and most source > > parameters are of the form "puppet://puppet/$dv/..." and some > > places decide > > on this variable which packages to install for example. > > > > The variable is currently set on the node level, but could be > > retrieved by > > facter. > > This is exactly the kind of thing people are looking for. I expect > others would quite appreciate it if you''d be willing to document this > on the wiki.I''ve started https://reductivelabs.com/trac/puppet/wiki/DebianRecipies Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFuLuX/Pp1N6Uzh0URAoxvAJ9MHIrgaF4WbD1GbO05WQDeHJEttQCfe7iL qJm/+aBi6xPrNwQeFsPdJ0o=crVD -----END PGP SIGNATURE-----
On Jan 25, 2007, at 8:15 AM, David Schmitt wrote:> > I''ve started > https://reductivelabs.com/trac/puppet/wiki/DebianRecipiesVery nice. Looks like consolidating to a single wiki was a good plan, since it''s gotten a good bit of use. I''ve been meaning to install a notification plugin, but in the meantime, note that one can subscribe to an RSS feed of the Trac timeline. You can selectively choose any combination of changes to the wiki, tickets, or code. -- Death and taxes are inevitable; at least death doesn''t get worse every year. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, 2007-01-18 at 20:47 +1100, Bradley Baetz wrote:> David Lutterkort wrote: > Even if each module has its own file (included into the > main manifest file via globbing), its still a lot easier to have a > process that says "To release an app into prod, change its tag to whats > in test, and to rollback just revert that change", rather than "take the > data from the app_test class, and make it match whats in the app_prod > class, then make sure that the source templates/files are made available > to the new machine as well".You definitely want to avoid any manual work in the test -> prod transition; but why can''t you do that with an SCM ? Make test run off a checkout with HEAD (or revision N), and prod run off a checkout of revision M (presumably M <= N) The sticky point is that you really want to keep manifests, files and templates closely together; ideally, you could have them all in one directory, like /some/where/autofs autofs.pp # Manifest for my autofs setup auto.homes # literal config file auto.master.templ # template for auto.master While you can do that with puppet right now, it''s probably not the most straightforward setup; in addition, you''ll have to make that dir available through the fileserver, which means that you are either serving all files in the dir through the fileserver, or have to play games with permissions. I think it would be immensely worthwhile to have a discussion about the ideal file layout on the server, and then make that puppet''s default config[1]. What do people think about this ''one dir per module'' layout ? (Module here could mean several classes, but classes that are tightly interconnected)> It also makes it easier to go back through SCM history if its not full > of gobs of config stuff being moved from one class definition to another.Agreed, but I think that''s an artifact of how you are using the SCM. David [1] That''s one thing that makes rails so much easier to work with than, say J2EE: everything is defaulted to something reasonable, and you are gently forced to follow best practices by those defaults.
On Jan 25, 2007, at 2:18 PM, David Lutterkort wrote:> > You definitely want to avoid any manual work in the test -> prod > transition; but why can''t you do that with an SCM ? Make test run > off a > checkout with HEAD (or revision N), and prod run off a checkout of > revision M (presumably M <= N)This just doesn''t work, in my experience. The only reasonable solution, IMO, is branches. Otherwise you can''t have some modules that take months to get to production and others that take weeks. Branches have been working well for decades in the development world; we should copy them and be done.> The sticky point is that you really want to keep manifests, files and > templates closely together; ideally, you could have them all in one > directory, like > /some/where/autofs > autofs.pp # Manifest for my autofs setup > auto.homes # literal config file > auto.master.templ # template for auto.master > > While you can do that with puppet right now, it''s probably not the > most > straightforward setup; in addition, you''ll have to make that dir > available through the fileserver, which means that you are either > serving all files in the dir through the fileserver, or have to play > games with permissions.We could add globbing to the fileserver.conf file, so you could do something like ''allow /some/where/*/files''. I think subdirectories are a good idea anyway, so that templates, manifests, and data files are all in separate directories.> I think it would be immensely worthwhile to have a discussion about > the > ideal file layout on the server, and then make that puppet''s default > config[1]. What do people think about this ''one dir per module'' > layout ? > (Module here could mean several classes, but classes that are tightly > interconnected)I completely agree.> [1] That''s one thing that makes rails so much easier to work with > than, > say J2EE: everything is defaulted to something reasonable, and you are > gently forced to follow best practices by those defaults.I definitely agree with this, too. I''ve done my best to have reasonable defaults everywhere in Puppet, but this is an area we haven''t explored much. PRM has touched on it some, but it has been basically lying fallow, unfortunately (and I''m certainly muchly to blame for that). Any defaults we choose here should be chosen towards the goal of making redistribution easy. That is, our default design should choose from among the intersection between those solutions that work well for normal management and those that work well for redistribution. -- Someday I want to be rich. Some people get so rich they lose all respect for humanity. That''s how rich I want to be. --Rita Rudner --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, 2007-01-25 at 15:14 -0600, Luke Kanies wrote:> On Jan 25, 2007, at 2:18 PM, David Lutterkort wrote: > > > > You definitely want to avoid any manual work in the test -> prod > > transition; but why can''t you do that with an SCM ? Make test run > > off a > > checkout with HEAD (or revision N), and prod run off a checkout of > > revision M (presumably M <= N)For starters, I got my ''>='' and ''<='' mixed up :(> This just doesn''t work, in my experience. The only reasonable > solution, IMO, is branches. Otherwise you can''t have some modules > that take months to get to production and others that take weeks. > > Branches have been working well for decades in the development world; > we should copy them and be done.What I meant to say is that things should be structured in such a way that ''releasing'' a config from test to prod should be no more than checking out a specific tag from the SCM, one that has been tested before, just as you would do when going from a release candidate to a release in a software release (minus the complications of packagign the release)> > The sticky point is that you really want to keep manifests, files and > > templates closely together; ideally, you could have them all in one > > directory, like > > /some/where/autofs > > autofs.pp # Manifest for my autofs setup > > auto.homes # literal config file > > auto.master.templ # template for auto.master > > > > While you can do that with puppet right now, it''s probably not the > > most > > straightforward setup; in addition, you''ll have to make that dir > > available through the fileserver, which means that you are either > > serving all files in the dir through the fileserver, or have to play > > games with permissions. > > We could add globbing to the fileserver.conf file, so you could do > something like ''allow /some/where/*/files''.I assume you mean ''path'' here, right ?> I think subdirectories are a good idea anyway, so that templates, manifests, > and data files are all in separate directories.Yep, that would work, too; I was thinking of going the other way and adding a ''exclude'' to the fileserver conf, but I like including things explicitly much better. So we would recommend that each module has a layout of MODULE_NAME/ # (e.g., autofs) MODULE_NAME.pp other.pp # additional files if wanted files/ # literal config files autofs.homes templates/ # templates auto.master.templ and all this should happen in a default directory somewhere, say ''/var/lib/puppet/config/modules'' on the puppetmaster (call that MODDIR for now), so that * $MODDIR is on puppet''s search path, i.e. you write ''import "autofs/autofs.pp"'' in your manifest (maybe the main .pp in a module should just be main.pp) * $MODDIR/*/files is the path for the ''modules'' module for the file server * $MODDIR/*/templates is searched for templates (small addition to how templatedir is used, since it now needs globbing, too) In addition, we should probably add a default ''private'' module to the fileserver, i.e. add [private] path /var/lib/puppet/config/private/%h allow * That gives people the chance to make /var/lib/puppet/config a checkout of their favorite SCM.> > [1] That''s one thing that makes rails so much easier to work with > > than, > > say J2EE: everything is defaulted to something reasonable, and you are > > gently forced to follow best practices by those defaults. > > I definitely agree with this, too. I''ve done my best to have > reasonable defaults everywhere in Puppet,Absolutely .. I am not saying you don''t.> PRM has touched on it some, but it has been > basically lying fallow, unfortunately (and I''m certainly muchly to > blame for that).Yeah, it certainly needs some love; if we settle on a recommended layout for these things, prm would become somewhat simpler.> Any defaults we choose here should be chosen towards the goal of > making redistribution easy. That is, our default design should > choose from among the intersection between those solutions that work > well for normal management and those that work well for redistribution.I think that''s a really good way to describe the goal of this exercise. David
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 26 January 2007 01:13, David Lutterkort wrote:> On Thu, 2007-01-25 at 15:14 -0600, Luke Kanies wrote: > > I think subdirectories are a good idea anyway, so that templates, > > manifests, and data files are all in separate directories. > > Yep, that would work, too; I was thinking of going the other way and > adding a ''exclude'' to the fileserver conf, but I like including things > explicitly much better. > > So we would recommend that each module has a layout of > MODULE_NAME/ # (e.g., autofs) > MODULE_NAME.pp > other.pp # additional files if wanted > files/ # literal config files > autofs.homes > templates/ # templates > auto.master.templ > > and all this should happen in a default directory somewhere, say > ''/var/lib/puppet/config/modules'' on the puppetmaster (call that MODDIR > for now), so that > * $MODDIR is on puppet''s search path, i.e. you write ''import > "autofs/autofs.pp"'' in your manifest (maybe the main .pp in a > module should just be main.pp) > * $MODDIR/*/files is the path for the ''modules'' module for the > file server > * $MODDIR/*/templates is searched for templates (small addition to > how templatedir is used, since it now needs globbing, too) > > In addition, we should probably add a default ''private'' module to the > fileserver, i.e. add > [private] > path /var/lib/puppet/config/private/%h > allow * > > That gives people the chance to make /var/lib/puppet/config a checkout > of their favorite SCM.ooooh, I like that stuff! Here some little bits and pieces that come to my mind: Big software systems like java use hierarchical namespaces to organise their packages. Other namespaces - like (Debian) packages or $PATH - are flat and time and again have problems with collisions. Therefore I would recommend having a semi-managed Namespace for MODULE_NAME with a prefix for "official only-one-implementation-per-functionality" (prm_?) and organisation prefixes like to have names like "at.edv-bus_DebianRecipies" for collecting all my content from http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/DebianRecipies without having troubles colliding with some other Debian-related manifest. If one wants to avoid collisions, the next thing to do is to scope all lookups by module. That would e.g. include puppet:// URLs in a way, that a module automatically exports MODULE_DIR/files as puppet://HOST/MODULE_DIR and some syntactic sugar to access this without huge paths in "source" parameters. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFujp+/Pp1N6Uzh0URAvqAAJoCil7/DdISaRQlVZsbPaPrfE3mIACcCc1p eLN2PdrCRtBibvDoqH6BoEY=NWOl -----END PGP SIGNATURE-----
On Jan 25, 2007, at 6:13 PM, David Lutterkort wrote:> > What I meant to say is that things should be structured in such a way > that ''releasing'' a config from test to prod should be no more than > checking out a specific tag from the SCM, one that has been tested > before, just as you would do when going from a release candidate to a > release in a software release (minus the complications of packagign > the > release)Ah; I completely agree with that. It''s all about the bit-flip.>> >> We could add globbing to the fileserver.conf file, so you could do >> something like ''allow /some/where/*/files''. > > I assume you mean ''path'' here, right ?Maybe. :P>> I think subdirectories are a good idea anyway, so that templates, >> manifests, >> and data files are all in separate directories. > > Yep, that would work, too; I was thinking of going the other way and > adding a ''exclude'' to the fileserver conf, but I like including things > explicitly much better. > > So we would recommend that each module has a layout of > MODULE_NAME/ # (e.g., autofs) > MODULE_NAME.pp > other.pp # additional files if wanted > files/ # literal config files > autofs.homes > templates/ # templates > auto.master.templI think I like the .pp files in a separate ''manifests'' directory, too. I know the manifests aren''t really executable code, but I just don''t like the thought of Puppet being able to accidentally load a file from the files/ subdirectory. That is, if you were (for some reason) distributing .pp files using Puppet, then it would be a simple mistake to load those files in files/ into the master.> and all this should happen in a default directory somewhere, say > ''/var/lib/puppet/config/modules'' on the puppetmaster (call that MODDIR > for now), so that > * $MODDIR is on puppet''s search path, i.e. you write ''import > "autofs/autofs.pp"'' in your manifest (maybe the main .pp in a > module should just be main.pp) > * $MODDIR/*/files is the path for the ''modules'' module for the > file server > * $MODDIR/*/templates is searched for templates (small > addition to > how templatedir is used, since it now needs globbing, too)I definitely like this. Is anyone willing to do a quick survey of how the different scripting languages handle mapping class names to files on disk? It''d be great if we could learn from their successes and failures here. Here''s what I can remember: Ruby: Libraries are loaded by file name, relative to the search path. E.g., ''require "puppet/transaction"'' iterates through the ruby search path ($:) looking for ''puppet/transaction.rb'', and loads the first one it finds. The semi-quirk here is that the class names don''t have to have anything to do with the file names, but at the least they''re always capitalized differently; in this case, the class is named Puppet::Transaction Perl: Libraries are loaded by class name, relative to the search path. E.g., ''require Perl::Doc'' iterates through the search path for a file named ''perl/doc.pl'' and loads that, with the assumption that that file contains the appropriately-named class. Python: (I barely remember this.) I don''t remember how the libraries are loaded, but I remember that the top-level files are named ''module/ __init__.py'', rather than having ''module.py'' and ''module/''. Does looking at them put us in a good position to understand how we want to map class names to file names? Now that Puppet (mostly) supports namespaces, I wouldn''t be too upset at going perl-style and assuming a relationship between class name and file name (e.g., you could do ''require os::debian'' and it would look for ''os/debian.pp''), although I also wouldn''t object to ruby''s version. Globbing becomes a bit weird, though; would ''require os/*'' search every searchpath directory for files in os/, or would it stop at the first os/ path it found and just load all of those? Seems like it should do the former, but I expect that would quite confuse some people (actually, I expect either answer would confuse some people).> In addition, we should probably add a default ''private'' module to the > fileserver, i.e. add > [private] > path /var/lib/puppet/config/private/%h > allow * > > That gives people the chance to make /var/lib/puppet/config a checkout > of their favorite SCM.I don''t understand the value of the %h here. I agree that we should probably have a default fileserver configuration, along with directions on how to set up the SCM so that it works with the default configuration, but I don''t see what role the %h plays.> Yeah, it certainly needs some love; if we settle on a recommended > layout > for these things, prm would become somewhat simpler.I concur.>> Any defaults we choose here should be chosen towards the goal of >> making redistribution easy. That is, our default design should >> choose from among the intersection between those solutions that work >> well for normal management and those that work well for >> redistribution. > > I think that''s a really good way to describe the goal of this > exercise.Great. -- There is no expedient to which a man will not go to avoid the labor of thinking. --Thomas A. Edison --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 26, 2007, at 11:29 AM, David Schmitt wrote:> > ooooh, I like that stuff! Here some little bits and pieces that > come to my > mind: > > Big software systems like java use hierarchical namespaces to > organise their > packages. Other namespaces - like (Debian) packages or $PATH - are > flat and > time and again have problems with collisions. Therefore I would > recommend > having a semi-managed Namespace for MODULE_NAME with a prefix for > "official > only-one-implementation-per-functionality" (prm_?) and organisation > prefixes > like to have names like "at.edv-bus_DebianRecipies" for collecting > all my > content from http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/ > DebianRecipies > without having troubles colliding with some other Debian-related > manifest.As mentioned in my other email on this thread, Puppet does have namespaces, they''re just undocumented and not quite bug-free yet. So, I agree that hierarchical namespaces should be used here, and the modules would map to namespaces. Companies could create their own mycompany:: namespace and put their recipes there. Once PRM got more heavily used, there would need to be a CPAN-like group responsible for arbitrating namespace choices.> If one wants to avoid collisions, the next thing to do is to scope > all lookups > by module. That would e.g. include puppet:// URLs in a way, that a > module > automatically exports MODULE_DIR/files as puppet://HOST/MODULE_DIR > and some > syntactic sugar to access this without huge paths in "source" > parameters.That''s a good idea. You wouldn''t want to have to individually enable each module, I expect; instead, the fileserver should follow the module search path, but looking for files instead of manifests (and, of course, templates should work similarly). I think there should be a separate ''modules/'' top-level path here (or something; I don''t care what the name is), so that, for instance, if you were looking for the file related to ''trac'', you''d ask for ''puppet://puppet/modules/trac/ config.ini''. Thus, ''modules'' (or whatever it''s called) would be a semi-magical module that knew how to search for modules correctly. -- If all the world''s a stage, I want to operate the trap door. -- Paul Beatty --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Fri, 2007-01-26 at 18:29 +0100, David Schmitt wrote:> Big software systems like java use hierarchical namespaces to organise their > packages. Other namespaces - like (Debian) packages or $PATH - are flat and > time and again have problems with collisions. Therefore I would recommend > having a semi-managed Namespace for MODULE_NAME with a prefix for "official > only-one-implementation-per-functionality" (prm_?) and organisation prefixes > like to have names like "at.edv-bus_DebianRecipies" for collecting all my > content from http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/DebianRecipies > without having troubles colliding with some other Debian-related manifest.Yeah some sort of namespace and conventions around it are needed for this; I would like to avoid the somewhat overblown Java design, having suffered through it. I think Ruby''s (or python''s, for that matter) relatively relaxed and flat namespacing works just as well in practice. The biggest danger for collision comes from site-specific stuff colliding with downloaded stuff IMHO, so a simple convention like ''the toplevel namespace and namespace "site" can not be used in distributed recipes'' should suffice.> If one wants to avoid collisions, the next thing to do is to scope all lookups > by module. That would e.g. include puppet:// URLs in a way, that a module > automatically exports MODULE_DIR/files as puppet://HOST/MODULE_DIR and some > syntactic sugar to access this without huge paths in "source" parameters.Good point, I kinda forgot that aspect in my directory layout proposal. I started a page on the Wiki [1] to track this. I''d love to get this figured out and start working on whatever changes are needed to make this happen. David [1] http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/MasterConfigLayout
On Fri, 2007-01-26 at 12:24 -0600, Luke Kanies wrote:> On Jan 25, 2007, at 6:13 PM, David Lutterkort wrote: > > So we would recommend that each module has a layout of > > MODULE_NAME/ # (e.g., autofs) > > MODULE_NAME.pp > > other.pp # additional files if wanted > > files/ # literal config files > > autofs.homes > > templates/ # templates > > auto.master.templ > > I think I like the .pp files in a separate ''manifests'' directory, > too. I know the manifests aren''t really executable code, but I just > don''t like the thought of Puppet being able to accidentally load a > file from the files/ subdirectory.Not sure if that was clear in my list above: the manifests should go into the same dir as the files/ directory, so given a properly configured fileserver, they won''t be accessible to clients.> That is, if you were (for some > reason) distributing .pp files using Puppet, then it would be a > simple mistake to load those files in files/ into the master.I don''t understand what you mean here. I am not principally opposed to putting them into a manifests/ dir; it''s just that we''d want people to say ''import "autofs/*.pp"'' without them having to mention manifests/, i.e. another place where we''d map a path into a special subdir.> Is anyone willing to do a quick survey of how the different scripting > languages handle mapping class names to files on disk? > ... > Does looking at them put us in a good position to understand how we > want to map class names to file names?The thing that makes this different from scripting language imports/path searching is that we want different parts if the system to look into different subdirs; in scripting languages the mapping of class/module name to file name doesn''t have that extra wrinkle. So far, we want * import "foo/bar.pp" to map to either "foo/bar.pp" or "foo/manifests/bar.pp" somewhere on the manifest search path * URL''s like ''puppet://server/lib/foo/bar.txt'' to map to ''foo/files/bar.txt'' in the ''lib'' module of the fileserver * template paths like ''foo/bar.erb'' to map to ''foo/templates/bar.erb'' somewhere on the templatepath (I hope I covered all of them) I don''t see a good alternative to all this mapping, though it might be confusing to users initially.> Now that Puppet (mostly) supports namespaces,Does that support work for variables, too ? Can I have a variable os::fedora::frob and use it whereever I can use a var w/o namespace qualification ? I also think there needs to be some sort of ''module'' or ''namespace'' construct in the language to make this more usable (similar to ruby''s module keyword)> I wouldn''t be too upset at going perl-style and > assuming a relationship between class name and file name (e.g., you > could do ''require os::debian'' and it would look for ''os/debian.pp''),you mean ''os/manifests/debian.pp'' ;)> although I also wouldn''t object to ruby''s version.Yeah, either version would work for me.> Globbing becomes a bit weird, though; would ''require os/*'' search every searchpath > directory for files in os/, or would it stop at the first os/ path it > found and just load all of those?Maybe import statements shouldn''t glob at all, and people should have to write ''container'' files like you do in ruby. Though the basic question is: when you import, do you import a file or a module/namespace ? Most languages make you think you import a module and make sure that maps very straightforward to a path in the filesystem, though that fiction falls apart in a lot of corner cases.> > In addition, we should probably add a default ''private'' module to the > > fileserver, i.e. add > > [private] > > path /var/lib/puppet/config/private/%h > > allow * > > > > That gives people the chance to make /var/lib/puppet/config a checkout > > of their favorite SCM. > > I don''t understand the value of the %h here. I agree that we should > probably have a default fileserver configuration, along with > directions on how to set up the SCM so that it works with the default > configuration, but I don''t see what role the %h plays.The point was that I wanted people to have a default host-private module for ssh keys and the like to save them from inventing their own schemes to distribute them. David
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 26 January 2007 20:38, David Lutterkort wrote:> The thing that makes this different from scripting language imports/path > searching is that we want different parts if the system to look into > different subdirs; in scripting languages the mapping of class/module > name to file name doesn''t have that extra wrinkle. So far, we want > * import "foo/bar.pp" to map to either "foo/bar.pp" or > "foo/manifests/bar.pp" somewhere on the manifest search path > * URL''s like ''puppet://server/lib/foo/bar.txt'' to map to > ''foo/files/bar.txt'' in the ''lib'' module of the fileserver > * template paths like ''foo/bar.erb'' to map to > ''foo/templates/bar.erb'' somewhere on the templatepath > (I hope I covered all of them) I don''t see a good alternative to all > this mapping, though it might be confusing to users initially.It is easier for me to not think of the proposed MODULES_DIR as part of any explicit searchpath. * import "foo/bar.pp" maps to the "bar.pp" of the "foo" module, or to "foo/bar.pp" somewhere on the manifest search path * "puppet://server/foo/..." map into the "files/" hierarchy of the module "foo" * template paths like ''foo/bar.erb'' fetches the "bar.erb" template from the "foo" module or the local "foo/bar.erb" from somewhere on the templatepath Especially if you take that a step further and replace the many different search paths with a module search path. With a module search path of "/etc/puppet:/var/lib/puppet/prm:/var/local/puppet_checkout" the layouts below these directories would be very uniform again. Unqualified references are search in all modules and generate a warning if not found in some kind of default module. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFumjw/Pp1N6Uzh0URAsbsAJ4pFAjisgKdZUJC/diJm53RoA2+XACdFIz/ 2ET567rre2vPnRYL1KeTlfA=sB38 -----END PGP SIGNATURE-----
On Jan 26, 2007, at 1:38 PM, David Lutterkort wrote:> On Fri, 2007-01-26 at 12:24 -0600, Luke Kanies wrote: >> On Jan 25, 2007, at 6:13 PM, David Lutterkort wrote: >>> So we would recommend that each module has a layout of >>> MODULE_NAME/ # (e.g., autofs) >>> MODULE_NAME.pp >>> other.pp # additional files if wanted >>> files/ # literal config files >>> autofs.homes >>> templates/ # templates >>> auto.master.templ >> >> I think I like the .pp files in a separate ''manifests'' directory, >> too. I know the manifests aren''t really executable code, but I just >> don''t like the thought of Puppet being able to accidentally load a >> file from the files/ subdirectory. > > Not sure if that was clear in my list above: the manifests should go > into the same dir as the files/ directory, so given a properly > configured fileserver, they won''t be accessible to clients.What if there''s a .pp file in the files/ directory? E.g.: module/ module.pp files/ myfile.pp Then, when myfile.pp is only meant for distribution, it could instead be sourced into the parser. E.g., someone could do ''import "module/ files/myfile", when that wasn''t the intention of the module developer. I don''t like it. There are three types of files, and I think they should each be in their own directories. The one exception I''d allow is that it''d be okay for ''module_name/module_name.pp'' to exist, but I think all other manifests should be in ''module_name/manifests''. That way there''s no potential for confusion.>> That is, if you were (for some >> reason) distributing .pp files using Puppet, then it would be a >> simple mistake to load those files in files/ into the master. > > I don''t understand what you mean here. I am not principally opposed to > putting them into a manifests/ dir; it''s just that we''d want > people to > say ''import "autofs/*.pp"'' without them having to mention manifests/, > i.e. another place where we''d map a path into a special subdir.Right, but we''re already doing this automatic mapping for two other directories (templates and files); why not do it for all three? In fact, it''d be *easier* to do the mapping for all of them, rather than just two. I''m already imagining the code that''s necessary...> The thing that makes this different from scripting language imports/ > path > searching is that we want different parts if the system to look into > different subdirs; in scripting languages the mapping of class/module > name to file name doesn''t have that extra wrinkle. So far, we want > * import "foo/bar.pp" to map to either "foo/bar.pp" or > "foo/manifests/bar.pp" somewhere on the manifest search path > * URL''s like ''puppet://server/lib/foo/bar.txt'' to map to > ''foo/files/bar.txt'' in the ''lib'' module of the fileserver > * template paths like ''foo/bar.erb'' to map to > ''foo/templates/bar.erb'' somewhere on the templatepath > (I hope I covered all of them) I don''t see a good alternative to all > this mapping, though it might be confusing to users initially.That''s not all that big of a difference; one small level of redirection. That''s not really the point I wanted to make, though. I was mainly interested in talking about how class names (e.g., http::server::apache) map to file names, especially if we look at each namespace as a potential module. With this (admittedly rather bad) class name, installed at /usr/lib/puppet, I would expect the following files to be in place: http/ manifests/ # empty, probably server/ manifests/ # empty, probably apache/ apache.pp manifests/ other.pp With this, you could import ''http/server/apache'' or ''http/server/ apache/other''. However, if someone wanted to create a module named ''http/server/apache/other'', then that would conflict with the module you created. How should we resolve that? This is at least one reason *not* to put the pp files into a separate manifests directory -- you''d get direct file conflicts when they existed, which would probably be better than having files just be shadowed by modules or vice versa. Should we still require separate import and include statements? I could pretty easily enhance ''include'' to know how to find and import files, but this would require (heh heh) that classes always match file names. That is, if we could assume that ''http/server/apache'' always mapped to ''http/server/apache.pp'', then people could just include the class directly and Puppet would know how to translate that to file paths. Incidentally, when writing the DSL demo stuff, in which both ''class'' and ''include'' were already keywords in Ruby, I used ''aspect'' and ''acquire'', respectively. I kind of like these terms better, and I''ve been thinking about starting a migration to them. Any comments? I''m not in a big hurry, and I''m not really committed or whatever, but...>> Now that Puppet (mostly) supports namespaces, > > Does that support work for variables, too ? Can I have a variable > os::fedora::frob and use it whereever I can use a var w/o namespace > qualification ? I also think there needs to be some sort of > ''module'' or > ''namespace'' construct in the language to make this more usable > (similar > to ruby''s module keyword)Nope, but this would be a good idea. Actually, it wouldn''t even be that difficult. Comments?>> I wouldn''t be too upset at going perl-style and >> assuming a relationship between class name and file name (e.g., you >> could do ''require os::debian'' and it would look for ''os/debian.pp''), > > you mean ''os/manifests/debian.pp'' ;)Yep. :P>> although I also wouldn''t object to ruby''s version. > > Yeah, either version would work for me. > >> Globbing becomes a bit weird, though; would ''require os/*'' search >> every searchpath >> directory for files in os/, or would it stop at the first os/ path it >> found and just load all of those? > > Maybe import statements shouldn''t glob at all, and people should > have to > write ''container'' files like you do in ruby. Though the basic question > is: when you import, do you import a file or a module/namespace ? Most > languages make you think you import a module and make sure that maps > very straightforward to a path in the filesystem, though that fiction > falls apart in a lot of corner cases.If we do that, then you get to take the blame. People *really* like globbing.> The point was that I wanted people to have a default host-private > module > for ssh keys and the like to save them from inventing their own > schemes > to distribute them.Ah. That''s a good idea. -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- (attributed to) Brian W. Kernighan (unconfirmed) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 26, 2007, at 1:05 PM, David Lutterkort wrote:> Yeah some sort of namespace and conventions around it are needed for > this; I would like to avoid the somewhat overblown Java design, having > suffered through it. I think Ruby''s (or python''s, for that matter) > relatively relaxed and flat namespacing works just as well in > practice.I don''t know Java''s, but I''m plenty happy with pretty flat namespaces.> The biggest danger for collision comes from site-specific stuff > colliding with downloaded stuff IMHO, so a simple convention like ''the > toplevel namespace and namespace "site" can not be used in distributed > recipes'' should suffice.Sounds good, although we''ll also have to follow the standard practice of having a ''lib'' and ''site_lib'' directory, e.g., /usr/lib/puppet and /var/lib/site_puppet or whatever the LSB standard is.>> If one wants to avoid collisions, the next thing to do is to scope >> all lookups >> by module. That would e.g. include puppet:// URLs in a way, that a >> module >> automatically exports MODULE_DIR/files as puppet://HOST/MODULE_DIR >> and some >> syntactic sugar to access this without huge paths in "source" >> parameters. > > Good point, I kinda forgot that aspect in my directory layout > proposal. > I started a page on the Wiki [1] to track this. I''d love to get this > figured out and start working on whatever changes are needed to make > this happen.Yeah, we definitely need this auto-export. This would basically just require some kind of autolookup ability within the fileserver, which wouldn''t be that difficult. As to how to do this... Ticket #287[1] has most of the work needed for search paths (although a good bit of that patch is obsolete since run-time imports will never be supported now), but I expect I''d implement this by creating a module that knew how to look through the search path using a method to determine which subdirectory it should look in (files, templates, or manifests). Then the module just needs to be plugged into the different locations. I think it''s reasonable to get this done in a few hours of work, but I won''t be able to do it in time for the kermit release, I expect. 1 - https://reductivelabs.com/trac/puppet/ticket/271> [1] http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/ > MasterConfigLayoutThanks for writing this up. -- The Internet, of course, is more than just a place to find pictures of people having sex with dogs. -- Time Magazine, 3 July 1995 --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 26, 2007, at 2:47 PM, David Schmitt wrote:> It is easier for me to not think of the proposed MODULES_DIR as > part of any > explicit searchpath. > > * import "foo/bar.pp" maps to the "bar.pp" of the "foo" module, or > to "foo/bar.pp" somewhere on the manifest search path > * "puppet://server/foo/..." map into the "files/" hierarchy of the > module "foo" > * template paths like ''foo/bar.erb'' fetches the "bar.erb" > template from > the "foo" module or the local "foo/bar.erb" from somewhere on the > templatepath > > Especially if you take that a step further and replace the many > different > search paths with a module search path. With a module search path > of "/etc/puppet:/var/lib/puppet/prm:/var/local/puppet_checkout" the > layouts > below these directories would be very uniform again. Unqualified > references > are search in all modules and generate a warning if not found in > some kind of > default module.This is definitely how I would implement it: One search path that sufficed for files, manifests, and templates. I''m still not sure about the filesystem layout, though, since the current design has plenty of room for overlaps. At the least, we need to have a spec for how to decide between, say, ''os/files/debian/ issue'' and ''os/debian/files/issue''. I guess the only way to do it is to specify which one wins in this kind of case. It''s kind of weird, because the extra level of indirection (i.e., the type-based subdirectory, which will be used implicitly) is what leaves open this opportunity for shadowing. -- Learning is not attained by chance, it must be sought for with ardor and attended to with diligence. -- Abigail Adams --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday 27 January 2007 21:53, Luke Kanies wrote:> This is definitely how I would implement it: One search path that > sufficed for files, manifests, and templates. > > I''m still not sure about the filesystem layout, though, since the > current design has plenty of room for overlaps. At the least, we > need to have a spec for how to decide between, say, ''os/files/debian/ > issue'' and ''os/debian/files/issue''. I guess the only way to do it is > to specify which one wins in this kind of case. It''s kind of weird, > because the extra level of indirection (i.e., the type-based > subdirectory, which will be used implicitly) is what leaves open this > opportunity for shadowing.A point for a flat namespace. If the module can always only be the part before the first slash (even if it contains the domain of the author) this cannot happen. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFvKHO/Pp1N6Uzh0URAnP9AJ4+1t18AVuRs7zNt1kx/VpG///yLQCfebSF Dh2eVkpRE2q2BLOrvPIqj04=Y5rS -----END PGP SIGNATURE-----
On Jan 28, 2007, at 7:14 AM, David Schmitt wrote:> > On Saturday 27 January 2007 21:53, Luke Kanies wrote: >> This is definitely how I would implement it: One search path that >> sufficed for files, manifests, and templates. >> >> I''m still not sure about the filesystem layout, though, since the >> current design has plenty of room for overlaps. At the least, we >> need to have a spec for how to decide between, say, ''os/files/debian/ >> issue'' and ''os/debian/files/issue''. I guess the only way to do it is >> to specify which one wins in this kind of case. It''s kind of weird, >> because the extra level of indirection (i.e., the type-based >> subdirectory, which will be used implicitly) is what leaves open this >> opportunity for shadowing. > > A point for a flat namespace. If the module can always only be the > part before > the first slash (even if it contains the domain of the author) this > cannot > happen.You would want to restrict the namespace that far, huh? I think that''s a bit dangerous. I''m all for relatively flat namespaces, but I''m not sure it''ll be possible to restrict it that far. I think two levels will end up being pretty common, with three not uncommon and one the most common. Unless... Can you can come up with a standard that will work for most cases with only one level of heirarchy? -- Should I say "I believe in physics", or "I know that physics is true"? -- Ludwig Wittgenstein, On Certainty, 602. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Sat, 2007-01-27 at 14:53 -0600, Luke Kanies wrote:> This is definitely how I would implement it: One search path that > sufficed for files, manifests, and templates.What happens to the existing paths ? We could implicitly add a new ''module_path'' to the search paths for manifests and templates, the fileserver config would need to be configrued just right manually for this to work.> At the least, we > need to have a spec for how to decide between, say, ''os/files/debian/ > issue'' and ''os/debian/files/issue''That would require having separate module and file paths when requesting files (and templates), i.e. you''d have to be able to tell which part of a puppet://.. URL is the stuff before and which is the stuff after the /files/ dir .. ugh. For the fileserver though, this example means that either one fileserver mdoule is backed by more than one directory or that the modules are nested in the filesystem ... we could require that modules have some sort of metadata file (with a prescribed haiku) in their directory hierarchy and make nesting modules strictly verboten. David
On Sat, 2007-01-27 at 14:31 -0600, Luke Kanies wrote:> What if there''s a .pp file in the files/ directory? E.g.: > > module/ > module.pp > files/ > myfile.pp > > Then, when myfile.pp is only meant for distribution, it could instead > be sourced into the parser. E.g., someone could do ''import "module/ > files/myfile", when that wasn''t the intention of the module developer.I don''t think that that would be a problem - it''s not a security issue, and if that does something useful that the original module writer didn''t intend, even better.> I was mainly interested in talking about how class names (e.g., > http::server::apache) map to file names, especially if we look at > each namespace as a potential module. With this (admittedly rather > bad) class name, installed at /usr/lib/puppet, I would expect the > following files to be in place: > > http/ > manifests/ # empty, probably > server/ > manifests/ # empty, probably > apache/ > apache.pp > manifests/ > other.pp > > With this, you could import ''http/server/apache'' or ''http/server/ > apache/other''. However, if someone wanted to create a module named > ''http/server/apache/other'', then that would conflict with the module > you created. How should we resolve that?Eeeew ... I hadn''t even thought about this kind of nesting; I don''t believe that that is something users will find easy to understand, and it becomes impossible to tell where one module ends and another begins. I was thinking of modules more in the way ruby does it, where each package/module owns a toplevel namespace - while nothing keeps me from putting files into the racc/ directory, convention dictates that I not do that. I am also wary of trying to use the namespace to categorize things - should it be http::server::apache or server::http::apache ? What if something is specific for apache on debian ? This categorization becomes ugly pretty quickly, and non-intuitive. (And let''s face it, there''s not an insane amount of modules out there right now) Modules should be leaf-nodes in a directory tree. There''s another good reason to make it so: for prm, each module should be a separate checkout from a SCM, which is much easier with leaf-node modules.> This is at least one reason *not* to put the pp files into a separate > manifests directory -- you''d get direct file conflicts when they > existed, which would probably be better than having files just be > shadowed by modules or vice versa.An alternative to the subdirectory layout and lots of remapping is to make a module just one directory, structured however, and make sure that the fileserver doesn''t serve certain files, either by file extension (e.g. suppress **/*.pp and **/*.templ) or through an explicit manifest/specfile in the module dir. The fileserver would then read the specfile to decide which files to serve (or not to serve) but wouldn''t do any magic remapping. I like the latter the best, since it makes it very explicit what is what in the module directory, and doesn''t rely on path remappings that will seem like black magic to the uninitiated. The specfile could be as simple as a listing of the manifests and the templates in the module; sooner or later we''ll need some sort of metadata file anyway. The prm prototype used it to (a) keep track of the original, proper upstream for a recipe (as a URL to an hg repo) and (b) list dependencies of the module (also as hg repo URL''s) Other uses will certainly be found, e.g. to include add''l docs etc. The prototype used XML for that format, but if we think that those files will be edited by humans a lot, we could switch to .ini-style files or similar.> Should we still require separate import and include statements? I > could pretty easily enhance ''include'' to know how to find and import > files, but this would require (heh heh) that classes always match > file names. That is, if we could assume that ''http/server/apache'' > always mapped to ''http/server/apache.pp'', then people could just > include the class directly and Puppet would know how to translate > that to file paths.But that would also require that there''s only one class per file, which would be kidna silly for very simple classes, or subclasses that only tweak a few things.> Incidentally, when writing the DSL demo stuff, in which both ''class'' > and ''include'' were already keywords in Ruby, I used ''aspect'' and > ''acquire'', respectively. I kind of like these terms better, and I''ve > been thinking about starting a migration to them. Any comments? I''m > not in a big hurry, and I''m not really committed or whatever, but...I don''t think that there''s a big need to move away from import/include just now - they seem pretty easy to understand.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday 29 January 2007 17:25, Luke Kanies wrote:> You would want to restrict the namespace that far, huh? I think > that''s a bit dangerous. I''m all for relatively flat namespaces, but > I''m not sure it''ll be possible to restrict it that far. I think two > levels will end up being pretty common, with three not uncommon and > one the most common. > > Unless... Can you can come up with a standard that will work for most > cases with only one level of heirarchy?Hmm, the module separator character doesn''t have to be a slash, does it? e.g using a dot: "edv-bus.debian_stuff" and "edv-bus.internal"; then it is always obvious what the module is and what part is the path. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFvxOL/Pp1N6Uzh0URAmIoAKCSTc5cokhfYdWkwC4ez60JCrmUkwCffb5n lhmfuGLyd5STRsgpF/I8+3o=1lYM -----END PGP SIGNATURE-----
On Jan 30, 2007, at 3:44 AM, David Schmitt wrote:> > Hmm, the module separator character doesn''t have to be a slash, > does it? > > e.g using a dot: "edv-bus.debian_stuff" and "edv-bus.internal"; > then it is > always obvious what the module is and what part is the path.That''s an interesting idea; no one else seems to be doing it, but that should work very well. I like the idea of allowing namespaces while still cleanly separating modules from each other. I''m not too keen on the ''.'', though, because you the files already have dots in them. Why not just use the double colon? This way, double colons would be used in both importing and inclusion: import "os::debian/apache" include os::debian::apache That might be even more confusing, but I''m not sure this is much better: import "os.debian/apache" include os::debian::apache" Yes, David Lutterkort is clearly right about the perils of hierarchical organization, especially in the above example, but it''s still pretty useful sometimes. -- Brand''s Asymmetry: The past can only be known, not changed. The future can only be changed, not known. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 02 February 2007 23:33, Luke Kanies wrote:> I''m not too keen on the ''.'', though, because you the files already > have dots in them. Why not just use the double colon? This way, > double colons would be used in both importing and inclusion: > > import "os::debian/apache" > include os::debian::apache > > That might be even more confusing, but I''m not sure this is much better: > > import "os.debian/apache" > include os::debian::apache"You don''t have to map it at all: import "os.debian/apache" include "os.debian::apache" # fully qualified ... include "apache" # ... or using a default search path Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFw8g5/Pp1N6Uzh0URAsELAJwKgjMs7ziJtqs1w/X/ADAUTsldFgCffdZL /+iGz6lMr+hPa7TTf9f2FCs=kg9G -----END PGP SIGNATURE-----
On Jan 29, 2007, at 12:20 PM, David Lutterkort wrote:> On Sat, 2007-01-27 at 14:53 -0600, Luke Kanies wrote: >> This is definitely how I would implement it: One search path that >> sufficed for files, manifests, and templates. > > What happens to the existing paths ? We could implicitly add a new > ''module_path'' to the search paths for manifests and templates, the > fileserver config would need to be configrued just right manually for > this to work.There''s going to need to be some magic anyway; the current choices will be non-magical items in a list, and the new stuff will be pseudo- magical. I don''t think it will be any problem providing backwards compatibility.> That would require having separate module and file paths when > requesting > files (and templates), i.e. you''d have to be able to tell which > part of > a puppet://.. URL is the stuff before and which is the stuff after > the /files/ dir .. ugh. > > For the fileserver though, this example means that either one > fileserver > mdoule is backed by more than one directory or that the modules are > nested in the filesystem ... we could require that modules have some > sort of metadata file (with a prescribed haiku) in their directory > hierarchy and make nesting modules strictly verboten.These modules will be pseudo-magical, in that installing a module will automatically install it for the fileserver; at least, that''s my plan. Basically, people will ask for ''/mymodule/file'' and the fileserver will be smart enough to search its module path for ''mymodule'' and see if ''files/file'' exists in that module. Seems pretty simple. -- The first time I see a jogger smiling, I''ll consider it. --Joan Rivers --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Feb 2, 2007, at 5:24 PM, David Schmitt wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Friday 02 February 2007 23:33, Luke Kanies wrote: >> I''m not too keen on the ''.'', though, because you the files already >> have dots in them. Why not just use the double colon? This way, >> double colons would be used in both importing and inclusion: >> >> import "os::debian/apache" >> include os::debian::apache >> >> That might be even more confusing, but I''m not sure this is much >> better: >> >> import "os.debian/apache" >> include os::debian::apache" > > You don''t have to map it at all: > > import "os.debian/apache" > include "os.debian::apache" # fully qualified ... > include "apache" # ... or using a default search pathHmm, that won''t be easy to parse, and it means having an extra dimension in class definitions, which I expect will be, um, weird. But then, I suppose it would be necessary, since the modules would have that extra dimension. Would someone be willing to write up the current state of these recommendations on the wiki? -- Some people are afraid of heights. I''m afraid of widths. -- Stephen Wright --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 29, 2007, at 1:34 PM, David Lutterkort wrote:> On Sat, 2007-01-27 at 14:31 -0600, Luke Kanies wrote: >> What if there''s a .pp file in the files/ directory? E.g.: >> >> module/ >> module.pp >> files/ >> myfile.pp >> >> Then, when myfile.pp is only meant for distribution, it could instead >> be sourced into the parser. E.g., someone could do ''import "module/ >> files/myfile", when that wasn''t the intention of the module >> developer. > > I don''t think that that would be a problem - it''s not a security > issue, > and if that does something useful that the original module writer > didn''t > intend, even better.I don''t understand why you''d single out the manifests not to get their own directory. Why should everything else get it, but not the manifests?> I was thinking of modules more in the way ruby does it, where each > package/module owns a toplevel namespace - while nothing keeps me from > putting files into the racc/ directory, convention dictates that I not > do that.Huh, I guess I hadn''t realized that Ruby does this. I suppose I''m not all that opposed, although it will clearly get very horizontal in a hurry.> I am also wary of trying to use the namespace to categorize things - > should it be http::server::apache or server::http::apache ? What if > something is specific for apache on debian ? This categorization > becomes > ugly pretty quickly, and non-intuitive. (And let''s face it, there''s > not > an insane amount of modules out there right now)This is a problem with any organization, but it''s not always the right answer to just throw away all nesting.> Modules should be leaf-nodes in a directory tree. There''s another good > reason to make it so: for prm, each module should be a separate > checkout > from a SCM, which is much easier with leaf-node modules.Yeah, I definitely agree with that.>> This is at least one reason *not* to put the pp files into a separate >> manifests directory -- you''d get direct file conflicts when they >> existed, which would probably be better than having files just be >> shadowed by modules or vice versa. > > An alternative to the subdirectory layout and lots of remapping is to > make a module just one directory, structured however, and make sure > that > the fileserver doesn''t serve certain files, either by file extension > (e.g. suppress **/*.pp and **/*.templ) or through an explicit > manifest/specfile in the module dir. The fileserver would then read > the > specfile to decide which files to serve (or not to serve) but wouldn''t > do any magic remapping. > > I like the latter the best, since it makes it very explicit what is > what > in the module directory, and doesn''t rely on path remappings that will > seem like black magic to the uninitiated. The specfile could be as > simple as a listing of the manifests and the templates in the module; > sooner or later we''ll need some sort of metadata file anyway. The prm > prototype used it to (a) keep track of the original, proper > upstream for > a recipe (as a URL to an hg repo) and (b) list dependencies of the > module (also as hg repo URL''s) Other uses will certainly be found, > e.g. > to include add''l docs etc.I think the spec file will be more opaque than one-directory-per- service, frankly, and it requires extra work to maintain. Add a file to the manifests directory, it''s available; no extra work necessary. Spec files are good ideas for things like the original upstream, but I don''t like the idea of having to modify them every time I add a file; next thing you know people are autogenerating their spec files, and having to generate files is a bug in my world.> The prototype used XML for that format, but if we think that those > files > will be edited by humans a lot, we could switch to .ini-style files or > similar.INI good, XML bad.>> Should we still require separate import and include statements? I >> could pretty easily enhance ''include'' to know how to find and import >> files, but this would require (heh heh) that classes always match >> file names. That is, if we could assume that ''http/server/apache'' >> always mapped to ''http/server/apache.pp'', then people could just >> include the class directly and Puppet would know how to translate >> that to file paths. > > But that would also require that there''s only one class per file, > which > would be kidna silly for very simple classes, or subclasses that only > tweak a few things.True; okay, nix that one then.>> Incidentally, when writing the DSL demo stuff, in which both ''class'' >> and ''include'' were already keywords in Ruby, I used ''aspect'' and >> ''acquire'', respectively. I kind of like these terms better, and I''ve >> been thinking about starting a migration to them. Any comments? I''m >> not in a big hurry, and I''m not really committed or whatever, but... > > I don''t think that there''s a big need to move away from import/include > just now - they seem pretty easy to understand.Heh, that has not been my experience. It''s a simple change, though, so we could make it whenever. I just think it''s a little more intuitive so say "host A is in class B and thus acquires aspect C" than "host A is in class B which has a bunch of resources, or maybe not". In other words, classes will often but not always have aspects, and right now it''s not quite possible to model things that way. -- I don''t want the world, I just want your half. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday 05 February 2007 20:08, Luke Kanies wrote:> Would someone be willing to write up the current state of these > recommendations on the wiki?I''ve written a (probably biased) summary on http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/ModuleOrganisation David, Bradly: I would appreciate corrections and feedback if I have missed something or got it wrong. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFx80Z/Pp1N6Uzh0URAtiPAJwPwZd8vUykOGQoNJIa5uSG+tI+jACeIGsf nJlqgxHzJ2Ecdskfespwwbw=N38s -----END PGP SIGNATURE-----
On Feb 5, 2007, at 6:34 PM, David Schmitt wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Monday 05 February 2007 20:08, Luke Kanies wrote: >> Would someone be willing to write up the current state of these >> recommendations on the wiki? > > I''ve written a (probably biased) summary on > > http://reductivelabs.com/cgi-bin/puppet.cgi/wiki/ModuleOrganisation > > David, Bradly: I would appreciate corrections and feedback if I > have missed > something or got it wrong.I''ve modified it a bit to try to cover some of what I see as the complex and important bits. -- Criminal: A person with predatory instincts who has not sufficient capital to form a corporation. --Howard Scott --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com