Hi Guys, At work we need a better way of running our machines. We''ve got 14 machines around the world that we look after (mostly web servers but a few run mail and dns as you''d expect) and we currently maintain them individually. They all log to a central syslog server over an openvpn interface, but aside from that they tend to be in different locations. Could puppet be any use to us in such a crazy environment, or is it unpractical because of all the network dependencies? Slightly less ambitious, and something we need to solve quite quickly, we have a rack in a datacentre where 4 machines are currently hosted, and we''ll be adding quite a few more machines over time as the service they host scales. They''re all on the same subnet and sit behind a firewall. I was thinking about putting puppetmaster on the firewall and running everything for these machines from there. The hosts all run FreeBSD 6.1-RELEASE and run quite a complicated set up of apache, tomcat, java, mysql, modpython, etc. So I guess my question is this: Is puppet ready for this kind of deployment, and will i run into any issues because I''m running on BSD? Also, more generally, noone wants to run anything that noone else is running, so I have to ask, are many people here using puppet in production yet? If not anyone got any ideas when they will, and what kind of issues are you running into day to day which might help me decide if it''s the way to go? I should say that I''m a sysadmin, not a programmer. I''ve written simple scripts that automate a lot of my life (webstats, backups, dns tests etc) but I''m really not very confident designing and writing programs that are important and complicated. I really like ruby, and see puppet as a good way to expose myself to it, but I''m worried that day to day I might spend ages battling with puppet problems. I guess what I''m asking is, what kind of impact is puppet going to make on my life? Will I be doing most of my work in puppet now? What does the day to day workflow look like? At present it goes something like this: install machine - unattended netinstall install packages - manually install the ports configure packages - copying an existing /usr/local/etc does most of that test that it worked fix it if it breaks - log into the host, fiddle with stuff till it works. My other confusion is how I would lay out my svn repository. I currently have something that looks like this: |-- hermes | |-- boot | | `-- loader.conf | |-- etc | | |-- cvsupfile | | |-- hosts | | |-- make.conf | | |-- newsyslog.conf | | |-- ntp.conf | | |-- pf.conf | | |-- rc.conf | | |-- resolv.conf | | |-- syslog.conf | | `-- zshrc | `-- usr | |-- local | | `-- etc | | |-- denyhosts.conf | | |-- dovecot.conf each important file is stored relative to where it would be on the machine in question. i''ve tried unsuccessfully to use svn "in place" by cp -R''ing my checked out tree into / which I can then maintain in the normal way, and check in my changes when I''m done, but that very quickly gets messy and confuses my head. i think ideally this is how i''d like to work: something breaks i log into the broken machine and fiddle with it make a change and get it working again commit that change into my local repository that change gets checked out by the broken machine, overwriting my earlier change and everything is back to normal (and logged in svn). will puppet help me do that? and will i need to change over to it overnight, having tested it on a test infrastructure first, or will i be able to move over to it gradually? i''ve read a lot of the docs on it, but I guess what i''m missing is the big picture, and seeing if it''s going to improve my life. this is all a bit rabid, so sorry about that. please ask me to clarify anything that isn''t clear - i''m just trying to work out what the options are at this stage. thanks for any help guys, Kep
On Sep 19, 2006, at 7:46 AM, Kep Woof wrote:> Hi Guys, > <snip> > Also, more generally, noone wants to run anything that noone else is > running, so I have to ask, are many people here using puppet in > production yet? If not anyone got any ideas when they will, and what > kind of issues are you running into day to day which might help me > decide if it''s the way to go?Joyent uses puppet to manage all our production Solaris boxen, most notably (simply because it was one of my first pure puppet deployments) a deployment of our Connector product, which entails 5 zones on 4 separate nodes talking to each other via public and 2 private networks all deployed by puppet w/o hands on. If you get into the mindset that if you do something in puppet once, you never have to do it again - it makes the initial time investment to developing automation for your environment a no-brainer IMO. All automation products will incur this extra effort, not just puppet, but puppet is the most flexible and augmentable system I''ve come across. It''s a difficult methodology to break in-to and habitualize since we all tend to do the hands-on fix, fix again, fix again (...) monkey dance ourselves when something like puppet can be taught to dance whilst we turn the crank. :-) For me at least, it just makes sense to put the extra effort in to get things automated when in the long term it''s going to allow me to do the research and development I love doing, rather than daily sysadmin gruntwork. Regards, -- Ryan Schwartz Joyent, Inc. TextDrive, Inc.
Kep Woof wrote:> Hi Guys, > > At work we need a better way of running our machines. We''ve got 14 > machines around the world that we look after (mostly web servers but a > few run mail and dns as you''d expect) and we currently maintain them > individually. They all log to a central syslog server over an openvpn > interface, but aside from that they tend to be in different locations. > Could puppet be any use to us in such a crazy environment, or is it > unpractical because of all the network dependencies?It should work fine for this kind of environment.> Slightly less ambitious, and something we need to solve quite quickly, > we have a rack in a datacentre where 4 machines are currently hosted, > and we''ll be adding quite a few more machines over time as the service > they host scales. They''re all on the same subnet and sit behind a > firewall. I was thinking about putting puppetmaster on the firewall > and running everything for these machines from there. The hosts all > run FreeBSD 6.1-RELEASE and run quite a complicated set up of apache, > tomcat, java, mysql, modpython, etc. So I guess my question is this: > Is puppet ready for this kind of deployment, and will i run into any > issues because I''m running on BSD?I seem to have *much* more trouble getting FreeBSD to behave as I want than I have with other operating systems, so I''m not as confident in saying that you will not have trouble. I think you will generally get good behaviour, but there are others who are actually using Puppet on FreeBSD who can maybe give better feedback. I will say that FreeBSD''s service system is not very functional, in that it does not support stop and restart functions as far as I can tell, and I''ve had trouble automating package maintenance (although apparently some of the code is written in Ruby and one of my clients has said they''ll be writing native Ruby support for managing FreeBSD packages).> Also, more generally, noone wants to run anything that noone else is > running, so I have to ask, are many people here using puppet in > production yet? If not anyone got any ideas when they will, and what > kind of issues are you running into day to day which might help me > decide if it''s the way to go?Heh, yeah, I''m interested in this too. :) Truly, though, it''s always a bit difficult to figure out who''s running it where. I have a few clients running it in production, one of which is using FreeBSD, but I know there are quite a few others who are using it, and at least one of those is a heavy FreeBSD user (at least, I assume so, since he''s the one that submits all the FreeBSD bug reports).> I should say that I''m a sysadmin, not a programmer. I''ve written > simple scripts that automate a lot of my life (webstats, backups, dns > tests etc) but I''m really not very confident designing and writing > programs that are important and complicated. I really like ruby, and > see puppet as a good way to expose myself to it, but I''m worried that > day to day I might spend ages battling with puppet problems.Well, the goal is that you only have to fight with Puppet instead of all of your computers, but it should certainly be a less time-consuming fight. You should not have to know ruby to use Puppet, but it will never hurt to know it.> I guess what I''m asking is, what kind of impact is puppet going to > make on my life? Will I be doing most of my work in puppet now? What > does the day to day workflow look like? At present it goes something > like this:Puppet is definitely a SLAM tool -- it''s meant for entirely ssh-less administration.> install machine - unattended netinstall > install packages - manually install the ports > configure packages - copying an existing /usr/local/etc does most of that > test that it worked > fix it if it breaks - log into the host, fiddle with stuff till it works.Puppet is great for doing steps 2 and 3, and it will allow your step 3 to be much more intelligent -- you can manage the config files according to local needs and maintain them going forward, rather than starting with a copy and managing manually. E.g., it''s easy to deploy different config files to different machines, or to use templates to slightly vary config files as necessary. As to fixing things that break, I''ve worked hard on making Puppet useful for interactive trouble-shooting. I think I''ve done relatively well, but centralized administration will always place a bit higher burden on troubleshooting.> My other confusion is how I would lay out my svn repository. I > currently have something that looks like this: > > |-- hermes > | |-- boot > | | `-- loader.conf > | |-- etc > | | |-- cvsupfile > | | |-- hosts > | | |-- make.conf > | | |-- newsyslog.conf > | | |-- ntp.conf > | | |-- pf.conf > | | |-- rc.conf > | | |-- resolv.conf > | | |-- syslog.conf > | | `-- zshrc > | `-- usr > | |-- local > | | `-- etc > | | |-- denyhosts.conf > | | |-- dovecot.conf > > each important file is stored relative to where it would be on the > machine in question. i''ve tried unsuccessfully to use svn "in place" > by cp -R''ing my checked out tree into / which I can then maintain in > the normal way, and check in my changes when I''m done, but that very > quickly gets messy and confuses my head.I *highly* recommend that you organize your svn repository semantically, rather than according to file structure. I usually have an ''apps'' subdirectory, and then have a separate subdirectory for each app I''m managing, and I''d put the different config files into the app-specific directories. Then I have a class file for each application, used for all work related to that service -- installing the package, updating the configuration, starting the service, etc. It''s true that this organization style is more important when you''re cross-platform, but I''ve always had less trouble using this system than copying the system file structure. As to just checking the code out directly on disk... I''ve never had much luck with that except in simple cases. You''ll almost always want to use a proxy like Puppet or rsync.> i think ideally this is how i''d like to work: > > something breaks > i log into the broken machine and fiddle with it > make a change and get it working again > commit that change into my local repository > that change gets checked out by the broken machine, overwriting my > earlier change and everything is back to normal (and logged in svn).This should basically be how it works when you use puppet, although you might want to lock the puppet client while you''re fiddling with the box. Then just commit the change to your central repository, and it will get deployed appropriately.> will puppet help me do that? and will i need to change over to it > overnight, having tested it on a test infrastructure first, or will i > be able to move over to it gradually? i''ve read a lot of the docs on > it, but I guess what i''m missing is the big picture, and seeing if > it''s going to improve my life.You definitely can migrate gradually. I always recommend starting somewhere simple like managing the sudoers file, and then gradually increasing the scope of your Puppet configuration.> this is all a bit rabid, so sorry about that. please ask me to > clarify anything that isn''t clear - i''m just trying to work out what > the options are at this stage.No problem. It sounds like you have pretty normal concerns and a relatively normal environment, so you shouldn''t have much trouble. -- If two men agree on everything, you may be sure that one of them is doing the thinking. -- Lyndon B. Johnson --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Tue, Sep 19, 2006 at 01:46:59PM +0100, Kep Woof wrote:> At work we need a better way of running our machines. We''ve got 14 > machines around the world that we look after (mostly web servers but a > few run mail and dns as you''d expect) and we currently maintain them > individually. They all log to a central syslog server over an openvpn > interface, but aside from that they tend to be in different locations. > Could puppet be any use to us in such a crazy environment, or is it > unpractical because of all the network dependencies?Sounds like a pretty reasonable requirement for Puppet.> Slightly less ambitious, and something we need to solve quite quickly, > we have a rack in a datacentre where 4 machines are currently hosted, > and we''ll be adding quite a few more machines over time as the service > they host scales. They''re all on the same subnet and sit behind a > firewall. I was thinking about putting puppetmaster on the firewall > and running everything for these machines from there. The hosts all > run FreeBSD 6.1-RELEASE and run quite a complicated set up of apache, > tomcat, java, mysql, modpython, etc. So I guess my question is this: > Is puppet ready for this kind of deployment, and will i run into any > issues because I''m running on BSD?I don''t know about the FreeBSD aspect, but I''m doing something similar to this for a client with Puppet on Ubuntu Breezy. We''re managing about 25 systems (17 physical with a number of Xen VMs for the non-heavy-lifting parts like the local package cache and the load balancers) in a data centre. The app is a large-scale web serving operation (Java and MySQL everywhere; oh the humanity!). Puppet manages all of these systems once the initial bootstrap install has been done (off a customised install CD for the physical boxes, and using xen-tools for the Xen VMs). We''ve got a delivery of 12 new machines arriving in a week or two (big load spike planned) and I''m expecting that those 12 will all be running the app in production within 24 hours of the boxes arriving at the datacentre. That''s with total staffing of: a junior sysadmin on-site to plug in cables and insert the CD, and me back at the office reading a book^W^W^W supervising. I''m confident that every machine will Just Work pretty much immediately, since the last 6 machines I deployed in a similar manner Just Worked.> Also, more generally, noone wants to run anything that noone else is > running, so I have to ask, are many people here using puppet in > production yet?Between the 25 systems in that data centre, about 50 systems at head office (developer workstations, a complete staging environment, masses of test and development support boxes), and about a dozen systems at a new data centre we''re configuring in a little while, I think you could say that we''re using Puppet in production now. <grin>> If not anyone got any ideas when they will, and what kind of issues are > you running into day to day which might help me decide if it''s the way to > go?Adoption amongst other staff is probably the biggest problem I''ve hit. The moment anything interrupts their mental model of how they think things should work, the tool gets vicious blame. I understand the reaction, but it still irritates me when people complain that the tool is crap because they fought it instead of working with it, and suggest getting rid of it (when previously it took about a full day to configure a machine and get it running, and now it''s about 15 minutes of interactive time and a pile of waiting to get a cluster of a dozen machines sorted).> I should say that I''m a sysadmin, not a programmer. I''ve written > simple scripts that automate a lot of my life (webstats, backups, dns > tests etc) but I''m really not very confident designing and writing > programs that are important and complicated. I really like ruby, and > see puppet as a good way to expose myself to it, but I''m worried that > day to day I might spend ages battling with puppet problems.You''ll spend ages battling with all sorts of things. It might as well be something useful and fun like Ruby. <grin> I''ve found the Puppet community to be one of the most useful and helpful groups of people I''ve ever dealt with, so I don''t think you''ll have any problems getting help if you get stuck up against something. Luke, in particular, just oozes helpfulness, always quick to answer a question or respond to a problem report or even enhancement requests. I''m kinda lucky in the timezone differences, though -- I''ll post a bug report just before I leave work, and then I wake up in the morning to find that Luke''s gone in and fixed it overnight for me.> I guess what I''m asking is, what kind of impact is puppet going to > make on my life?It will make you more attractive to the opposite sex.> Will I be doing most of my work in puppet now?Hopefully.> What does the day to day workflow look like? At present it goes something > like this: > > install machine - unattended netinstallPuppet doesn''t help with this, but automating a basic install is usually pretty simple. That''s all it needs to be, too -- you need just enough software to be able to run puppet. Let the tool do everything from there. In the ''install machine'' step, you''ll also want to tell Puppet about the new machine. In a cluster sitatuin, that usually involves finding the appropriate node stanza: node web1, web2, web3, web4 { include app_server } and adding the new node to it: node web1, web2, web3, web4, web5 { include app_server }> install packages - manually install the portsTell Puppet "I want these packages" for the class you care about.> configure packages - copying an existing /usr/local/etc does most of thatYou''ll have described to Puppet all of the config files you want modified and how you want them modified, and it''ll make sure that It Is Done.> test that it workedPuppet can do this for you, but I usually delegate it to something external like Nagios, which then also *keeps* making sure it''s working right.> fix it if it breaks - log into the host, fiddle with stuff till it works.Puppet can''t help directly here, but it can certainly help to make sure that you can easily replicate any fixes you need to deploy to a number of machines, and that all new machines will get the fixes too. - Matt -- That''s why I love VoIP. You don''t get people phoning up to complain that the network is down. -- Peter Corlett, in the Monastery
> tomcat, java, mysql, modpython, etc. So I guess my question is this: > Is puppet ready for this kind of deployment, and will i run into any > issues because I''m running on BSD? > >i use puppet in quasi production stage (still wondering the best way to write my classes etc...). I want to use it for managing little Virtual private servers. They run on freeBSD , are simple lamp servers but the goal is to make sure some config are enforced and that i can make a simple push to all the server with just one line of puppet code. FreeBSD as some little special things (path mainly and cron thingy) but luke has allway corrected all the issue so now the type i use work okay. With the new cron''s type environement setting i am pretty up and running. Just finishing some test. Right now puppet manage for me to install some config files, create admin account, backup account, configure the sudo, upload ssh keys etc.. this work quite well even in my beta test phase. I will go in "production mode" very soon with .19.1 so i use it in prod on 3 servers, will make it run for 42 vps servers soon and a dozen of debian/redhat boxes also.> Also, more generally, noone wants to run anything that noone else is > running, so I have to ask, are many people here using puppet in > production yet? If not anyone got any ideas when they will, and what > kind of issues are you running into day to day which might help me > decide if it''s the way to go? >having to change one line in puppet to have all the boxes updated is a real pleasure. You will find yourself adding more and more thing to your pp files as you learn the tool and your life will be easier day after day. My main issue is to have to install ruby and facter/puppet on each box, redhat use old version o ruby (stupid rpm system using version of 5year age and no supported way of upgrading..). I use gem as the packages system install such things in so many different places that i prefer a "standard way". Of course there is the startup script problem but 90% of my nodes run puppet on cron so...> I should say that I''m a sysadmin, not a programmer. I''ve written > simple scripts that automate a lot of my life (webstats, backups, dns > tests etc) but I''m really not very confident designing and writing > programs that are important and complicated. I really like ruby, and > see puppet as a good way to expose myself to it, but I''m worried that > day to day I might spend ages battling with puppet problems. >i am like you. I have found puppet an easy partner at reducing repetitive task. each day i find a new way to add a pp file to do some little parts i was doing by hand. With the added benefit that now it is ALLWAYS done the same way, it seems i have come with hundreds of ways to configure phpmyadmin on my boxes...hopefully those days are over> I guess what I''m asking is, what kind of impact is puppet going to > make on my life? Will I be doing most of my work in puppet now? What > does the day to day workflow look like? At present it goes something > like this: > > install packages - manually install the ports >ports are resisting to me because most i use require to answer questions so the package install just failed waiting for inputs. I then just use an exec that runs the make files manualy to install them and after that i keep them up to date with puppet. regards, Ghislain. _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Tue, Sep 19, 2006 at 10:12:55AM -0500, Ryan Schwartz wrote:> If you get into the mindset that if you do something in puppet once, > you never have to do it again - it makes the initial time investment > to developing automation for your environment a no-brainer IMO. All > automation products will incur this extra effort, not just puppet, > but puppet is the most flexible and augmentable system I''ve come across. > > It''s a difficult methodology to break in-to and habitualize since we > all tend to do the hands-on fix, fix again, fix again (...) monkey > dance ourselves when something like puppet can be taught to dance > whilst we turn the crank. :-) For me at least, it just makes sense to > put the extra effort in to get things automated when in the long term > it''s going to allow me to do the research and development I love > doing, rather than daily sysadmin gruntwork.Possibly the best explanation of why Puppet (and system automation in general) is a Good Thing. Thanks. - Matt -- It has become trendy, in some circles, to lament the Internet''s poor performance/congestion/[...]/<insert issue here>. After firmly denouncing the Internet, the company or individual then touts their product, which will fix/replace/augment the Internet. -- Daniel Golding, NANOG