Folks, I''ve been struggling a bit with how we''re using puppet (at my job): how do you validate that puppet has done what it is supposed to, and even troublesome, how you validate that it has done what you intended it to do? Since I''m the only one who is writing the puppet profiles and working with it on a daily basis, I''m the only one who can decipher the puppet logs. I often get the question: how do we know when the system is ready for production? I''ve been playing around[1] with cucumber & webrat, and have pieced together a way to do behavior driven infrastructure testing. Puppet takes care of getting the system configured correctly, but there are often other pieces involved, like opening firewall ports, adding DNS entries, sendmail routing, etc. Which must be done outside of puppet to get the system ready for release. When you write code, you always use unit testing & integration testing to verify that the application is working as expected, but why don''t we use that when we install a system? What are you using to verify that your system is correctly configured and behaves the way you want? [1] <http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> cheers, /Martin --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2009/11/6 Martin Englund <martin.englund@sun.com>:> > Folks, > > I''ve been struggling a bit with how we''re using puppet (at my job): > how do you validate that puppet has done what it is supposed to, and > even troublesome, how you validate that it has done what you intended > it to do? > > Since I''m the only one who is writing the puppet profiles and working > with it on a daily basis, I''m the only one who can decipher the puppet > logs. I often get the question: how do we know when the system is > ready for production? > > I''ve been playing around[1] with cucumber & webrat, and have pieced > together a way to do behavior driven infrastructure testing. Puppet > takes care of getting the system configured correctly, but there are > often other pieces involved, like opening firewall ports, adding DNS > entries, sendmail routing, etc. Which must be done outside of puppet > to get the system ready for release. > > When you write code, you always use unit testing & integration testing > to verify that the application is working as expected, but why don''t > we use that when we install a system? > > What are you using to verify that your system is correctly configured > and behaves the way you want? > > [1] <http://blogs.sun.com/martin/entry/behavior_driven_infrastructure>[although i''ve commented on the excellent blog entry, i''m posting here] Hi Martin, It looks like there''s a bit of crossover here with a project i''ve been working on the last few months called cucumber-nagios[0]. It takes the result of a Cucumber run and outputs it in the Nagios plugin format. Essentially you use it to express your intentions in plain language, and verify your intentions periodically through your monitoring system. Just like what you''ve posted about. :-) Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in Belgium last weekend, and I got talking with people about expanding the library of steps to cover things like logins over SSH, file manipulation, and mail delivery. It would be cool if we could centralise our efforts and focus on building an awesome library of reusable steps to test our infrastructure. Your point about doing behaviour driven development when writing software is right on the mark. From an infrastructure perspective, I like to think of Cucumber as the testing tool, and Puppet as the programming language. Anyhow, i''d be interested to hear what other people think about this idea! Cheers, Lindsay [0] http://auxesis.github.com/cucumber-nagios -- http://holmwood.id.au/~lindsay/ (me) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Fri, Nov 6, 2009 at 2:56 PM, Lindsay Holmwood <lindsay@holmwood.id.au>wrote:> > 2009/11/6 Martin Englund <martin.englund@sun.com>: > > > > Folks, > > > > I''ve been struggling a bit with how we''re using puppet (at my job): > > how do you validate that puppet has done what it is supposed to, and > > even troublesome, how you validate that it has done what you intended > > it to do? > > > > Since I''m the only one who is writing the puppet profiles and working > > with it on a daily basis, I''m the only one who can decipher the puppet > > logs. I often get the question: how do we know when the system is > > ready for production? > > > > I''ve been playing around[1] with cucumber & webrat, and have pieced > > together a way to do behavior driven infrastructure testing. Puppet > > takes care of getting the system configured correctly, but there are > > often other pieces involved, like opening firewall ports, adding DNS > > entries, sendmail routing, etc. Which must be done outside of puppet > > to get the system ready for release. > > > > When you write code, you always use unit testing & integration testing > > to verify that the application is working as expected, but why don''t > > we use that when we install a system? > > > > What are you using to verify that your system is correctly configured > > and behaves the way you want? > > > > [1] <http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> > > [although i''ve commented on the excellent blog entry, i''m posting here] > > Hi Martin, > It looks like there''s a bit of crossover here with a project i''ve been > working on the last few months called cucumber-nagios[0]. It takes the > result of a Cucumber run and outputs it in the Nagios plugin format. > Essentially you use it to express your intentions in plain language, > and verify your intentions periodically through your monitoring > system. Just like what you''ve posted about. :-) > > Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in > Belgium last weekend, and I got talking with people about expanding > the library of steps to cover things like logins over SSH, file > manipulation, and mail delivery. It would be cool if we could > centralise our efforts and focus on building an awesome library of > reusable steps to test our infrastructure. > > Your point about doing behaviour driven development when writing > software is right on the mark. From an infrastructure perspective, I > like to think of Cucumber as the testing tool, and Puppet as the > programming language. > > Anyhow, i''d be interested to hear what other people think about this idea! > > Cheers, > Lindsay > > [0] http://auxesis.github.com/cucumber-nagios > > > -- > http://holmwood.id.au/~lindsay/ <http://holmwood.id.au/%7Elindsay/> (me) > > > >I am certainly interested from a infrastructure point of view having like you say a library of reusable steps which would fit the bill nicely, The ability to build with Puppet and check that build with Cucumber-Nagios would be awesome and more in line in which I would consider Test Driven Infrastructure. Will --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Lindsay! On Nov 6, 3:56 pm, Lindsay Holmwood <lind...@holmwood.id.au> wrote:> It looks like there''s a bit of crossover here with a project i''ve been > working on the last few months called cucumber-nagios[0]. It takes the > result of a Cucumber run and outputs it in the Nagios plugin format. > Essentially you use it to express your intentions in plain language, > and verify your intentions periodically through your monitoring > system. Just like what you''ve posted about. :-) >It sure does look like there is an overlap :)> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in > Belgium last weekend, and I got talking with people about expanding > the library of steps to cover things like logins over SSH, file > manipulation, and mail delivery. It would be cool if we could > centralise our efforts and focus on building an awesome library of > reusable steps to test our infrastructure. >It is indeed better to work on this jointly. You are much further along the way, so I''ll clone your cucumber-nagios repository and work in integrating the stuff I''ve done into your code. You''ll be hearing from me shortly... I''m also talking to Ohad about integrating this into the foreman[1]. Then it will become the IDE of system/infrastructure development :) [1] <http://theforeman.org/> cheers, /Martin --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
for what it''s worth: for a while I was experimenting with cucumber to test scripts running on linux machines. And yes , IMHO it is the way to go . It worked well with the behavior testing just a you guys described. still just as in regular development, bdd can be complemented with tdd. The way i did the tdd, was by using the exit codes of the shell commands. I created an abstraction in ruby for execute (local, remote) and upload, download of files. If the command did not have exit 0 (the default) then i imagined the test being failed. Using snapshots of the virtual machines, I ran the scripts against a virtual machine (used virtualbox), with the ability to do a rollback. I know this is not the way puppet normally operates, but I used the (if i recall correctly) the puppetrun command to execute it and see what happened If all of these exit codes where ok (similar to unit tests), i ran the bdd tests to see if everything was correct. By integrating the tdd, and the bdd within a CI-system, I was able to have it run continuously and rebuild the system over and over again. Another thing I want to mention is that you can easily abstract commands in a custom dsl, but I really like to see what actual commands got executed. This is slightly off topic maybe, but by logging/running the actual commands and have the whole environment build like that (including making a floppy, booting the vms) it allowed me to create a kind of install document with the actual commands instead of pointing to a newly invented DSL. So I say yes to an abstraction layer, but no to hiding the actual commands. just my 2 c On Nov 6, 3:56 pm, Lindsay Holmwood <lind...@holmwood.id.au> wrote:> 2009/11/6 Martin Englund <martin.engl...@sun.com>: > > > > > > > Folks, > > > I''ve been struggling a bit with how we''re using puppet (at my job): > > how do you validate that puppet has done what it is supposed to, and > > even troublesome, how you validate that it has done what you intended > > it to do? > > > Since I''m the only one who is writing the puppet profiles and working > > with it on a daily basis, I''m the only one who can decipher the puppet > > logs. I often get the question: how do we know when the system is > > ready for production? > > > I''ve been playing around[1] with cucumber & webrat, and have pieced > > together a way to do behavior driven infrastructure testing. Puppet > > takes care of getting the system configured correctly, but there are > > often other pieces involved, like opening firewall ports, adding DNS > > entries, sendmail routing, etc. Which must be done outside of puppet > > to get the system ready for release. > > > When you write code, you always use unit testing & integration testing > > to verify that the application is working as expected, but why don''t > > we use that when we install a system? > > > What are you using to verify that your system is correctly configured > > and behaves the way you want? > > > [1] <http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> > > [although i''ve commented on the excellent blog entry, i''m posting here] > > Hi Martin, > It looks like there''s a bit of crossover here with a project i''ve been > working on the last few months called cucumber-nagios[0]. It takes the > result of a Cucumber run and outputs it in the Nagios plugin format. > Essentially you use it to express your intentions in plain language, > and verify your intentions periodically through your monitoring > system. Just like what you''ve posted about. :-) > > Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in > Belgium last weekend, and I got talking with people about expanding > the library of steps to cover things like logins over SSH, file > manipulation, and mail delivery. It would be cool if we could > centralise our efforts and focus on building an awesome library of > reusable steps to test our infrastructure. > > Your point about doing behaviour driven development when writing > software is right on the mark. From an infrastructure perspective, I > like to think of Cucumber as the testing tool, and Puppet as the > programming language. > > Anyhow, i''d be interested to hear what other people think about this idea! > > Cheers, > Lindsay > > [0]http://auxesis.github.com/cucumber-nagios > > --http://holmwood.id.au/~lindsay/(me)--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Isaac Christoffersen
2009-Nov-07 19:55 UTC
[Puppet Users] Re: Behavior Driven Infrastructure
I''d be interested in hearing more about this. I''ve been working on a project in which we do a lot of automated server builds and configuration using various OSS tools, including puppet. However, one thing that I''ve been looking for is a way to write a set of test suites to validate the server configuration. The combination of TDD and BDD with server rollouts is very attractive. I''d also be curious to know what CI server people are using to help coordinate these intermediate builds and tests. On 11/7/09 11:00 AM, Patrick Debois wrote:> for what it''s worth: > > for a while I was experimenting with cucumber to test scripts running > on linux machines. And yes , IMHO it is the way to go . > > It worked well with the behavior testing just a you guys described. > still just as in regular development, bdd can be complemented with > tdd. > The way i did the tdd, was by using the exit codes of the shell > commands. I created an abstraction in ruby for execute (local, remote) > and upload, download of files. > > If the command did not have exit 0 (the default) then i imagined the > test being failed. > Using snapshots of the virtual machines, I ran the scripts against a > virtual machine (used virtualbox), with the ability to do a rollback. > I know this is not the way puppet normally operates, but I used the > (if i recall correctly) the puppetrun command to execute it and see > what happened > If all of these exit codes where ok (similar to unit tests), i ran the > bdd tests to see if everything was correct. > > By integrating the tdd, and the bdd within a CI-system, I was able to > have it run continuously and rebuild the system over and over again. > > Another thing I want to mention is that you can easily abstract > commands in a custom dsl, but I really like to see what actual > commands got executed. > This is slightly off topic maybe, but by logging/running the actual > commands and have the whole environment build like that (including > making a floppy, booting the vms) > it allowed me to create a kind of install document with the actual > commands instead of pointing to a newly invented DSL. > So I say yes to an abstraction layer, but no to hiding the actual > commands. > > just my 2 c > > On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote: > >> 2009/11/6 Martin Englund<martin.engl...@sun.com>: >> >> >> >> >> >> >>> Folks, >>> >> >>> I''ve been struggling a bit with how we''re using puppet (at my job): >>> how do you validate that puppet has done what it is supposed to, and >>> even troublesome, how you validate that it has done what you intended >>> it to do? >>> >> >>> Since I''m the only one who is writing the puppet profiles and working >>> with it on a daily basis, I''m the only one who can decipher the puppet >>> logs. I often get the question: how do we know when the system is >>> ready for production? >>> >> >>> I''ve been playing around[1] with cucumber& webrat, and have pieced >>> together a way to do behavior driven infrastructure testing. Puppet >>> takes care of getting the system configured correctly, but there are >>> often other pieces involved, like opening firewall ports, adding DNS >>> entries, sendmail routing, etc. Which must be done outside of puppet >>> to get the system ready for release. >>> >> >>> When you write code, you always use unit testing& integration testing >>> to verify that the application is working as expected, but why don''t >>> we use that when we install a system? >>> >> >>> What are you using to verify that your system is correctly configured >>> and behaves the way you want? >>> >> >>> [1]<http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> >>> >> [although i''ve commented on the excellent blog entry, i''m posting here] >> >> Hi Martin, >> It looks like there''s a bit of crossover here with a project i''ve been >> working on the last few months called cucumber-nagios[0]. It takes the >> result of a Cucumber run and outputs it in the Nagios plugin format. >> Essentially you use it to express your intentions in plain language, >> and verify your intentions periodically through your monitoring >> system. Just like what you''ve posted about. :-) >> >> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in >> Belgium last weekend, and I got talking with people about expanding >> the library of steps to cover things like logins over SSH, file >> manipulation, and mail delivery. It would be cool if we could >> centralise our efforts and focus on building an awesome library of >> reusable steps to test our infrastructure. >> >> Your point about doing behaviour driven development when writing >> software is right on the mark. From an infrastructure perspective, I >> like to think of Cucumber as the testing tool, and Puppet as the >> programming language. >> >> Anyhow, i''d be interested to hear what other people think about this idea! >> >> Cheers, >> Lindsay >> >> [0]http://auxesis.github.com/cucumber-nagios >> >> --http://holmwood.id.au/~lindsay/(me) >> > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I use Hudson. I''ve used Test::Unit to wrap validations of generated config files in the past. I want to know as early as possible if something isn''t going to work. Functionally testing the config from the outside in might be easy but slow. You can deploy to a VM and actually test it using the goodness of cucumber. I''m still looking for the equivalent of unit tests: how do you quickly test function from the inside to give enough confidence and a fast feedback loop? Maybe we need to build out Patrick''s approach with a library of command-line tools to verify that everything works. J. 2009/11/7 Isaac Christoffersen <isaac.christoffersen@gmail.com>:> > I''d be interested in hearing more about this. I''ve been working on a > project in which we do a lot of automated server builds and > configuration using various OSS tools, including puppet. However, one > thing that I''ve been looking for is a way to write a set of test suites > to validate the server configuration. The combination of TDD and BDD > with server rollouts is very attractive. > > I''d also be curious to know what CI server people are using to help > coordinate these intermediate builds and tests. > > > > On 11/7/09 11:00 AM, Patrick Debois wrote: >> for what it''s worth: >> >> for a while I was experimenting with cucumber to test scripts running >> on linux machines. And yes , IMHO it is the way to go . >> >> It worked well with the behavior testing just a you guys described. >> still just as in regular development, bdd can be complemented with >> tdd. >> The way i did the tdd, was by using the exit codes of the shell >> commands. I created an abstraction in ruby for execute (local, remote) >> and upload, download of files. >> >> If the command did not have exit 0 (the default) then i imagined the >> test being failed. >> Using snapshots of the virtual machines, I ran the scripts against a >> virtual machine (used virtualbox), with the ability to do a rollback. >> I know this is not the way puppet normally operates, but I used the >> (if i recall correctly) the puppetrun command to execute it and see >> what happened >> If all of these exit codes where ok (similar to unit tests), i ran the >> bdd tests to see if everything was correct. >> >> By integrating the tdd, and the bdd within a CI-system, I was able to >> have it run continuously and rebuild the system over and over again. >> >> Another thing I want to mention is that you can easily abstract >> commands in a custom dsl, but I really like to see what actual >> commands got executed. >> This is slightly off topic maybe, but by logging/running the actual >> commands and have the whole environment build like that (including >> making a floppy, booting the vms) >> it allowed me to create a kind of install document with the actual >> commands instead of pointing to a newly invented DSL. >> So I say yes to an abstraction layer, but no to hiding the actual >> commands. >> >> just my 2 c >> >> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote: >> >>> 2009/11/6 Martin Englund<martin.engl...@sun.com>: >>> >>> >>> >>> >>> >>> >>>> Folks, >>>> >>> >>>> I''ve been struggling a bit with how we''re using puppet (at my job): >>>> how do you validate that puppet has done what it is supposed to, and >>>> even troublesome, how you validate that it has done what you intended >>>> it to do? >>>> >>> >>>> Since I''m the only one who is writing the puppet profiles and working >>>> with it on a daily basis, I''m the only one who can decipher the puppet >>>> logs. I often get the question: how do we know when the system is >>>> ready for production? >>>> >>> >>>> I''ve been playing around[1] with cucumber& webrat, and have pieced >>>> together a way to do behavior driven infrastructure testing. Puppet >>>> takes care of getting the system configured correctly, but there are >>>> often other pieces involved, like opening firewall ports, adding DNS >>>> entries, sendmail routing, etc. Which must be done outside of puppet >>>> to get the system ready for release. >>>> >>> >>>> When you write code, you always use unit testing& integration testing >>>> to verify that the application is working as expected, but why don''t >>>> we use that when we install a system? >>>> >>> >>>> What are you using to verify that your system is correctly configured >>>> and behaves the way you want? >>>> >>> >>>> [1]<http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> >>>> >>> [although i''ve commented on the excellent blog entry, i''m posting here] >>> >>> Hi Martin, >>> It looks like there''s a bit of crossover here with a project i''ve been >>> working on the last few months called cucumber-nagios[0]. It takes the >>> result of a Cucumber run and outputs it in the Nagios plugin format. >>> Essentially you use it to express your intentions in plain language, >>> and verify your intentions periodically through your monitoring >>> system. Just like what you''ve posted about. :-) >>> >>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in >>> Belgium last weekend, and I got talking with people about expanding >>> the library of steps to cover things like logins over SSH, file >>> manipulation, and mail delivery. It would be cool if we could >>> centralise our efforts and focus on building an awesome library of >>> reusable steps to test our infrastructure. >>> >>> Your point about doing behaviour driven development when writing >>> software is right on the mark. From an infrastructure perspective, I >>> like to think of Cucumber as the testing tool, and Puppet as the >>> programming language. >>> >>> Anyhow, i''d be interested to hear what other people think about this idea! >>> >>> Cheers, >>> Lindsay >>> >>> [0]http://auxesis.github.com/cucumber-nagios >>> >>> --http://holmwood.id.au/~lindsay/(me) >>> >> > >> > > > >-- Julian Simpson Software Build and Deployment http://www.build-doctor.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Isaac Christoffersen
2009-Nov-08 14:52 UTC
[Puppet Users] Re: Behavior Driven Infrastructure
I''ve been thinking about using Hudson for this, so its good to know that there are others out there doing this. 3 projects that I''ve bookmarked to look at for unit testing frameworks for the OS are: https://fedoraproject.org/wiki/AutoQA http://autotest.kernel.org/ http://www.linux-kvm.org/page/Main_Page I haven''t had time to do much more than scan the web site, but they seem to be worth watching. I''ve also started including test.pp scripts as part of each module. These scripts check directories or do some execs (i.e. wgets, pings, lookups). These scripts then run with every puppet update push. However, this isn''t ideal for production and there needs to be a better way to trigger the running of unit tests. But it has been helpful in validating iptables configuration and ensuring services and their clients are on the same network. On 11/8/09 7:43 AM, Julian Simpson wrote:> I use Hudson. > > I''ve used Test::Unit to wrap validations of generated config files in > the past. I want to know as early as possible if something isn''t > going to work. > > Functionally testing the config from the outside in might be easy but > slow. You can deploy to a VM and actually test it using the goodness > of cucumber. I''m still looking for the equivalent of unit tests: how > do you quickly test function from the inside to give enough confidence > and a fast feedback loop? Maybe we need to build out Patrick''s > approach with a library of command-line tools to verify that > everything works. > > J. > > > 2009/11/7 Isaac Christoffersen<isaac.christoffersen@gmail.com>: > >> I''d be interested in hearing more about this. I''ve been working on a >> project in which we do a lot of automated server builds and >> configuration using various OSS tools, including puppet. However, one >> thing that I''ve been looking for is a way to write a set of test suites >> to validate the server configuration. The combination of TDD and BDD >> with server rollouts is very attractive. >> >> I''d also be curious to know what CI server people are using to help >> coordinate these intermediate builds and tests. >> >> >> >> On 11/7/09 11:00 AM, Patrick Debois wrote: >> >>> for what it''s worth: >>> >>> for a while I was experimenting with cucumber to test scripts running >>> on linux machines. And yes , IMHO it is the way to go . >>> >>> It worked well with the behavior testing just a you guys described. >>> still just as in regular development, bdd can be complemented with >>> tdd. >>> The way i did the tdd, was by using the exit codes of the shell >>> commands. I created an abstraction in ruby for execute (local, remote) >>> and upload, download of files. >>> >>> If the command did not have exit 0 (the default) then i imagined the >>> test being failed. >>> Using snapshots of the virtual machines, I ran the scripts against a >>> virtual machine (used virtualbox), with the ability to do a rollback. >>> I know this is not the way puppet normally operates, but I used the >>> (if i recall correctly) the puppetrun command to execute it and see >>> what happened >>> If all of these exit codes where ok (similar to unit tests), i ran the >>> bdd tests to see if everything was correct. >>> >>> By integrating the tdd, and the bdd within a CI-system, I was able to >>> have it run continuously and rebuild the system over and over again. >>> >>> Another thing I want to mention is that you can easily abstract >>> commands in a custom dsl, but I really like to see what actual >>> commands got executed. >>> This is slightly off topic maybe, but by logging/running the actual >>> commands and have the whole environment build like that (including >>> making a floppy, booting the vms) >>> it allowed me to create a kind of install document with the actual >>> commands instead of pointing to a newly invented DSL. >>> So I say yes to an abstraction layer, but no to hiding the actual >>> commands. >>> >>> just my 2 c >>> >>> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote: >>> >>> >>>> 2009/11/6 Martin Englund<martin.engl...@sun.com>: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> Folks, >>>>> >>>>> >>>> >>>>> I''ve been struggling a bit with how we''re using puppet (at my job): >>>>> how do you validate that puppet has done what it is supposed to, and >>>>> even troublesome, how you validate that it has done what you intended >>>>> it to do? >>>>> >>>>> >>>> >>>>> Since I''m the only one who is writing the puppet profiles and working >>>>> with it on a daily basis, I''m the only one who can decipher the puppet >>>>> logs. I often get the question: how do we know when the system is >>>>> ready for production? >>>>> >>>>> >>>> >>>>> I''ve been playing around[1] with cucumber& webrat, and have pieced >>>>> together a way to do behavior driven infrastructure testing. Puppet >>>>> takes care of getting the system configured correctly, but there are >>>>> often other pieces involved, like opening firewall ports, adding DNS >>>>> entries, sendmail routing, etc. Which must be done outside of puppet >>>>> to get the system ready for release. >>>>> >>>>> >>>> >>>>> When you write code, you always use unit testing& integration testing >>>>> to verify that the application is working as expected, but why don''t >>>>> we use that when we install a system? >>>>> >>>>> >>>> >>>>> What are you using to verify that your system is correctly configured >>>>> and behaves the way you want? >>>>> >>>>> >>>> >>>>> [1]<http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> >>>>> >>>>> >>>> [although i''ve commented on the excellent blog entry, i''m posting here] >>>> >>>> Hi Martin, >>>> It looks like there''s a bit of crossover here with a project i''ve been >>>> working on the last few months called cucumber-nagios[0]. It takes the >>>> result of a Cucumber run and outputs it in the Nagios plugin format. >>>> Essentially you use it to express your intentions in plain language, >>>> and verify your intentions periodically through your monitoring >>>> system. Just like what you''ve posted about. :-) >>>> >>>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in >>>> Belgium last weekend, and I got talking with people about expanding >>>> the library of steps to cover things like logins over SSH, file >>>> manipulation, and mail delivery. It would be cool if we could >>>> centralise our efforts and focus on building an awesome library of >>>> reusable steps to test our infrastructure. >>>> >>>> Your point about doing behaviour driven development when writing >>>> software is right on the mark. From an infrastructure perspective, I >>>> like to think of Cucumber as the testing tool, and Puppet as the >>>> programming language. >>>> >>>> Anyhow, i''d be interested to hear what other people think about this idea! >>>> >>>> Cheers, >>>> Lindsay >>>> >>>> [0]http://auxesis.github.com/cucumber-nagios >>>> >>>> --http://holmwood.id.au/~lindsay/(me) >>>> >>>> >>>> >>> >> >>> >> > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Isaac Christoffersen schrieb:> to validate the server configuration. The combination of TDD and BDD > with server rollouts is very attractive.I think the whole topic is really hot. :-) It would probably a good idea to expand what the difference between TDD and BDD in systems management actually is about. Not everyone here on the list is fine with the subtle differences in terminology (for example, me ;-) ) Regards Christian -- Dipl.-Inf. Christian Kauhaus <>< · kc@gocept.com · systems administration gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1 Zope and Plone consulting and development --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
My understanding is that Test Driven Development hit the mainstream (did smalltalk have a unit test framework?) with JUnit.[1] Junit allowed developers to automate their unit tests. All the unit tests for your project should run in seconds, if your codebase and tests are loosely coupled. e.g. testThatTheFooObjectReturnsABarWhenPassedABaz Those TDD style tests can be pretty ugly. In theory you can use them as a spec to describe what code does, but in practice, no. So a guy called Dan North coined the term Behaviour Driven Development, and focussed on the language used to help Business Analysts or other guys in suits to specify. This is where the Given/When/Then syntax comes from. He worked on JBehave as an alternative to JUnit so you can specify behavior. But Rspec and Cucumber are where the notion has taken off. I''d write more but I have to make my kids'' dinner :) J. [1] http://www.junit.org/ 2009/11/9 Christian Kauhaus <kc@gocept.com>:> > Isaac Christoffersen schrieb: >> to validate the server configuration. The combination of TDD and BDD >> with server rollouts is very attractive. > > I think the whole topic is really hot. :-) > > It would probably a good idea to expand what the difference between TDD > and BDD in systems management actually is about. > > Not everyone here on the list is fine with the subtle differences in > terminology (for example, me ;-) ) > > Regards > > Christian > > -- > Dipl.-Inf. Christian Kauhaus <>< · kc@gocept.com · systems administration > gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany > http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1 > Zope and Plone consulting and development > > > >-- Julian Simpson Software Build and Deployment http://www.build-doctor.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2009/11/6 Martin Englund <martin.englund@sun.com>:>> > It is indeed better to work on this jointly. You are much further > along the way, so I''ll clone your cucumber-nagios repository and work > in integrating the stuff I''ve done into your code. You''ll be hearing > from me shortly...So i''ve added some SSH steps to cucumber-nagios, along the lines of what Martin blogged about a few days ago. I''ve published a blog post about the new feature, and how this could potentially be used withina Behaviour Driven Infrastructure techniques: http://holmwood.id.au/~lindsay/2009/11/09/behaviour-driven-infrastructure-through-cucumber/ Cheers, Lindsay -- http://holmwood.id.au/~lindsay/ (me) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Julian Simpson schrieb:> Those TDD style tests can be pretty ugly. In theory you can use them > as a spec to describe what code does, but in practice, no. So a guy > called Dan North coined the term Behaviour Driven Development, and > focussed on the language used to help Business Analysts or other guys > in suits to specify. This is where the Given/When/Then syntax comes > from.We use Puppet at our site quite extensively and try to start implementing a feature by writing a custom Nagios check[1]. This starts as a red one. After that we modify the manifests so that the check passes. I think this is pretty much a TDD style of development. What part of the procedure would I need to change in a BDD style of development? Is this just another representation of the checks? Are there differences in the process? Regards Christian [1] With the Nagios::Plugin Perl library, this is just a breeze. -- Dipl.-Inf. Christian Kauhaus <>< · kc@gocept.com · systems administration gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1 Zope and Plone consulting and development --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
FYI: for those looking for examples and more discussions. I''ve put my collection of infrastructure testing links online: http://www.jedi.be/blog/2009/11/12/collection-of-test-driven-infrastructure-links/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Patrick Debois schrieb:> I''ve put my collection of infrastructure testing links online: > http://www.jedi.be/blog/2009/11/12/collection-of-test-driven-infrastructure-links/Great compilation. Thank you for sharing it with us. :-) Regards Christian -- Dipl.-Inf. Christian Kauhaus <>< · kc@gocept.com · systems administration gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1 Zope and Plone consulting and development --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Agreed. Great collection of testing links. Cheers, Ryan On Nov 13, 2009, at 1:15 PM, Christian Kauhaus wrote:> > Patrick Debois schrieb: >> I''ve put my collection of infrastructure testing links online: >> http://www.jedi.be/blog/2009/11/12/collection-of-test-driven-infrastructure-links/ > > Great compilation. Thank you for sharing it with us. :-) > > Regards > > Christian > > -- > Dipl.-Inf. Christian Kauhaus <>< · kc@gocept.com · systems administration > gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany > http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1 > Zope and Plone consulting and development > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---