This was: ruby 1.8.4 (2005-12-24) [x86_64-linux] vs ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] Ruby Enterprise Edition 20090421 on Ubuntu Dapper, using Passenger with Apache. I used the puppet-test script in the ext/ directory to simulate 50 simultaneous runs with a repeat of 10. Both runs ran from the same clean state. I measured peak memory usage and total catalog compilation time (ie adding up the "compiled catalog in xxx.x seconds" output) with our actual Puppet configuration. Ruby 1.8.4: peak memory consumption: 2686 MB total catalog compilation time: 26632 seconds Ruby EE 20090421: peak memory consumption: 1462 MB total catalog compilation time: 13693.5 seconds wow. -- Nigel Kersten nigelk@google.com System Administrator Google, Inc. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Arjuna Christensen
2009-May-12 01:40 UTC
[Puppet Users] Re: Ruby EE vs Ruby 1.8.4 results.
Hey, On 12/05/2009, at 1:35 PM, Nigel Kersten wrote:> > This was: > > ruby 1.8.4 (2005-12-24) [x86_64-linux] > vs > ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] > Ruby Enterprise Edition 20090421Might I recommend also testing the MBARI[1] patches[2] for Ruby 1.8.6/1.8.7? I''ve seen massive reductions (-30% real memory, 2-3x req/ s) across a variety of applications. http://wiki.github.com/brentr/matzruby http://github.com/brentr/matzruby/tree/v1_8_7_72-mbari Regards, AJ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mon, May 11, 2009 at 6:40 PM, Arjuna Christensen <aj@junglist.gen.nz> wrote:> > Hey, > > On 12/05/2009, at 1:35 PM, Nigel Kersten wrote: > >> >> This was: >> >> ruby 1.8.4 (2005-12-24) [x86_64-linux] >> vs >> ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] >> Ruby Enterprise Edition 20090421 > > Might I recommend also testing the MBARI[1] patches[2] for Ruby > 1.8.6/1.8.7? I''ve seen massive reductions (-30% real memory, 2-3x req/ > s) across a variety of applications. > > http://wiki.github.com/brentr/matzruby > http://github.com/brentr/matzruby/tree/v1_8_7_72-mbariYep. That''s next on my list. -- Nigel Kersten nigelk@google.com System Administrator Google, Inc. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Thats a wow... Did you ever compare 32bit? I am using 32bit OS (RHE5) on all of our puppetmasters as ive noticed the 64bit of ruby is more hungry... First you make me considering passenger now ree....:) many thanks, Ohad On 5/12/09, Nigel Kersten <nigelk@google.com> wrote:> > This was: > > ruby 1.8.4 (2005-12-24) [x86_64-linux] > vs > ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] > Ruby Enterprise Edition 20090421 > > on Ubuntu Dapper, using Passenger with Apache. > > I used the puppet-test script in the ext/ directory to simulate 50 > simultaneous runs with a repeat of 10. Both runs ran from the same > clean state. > > I measured peak memory usage and total catalog compilation time (ie > adding up the "compiled catalog in xxx.x seconds" output) with our > actual Puppet configuration. > > > Ruby 1.8.4: > peak memory consumption: 2686 MB > total catalog compilation time: 26632 seconds > > > Ruby EE 20090421: > peak memory consumption: 1462 MB > total catalog compilation time: 13693.5 seconds > > > wow. > > > -- > Nigel Kersten > nigelk@google.com > System Administrator > Google, Inc. > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''ve been trying to reproduce your results, and got mixed results myself. I did a similar test with puppet-test (ignoring for a second that a major memory abuser is the file copying) i did simple compile tests. the setup include 3 main scenarios: 1. latest passenger + ree 2. passenger with RHEL5 ruby 3. simple mongrel setup with RHEL5. for fun, I also added a yaml/marshal comparison. my test client was always the same with this arguments: /puppet-test --server serverX --fork 5 --catalog_format marshal/yaml --repeat 10 according to my calculation 60 (1+5 fork x 10) clients is at least 60 clients per minute ==> 1800 clients per puppetmaster. my output is: passenger + ree + marshal: AVG: 42.4674576271186, OK: 59 failed: 1 passenger + ree + yaml: AVG: 46.9481666666667, OK: 60 failed: 0 passenger + RHE5 ruby / marshal: AVG: 50.6046666666667, OK: 60 failed: 0 passenger + RHE5 ruby / yaml: AVG: 52.1456666666667, OK: 60 failed: 0 mongrel (4 processes) + with RHE5 ruby / marshal: AVG: 48.5047272727273, OK: 55 failed: 5 mongrel (4 processes) + with RHE5 ruby / yaml: AVG: 38.0254347826087, OK: 46 failed: 14 as it goes for memory consumptions, no surprises here, passenger processes retain within 100mb per process, mongrel went to an average of 220mb, however, passenger used 6 processes, and mongrel only 4.. so maybe I could run 6 mongrels and compare further. I did not see a huge different in memory consumption between REE and normal RHE5 ruby when using passenger. *in my setup, failed are clients who didnt get a catalog within 120 seconds. Nigel, I''m surpized how you got such a different values for compile time, could it be that you server was swapping? another side effect that I found in more than one occasion that if I killed the puppet-test, the passenger processes went on for a long time ( I killed them manually after 15 minutes) and passenger-status reported that there are no active processes but cpu usage was 100%..... so bottom line so far, even if you have a balanced mongrel / cpu /client ratio, performance-wise there is a slight difference towards ree (in my tests), passenger is handling memory usage better than mongrels, but maybe have some hiccups and handles high load much better than mongrels... Ohad On Tue, May 12, 2009 at 9:35 AM, Nigel Kersten <nigelk@google.com> wrote:> > This was: > > ruby 1.8.4 (2005-12-24) [x86_64-linux] > vs > ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] > Ruby Enterprise Edition 20090421 > > on Ubuntu Dapper, using Passenger with Apache. > > I used the puppet-test script in the ext/ directory to simulate 50 > simultaneous runs with a repeat of 10. Both runs ran from the same > clean state. > > I measured peak memory usage and total catalog compilation time (ie > adding up the "compiled catalog in xxx.x seconds" output) with our > actual Puppet configuration. > > > Ruby 1.8.4: > peak memory consumption: 2686 MB > total catalog compilation time: 26632 seconds > > > Ruby EE 20090421: > peak memory consumption: 1462 MB > total catalog compilation time: 13693.5 seconds > > > wow. > > > -- > Nigel Kersten > nigelk@google.com > System Administrator > Google, Inc. > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2009/5/12 Nigel Kersten <nigelk@google.com>:> > This was: > > ruby 1.8.4 (2005-12-24) [x86_64-linux] > vs > ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] > Ruby Enterprise Edition 20090421 > > on Ubuntu Dapper, using Passenger with Apache.Interesting results. Which version of puppet were you testing. Any chance you could try 0.25 too for comparison. Paul --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, May 12, 2009 at 2:05 AM, Ohad Levy <ohadlevy@gmail.com> wrote:> I''ve been trying to reproduce your results, and got mixed results myself. > > I did a similar test with puppet-test (ignoring for a second that a major > memory abuser is the file copying) i did simple compile tests. > > the setup include 3 main scenarios: > > latest passenger + ree > passenger with RHEL5 ruby > simple mongrel setup with RHEL5. > > for fun, I also added a yaml/marshal comparison. > > my test client was always the same with this arguments: > /puppet-test --server serverX --fork 5 --catalog_format marshal/yaml > --repeat 10 > > according to my calculation 60 (1+5 fork x 10) clients is at least 60 > clients per minute ==> 1800 clients per puppetmaster. > > > my output is: > passenger + ree + marshal: > AVG: 42.4674576271186, OK: 59 failed: 1 > passenger + ree + yaml: > AVG: 46.9481666666667, OK: 60 failed: 0 > > passenger + RHE5 ruby / marshal: > AVG: 50.6046666666667, OK: 60 failed: 0 > passenger + RHE5 ruby / yaml: > AVG: 52.1456666666667, OK: 60 failed: 0 > > mongrel (4 processes) + with RHE5 ruby / marshal: > AVG: 48.5047272727273, OK: 55 failed: 5 > mongrel (4 processes) + with RHE5 ruby / yaml: > AVG: 38.0254347826087, OK: 46 failed: 14 > > > as it goes for memory consumptions, no surprises here, passenger processes > retain within 100mb per process, mongrel went to an average of 220mb, > however, passenger used 6 processes, and mongrel only 4.. so maybe I could > run 6 mongrels and compare further. > > I did not see a huge different in memory consumption between REE and normal > RHE5 ruby when using passenger. > > *in my setup, failed are clients who didnt get a catalog within 120 seconds. > > Nigel, I''m surpized how you got such a different values for compile time, > could it be that you server was swapping?No. 8GB of RAM, plus I was actually stopping all services, starting again, doing a single run to cache client information, then starting the benchmark. I''m doing fork 50, repeat 10. From prior benchmarking that showed Passenger far far ahead of Mongrel for concurrent client runs, I haven''t even bothered testing that, but I could. I''ll do a run with the same environments today, and repeat this a few more times. Perhaps the difference is more Ruby 1.8.4 and 1.8.6 than the EE patches? I''ll post the versions of everything else later, but this was with Passenger 2.1.1 and puppet 0.24.8 I haven''t actually been able to get 2.1.2 working on Dapper, so I can''t reproduce the same environment.> > another side effect that I found in more than one occasion that if I killed > the puppet-test, the passenger processes went on for a long time ( I killed > them manually after 15 minutes) and passenger-status reported that there are > no active processes but cpu usage was 100%..... > > so bottom line so far, even if you have a balanced mongrel / cpu /client > ratio, performance-wise there is a slight difference towards ree (in my > tests), passenger is handling memory usage better than mongrels, but maybe > have some hiccups and handles high load much better than mongrels... > > Ohad > > > On Tue, May 12, 2009 at 9:35 AM, Nigel Kersten <nigelk@google.com> wrote: >> >> This was: >> >> ruby 1.8.4 (2005-12-24) [x86_64-linux] >> vs >> ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] >> Ruby Enterprise Edition 20090421 >> >> on Ubuntu Dapper, using Passenger with Apache. >> >> I used the puppet-test script in the ext/ directory to simulate 50 >> simultaneous runs with a repeat of 10. Both runs ran from the same >> clean state. >> >> I measured peak memory usage and total catalog compilation time (ie >> adding up the "compiled catalog in xxx.x seconds" output) with our >> actual Puppet configuration. >> >> >> Ruby 1.8.4: >> peak memory consumption: 2686 MB >> total catalog compilation time: 26632 seconds >> >> >> Ruby EE 20090421: >> peak memory consumption: 1462 MB >> total catalog compilation time: 13693.5 seconds >> >> >> wow. >> >> >> -- >> Nigel Kersten >> nigelk@google.com >> System Administrator >> Google, Inc. >> >> > > > > >-- Nigel Kersten nigelk@google.com System Administrator Google, Inc. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
which passenger settings did you use for the testing? Ohad On 5/12/09, Nigel Kersten <nigelk@google.com> wrote:> > On Tue, May 12, 2009 at 2:05 AM, Ohad Levy <ohadlevy@gmail.com> wrote: >> I''ve been trying to reproduce your results, and got mixed results myself. >> >> I did a similar test with puppet-test (ignoring for a second that a major >> memory abuser is the file copying) i did simple compile tests. >> >> the setup include 3 main scenarios: >> >> latest passenger + ree >> passenger with RHEL5 ruby >> simple mongrel setup with RHEL5. >> >> for fun, I also added a yaml/marshal comparison. >> >> my test client was always the same with this arguments: >> /puppet-test --server serverX --fork 5 --catalog_format marshal/yaml >> --repeat 10 >> >> according to my calculation 60 (1+5 fork x 10) clients is at least 60 >> clients per minute ==> 1800 clients per puppetmaster. >> >> >> my output is: >> passenger + ree + marshal: >> AVG: 42.4674576271186, OK: 59 failed: 1 >> passenger + ree + yaml: >> AVG: 46.9481666666667, OK: 60 failed: 0 >> >> passenger + RHE5 ruby / marshal: >> AVG: 50.6046666666667, OK: 60 failed: 0 >> passenger + RHE5 ruby / yaml: >> AVG: 52.1456666666667, OK: 60 failed: 0 >> >> mongrel (4 processes) + with RHE5 ruby / marshal: >> AVG: 48.5047272727273, OK: 55 failed: 5 >> mongrel (4 processes) + with RHE5 ruby / yaml: >> AVG: 38.0254347826087, OK: 46 failed: 14 >> >> >> as it goes for memory consumptions, no surprises here, passenger processes >> retain within 100mb per process, mongrel went to an average of 220mb, >> however, passenger used 6 processes, and mongrel only 4.. so maybe I could >> run 6 mongrels and compare further. >> >> I did not see a huge different in memory consumption between REE and >> normal >> RHE5 ruby when using passenger. >> >> *in my setup, failed are clients who didnt get a catalog within 120 >> seconds. >> >> Nigel, I''m surpized how you got such a different values for compile time, >> could it be that you server was swapping? > > No. 8GB of RAM, plus I was actually stopping all services, starting > again, doing a single run to cache client information, then starting > the benchmark. > > I''m doing fork 50, repeat 10. > > From prior benchmarking that showed Passenger far far ahead of Mongrel > for concurrent client runs, I haven''t even bothered testing that, but > I could. I''ll do a run with the same environments today, and repeat > this a few more times. > > Perhaps the difference is more Ruby 1.8.4 and 1.8.6 than the EE > patches? I''ll post the versions of everything else later, but this was > with Passenger 2.1.1 and puppet 0.24.8 > > I haven''t actually been able to get 2.1.2 working on Dapper, so I > can''t reproduce the same environment. > >> >> another side effect that I found in more than one occasion that if I >> killed >> the puppet-test, the passenger processes went on for a long time ( I >> killed >> them manually after 15 minutes) and passenger-status reported that there >> are >> no active processes but cpu usage was 100%..... >> >> so bottom line so far, even if you have a balanced mongrel / cpu /client >> ratio, performance-wise there is a slight difference towards ree (in my >> tests), passenger is handling memory usage better than mongrels, but maybe >> have some hiccups and handles high load much better than mongrels... >> >> Ohad >> >> >> On Tue, May 12, 2009 at 9:35 AM, Nigel Kersten <nigelk@google.com> wrote: >>> >>> This was: >>> >>> ruby 1.8.4 (2005-12-24) [x86_64-linux] >>> vs >>> ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] >>> Ruby Enterprise Edition 20090421 >>> >>> on Ubuntu Dapper, using Passenger with Apache. >>> >>> I used the puppet-test script in the ext/ directory to simulate 50 >>> simultaneous runs with a repeat of 10. Both runs ran from the same >>> clean state. >>> >>> I measured peak memory usage and total catalog compilation time (ie >>> adding up the "compiled catalog in xxx.x seconds" output) with our >>> actual Puppet configuration. >>> >>> >>> Ruby 1.8.4: >>> peak memory consumption: 2686 MB >>> total catalog compilation time: 26632 seconds >>> >>> >>> Ruby EE 20090421: >>> peak memory consumption: 1462 MB >>> total catalog compilation time: 13693.5 seconds >>> >>> >>> wow. >>> >>> >>> -- >>> Nigel Kersten >>> nigelk@google.com >>> System Administrator >>> Google, Inc. >>> >>> >> >> >> > >> > > > > -- > Nigel Kersten > nigelk@google.com > System Administrator > Google, Inc. > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, May 12, 2009 at 6:58 AM, Ohad Levy <ohadlevy@gmail.com> wrote:> > which passenger settings did you use for the testing?Ruby 1.8.4 PassengerRoot /usr PassengerRuby /usr/bin/ruby1.8 PassengerMaxRequests 3000 PassengerMaxPoolSize 15 PassengerStatThrottleRate 600 <VirtualHost *:9140> RackAutoDetect On </Directory> Ruby EE: PassengerRoot /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/passenger-2.2.2 PassengerRuby /opt/ruby-enterprise-1.8.6-20090421/bin/ruby PassengerMaxRequests 3000 PassengerMaxPoolSize 15 PassengerStatThrottleRate 600 <VirtualHost *:9140> RackAutoDetect On </Directory> Actually, I''ve just realized the Ruby EE passenger install was 2.2.2 whereas the other was 2.2.1. It''s possible that was the cause of the performance boost, I''ll try to flip over to the 2.2.2 gem on Ruby 1.8.4 today, but I have a feeling that 1.8.4 doesn''t work with 2.2.2.> > Ohad > > On 5/12/09, Nigel Kersten <nigelk@google.com> wrote: >> >> On Tue, May 12, 2009 at 2:05 AM, Ohad Levy <ohadlevy@gmail.com> wrote: >>> I''ve been trying to reproduce your results, and got mixed results myself. >>> >>> I did a similar test with puppet-test (ignoring for a second that a major >>> memory abuser is the file copying) i did simple compile tests. >>> >>> the setup include 3 main scenarios: >>> >>> latest passenger + ree >>> passenger with RHEL5 ruby >>> simple mongrel setup with RHEL5. >>> >>> for fun, I also added a yaml/marshal comparison. >>> >>> my test client was always the same with this arguments: >>> /puppet-test --server serverX --fork 5 --catalog_format marshal/yaml >>> --repeat 10 >>> >>> according to my calculation 60 (1+5 fork x 10) clients is at least 60 >>> clients per minute ==> 1800 clients per puppetmaster. >>> >>> >>> my output is: >>> passenger + ree + marshal: >>> AVG: 42.4674576271186, OK: 59 failed: 1 >>> passenger + ree + yaml: >>> AVG: 46.9481666666667, OK: 60 failed: 0 >>> >>> passenger + RHE5 ruby / marshal: >>> AVG: 50.6046666666667, OK: 60 failed: 0 >>> passenger + RHE5 ruby / yaml: >>> AVG: 52.1456666666667, OK: 60 failed: 0 >>> >>> mongrel (4 processes) + with RHE5 ruby / marshal: >>> AVG: 48.5047272727273, OK: 55 failed: 5 >>> mongrel (4 processes) + with RHE5 ruby / yaml: >>> AVG: 38.0254347826087, OK: 46 failed: 14 >>> >>> >>> as it goes for memory consumptions, no surprises here, passenger processes >>> retain within 100mb per process, mongrel went to an average of 220mb, >>> however, passenger used 6 processes, and mongrel only 4.. so maybe I could >>> run 6 mongrels and compare further. >>> >>> I did not see a huge different in memory consumption between REE and >>> normal >>> RHE5 ruby when using passenger. >>> >>> *in my setup, failed are clients who didnt get a catalog within 120 >>> seconds. >>> >>> Nigel, I''m surpized how you got such a different values for compile time, >>> could it be that you server was swapping? >> >> No. 8GB of RAM, plus I was actually stopping all services, starting >> again, doing a single run to cache client information, then starting >> the benchmark. >> >> I''m doing fork 50, repeat 10. >> >> From prior benchmarking that showed Passenger far far ahead of Mongrel >> for concurrent client runs, I haven''t even bothered testing that, but >> I could. I''ll do a run with the same environments today, and repeat >> this a few more times. >> >> Perhaps the difference is more Ruby 1.8.4 and 1.8.6 than the EE >> patches? I''ll post the versions of everything else later, but this was >> with Passenger 2.1.1 and puppet 0.24.8 >> >> I haven''t actually been able to get 2.1.2 working on Dapper, so I >> can''t reproduce the same environment. >> >>> >>> another side effect that I found in more than one occasion that if I >>> killed >>> the puppet-test, the passenger processes went on for a long time ( I >>> killed >>> them manually after 15 minutes) and passenger-status reported that there >>> are >>> no active processes but cpu usage was 100%..... >>> >>> so bottom line so far, even if you have a balanced mongrel / cpu /client >>> ratio, performance-wise there is a slight difference towards ree (in my >>> tests), passenger is handling memory usage better than mongrels, but maybe >>> have some hiccups and handles high load much better than mongrels... >>> >>> Ohad >>> >>> >>> On Tue, May 12, 2009 at 9:35 AM, Nigel Kersten <nigelk@google.com> wrote: >>>> >>>> This was: >>>> >>>> ruby 1.8.4 (2005-12-24) [x86_64-linux] >>>> vs >>>> ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] >>>> Ruby Enterprise Edition 20090421 >>>> >>>> on Ubuntu Dapper, using Passenger with Apache. >>>> >>>> I used the puppet-test script in the ext/ directory to simulate 50 >>>> simultaneous runs with a repeat of 10. Both runs ran from the same >>>> clean state. >>>> >>>> I measured peak memory usage and total catalog compilation time (ie >>>> adding up the "compiled catalog in xxx.x seconds" output) with our >>>> actual Puppet configuration. >>>> >>>> >>>> Ruby 1.8.4: >>>> peak memory consumption: 2686 MB >>>> total catalog compilation time: 26632 seconds >>>> >>>> >>>> Ruby EE 20090421: >>>> peak memory consumption: 1462 MB >>>> total catalog compilation time: 13693.5 seconds >>>> >>>> >>>> wow. >>>> >>>> >>>> -- >>>> Nigel Kersten >>>> nigelk@google.com >>>> System Administrator >>>> Google, Inc. >>>> >>>> >>> >>> >>> > >>> >> >> >> >> -- >> Nigel Kersten >> nigelk@google.com >> System Administrator >> Google, Inc. >> >> > >> > > > >-- Nigel Kersten nigelk@google.com System Administrator Google, Inc. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---