Hi all, According to some work done by Google, up to 70% of the compile time on the server is spent converting the catalog to yaml. In addition, yaml is drastically slower than Marshal, which also ships with Ruby. (This is a bit ironic, considering I switched from Marshal to Yaml a couple of years ago because I thought it made it more "standard".) Considering that the server is usually the performance bottleneck, and it''s really very easy to do, I made a quick change that enabled Marshal as an option: sudo puppetd --test --config_format marshal Yes, now it''s configurable, and the default is still yaml so you shouldn''t see any behaviour change by default. The configuration is entirely client-side; the server just returns whatever the client asks for. Please test this out (current HEAD in 0.24.x), especially if you are hitting your server pretty hard. I''d really like to see if this makes a difference, and, more importantly, I''d like to make sure there aren''t any problems. -- A bore is a man who deprives you of solitude without providing you with company. -- Gian Vincenzo Gravina --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
If we''re rocking config formats, JSON will also be significantly faster. Adam On Fri, Jul 18, 2008 at 8:37 AM, Luke Kanies <luke@madstop.com> wrote:> > Hi all, > > According to some work done by Google, up to 70% of the compile time > on the server is spent converting the catalog to yaml. In addition, > yaml is drastically slower than Marshal, which also ships with Ruby. > > (This is a bit ironic, considering I switched from Marshal to Yaml a > couple of years ago because I thought it made it more "standard".) > > Considering that the server is usually the performance bottleneck, and > it''s really very easy to do, I made a quick change that enabled > Marshal as an option: > > sudo puppetd --test --config_format marshal > > Yes, now it''s configurable, and the default is still yaml so you > shouldn''t see any behaviour change by default. The configuration is > entirely client-side; the server just returns whatever the client asks > for. > > Please test this out (current HEAD in 0.24.x), especially if you are > hitting your server pretty hard. I''d really like to see if this makes > a difference, and, more importantly, I''d like to make sure there > aren''t any problems. > > -- > A bore is a man who deprives you of solitude without providing you > with company. -- Gian Vincenzo Gravina > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > > >-- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2008/7/18 Luke Kanies <luke@madstop.com>:> > Hi all, >> Considering that the server is usually the performance bottleneck, and > it''s really very easy to do, I made a quick change that enabled > Marshal as an option: > > sudo puppetd --test --config_format marshal > > Yes, now it''s configurable, and the default is still yaml so you > shouldn''t see any behaviour change by default. The configuration is > entirely client-side; the server just returns whatever the client asks > for. > > Please test this out (current HEAD in 0.24.x), especially if you are > hitting your server pretty hard. I''d really like to see if this makes > a difference, and, more importantly, I''d like to make sure there > aren''t any problems. >Running the RC3 code with catalog_format=marshal we''re seeing some fairly major improvements in catalog run times; at least 3-5 secs per run (or about 1/4 of the total run time) usually. The change doesn''t appear to have any unexpected side effects thus far. Cheers, -Thom --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jul 22, 2008 at 6:55 AM, Thom May <thom@joost.com> wrote:> > 2008/7/18 Luke Kanies <luke@madstop.com>: >> >> Hi all, >> > >> Considering that the server is usually the performance bottleneck, and >> it''s really very easy to do, I made a quick change that enabled >> Marshal as an option: >> >> sudo puppetd --test --config_format marshalAs Thom pointed out, it''s --catalog_format marshal . So I just ran some tests, and no ill side effects so far, and the difference in a client run is about 3 seconds on my test server. The difference server side is much more striking however, I patched some benchmark lines around the code, and check this out.... Time to return CGI escaped yaml catalog: 1.96 seconds Time to return CGI escaped marshal catalog: 0.17 seconds These times varied very little over > 10 runs. Nice work... This should make a reasonably significant difference under heavy load in production. -- Nigel Kersten Systems Administrator Tech Lead - MacOps --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Jul 18, 2008, at 5:17 PM, Adam Jacob wrote:> If we''re rocking config formats, JSON will also be significantly > faster.(Sorry for the late reply.) JSON also doesn''t ship with Ruby, so I can''t depend on it being there, which is not the case for Marshal. And, somewhat more importantly, JSON requires me to build per-class render/deserialize methods. IMO, this is a feature, but it''s also a lot more work. The intention with this change was to make a hopefully-big hit for very little effort, which JSON decidedly would not do. I''ve already pushed the code to master that will enable other formats, including JSON, to be supported in the future, but it will be done on a per-class basis (e.g., there''s no need for it for certificates). Lastly, from the tests I''ve seen, JSON and Marshal are *far* closer to each other than they are to yaml, so it''s still a big win. -- There is no expedient to which a man will not go to avoid the labor of thinking. --Thomas A. Edison --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mon, Aug 4, 2008 at 5:06 PM, Luke Kanies <luke@madstop.com> wrote:> JSON also doesn''t ship with Ruby, so I can''t depend on it being there, > which is not the case for Marshal.I''m not sure about Ruby, but I know that using the Perl equivilant here (Storable) is much faster, but is liable to introduce binary incompatability between Perl versions. I imagine the same is true with Marshal, although I don''t know how often the Major/Minor version numbers change. Not a reason not to use Marshal, but something people should be aware of if they start using it in wildly mixed environments, or if they upgrade between Ruby major versions. (The Marshal version changes independently of Ruby, but I don''t know how often.) Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---