I use, and depend on, puppet extensively and a good portion of the servers (nodes) are remote, across a WAN. I just started a WAN Optimization trial with some Juniper gear, one of the components of WAN optimization is compression and the puppet client/server communication is compressing pretty well. According to the Juniper reporting the puppet data is being compressed between 25-50%. Is puppet using compression between client and server? If it is not, has it been considered? Considering most of the data exchanged between client and server would be text in the form of manifests and config files i think it could achieve good compression with low overhead. I have nginx in front of my puppetmasters, if i enable gzip compression there will the puppet client still work? Any thoughts? Ben --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Larry Ludwig
2009-Mar-26 01:43 UTC
[Puppet Users] Re: compressing client/server communications
Hmm interesting idea. While the puppetmaster will work without issue, the issue becomes the client (puppetd) must decode it. I suspect a code change. -L On Mar 25, 2009, at 8:56 PM, Ben wrote:> > I use, and depend on, puppet extensively and a good portion of the > servers (nodes) are remote, across a WAN. > > I just started a WAN Optimization trial with some Juniper gear, one of > the components of WAN optimization is compression and the puppet > client/server communication is compressing pretty well. According to > the Juniper reporting the puppet data is being compressed between > 25-50%. > > > Is puppet using compression between client and server? > > If it is not, has it been considered? Considering most of the data > exchanged between client and server would be text in the form of > manifests and config files i think it could achieve good compression > with low overhead. > > I have nginx in front of my puppetmasters, if i enable gzip > compression > there will the puppet client still work? > > Any thoughts? > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Larry Ludwig wrote:> Hmm interesting idea. > > While the puppetmaster will work without issue, the issue becomes the > client (puppetd) must decode it. > > I suspect a code change. > > -L > > > On Mar 25, 2009, at 8:56 PM, Ben wrote: > > >> I just started a WAN Optimization trial with some Juniper gear, one of >> the components of WAN optimization is compression and the puppet >> client/server communication is compressing pretty well. According to >> the Juniper reporting the puppet data is being compressed between >> 25-50%. >> >> >> Is puppet using compression between client and server? >> >> If it is not, has it been considered? Considering most of the data >> exchanged between client and server would be text in the form of >> manifests and config files i think it could achieve good compression >> with low overhead. >> >> I have nginx in front of my puppetmasters, if i enable gzip >> compression >> there will the puppet client still work? >>If you were going to make code changes in the client you may as well implement compression into puppet itself as a whole. In other words do not implement it at the "nginx" level for example but have puppetmaster and fileserver compress it''s data using Zlib::GzipWriter and Zlib::GzipReader before serving it to the client and the client can use the same to do the reverse. This way a simple puppet.conf option would enable or disable compression in any deployment even without a more complicated configuration using "nginx", "apache" or whatever. Ben --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy
2009-Mar-26 06:54 UTC
[Puppet Users] Re: compressing client/server communications
I would suggest to have an option to enable/disable it per client, as my puppetmasters are low on cpu...(e.g. too many clients ;)) did you file a feature request for it? On Thu, Mar 26, 2009 at 2:21 PM, Ben <abnormaliti@clivepeeters.com.au>wrote:> > Larry Ludwig wrote: > > Hmm interesting idea. > > > > While the puppetmaster will work without issue, the issue becomes the > > client (puppetd) must decode it. > > > > I suspect a code change. > > > > -L > > > > > > On Mar 25, 2009, at 8:56 PM, Ben wrote: > > > > > >> I just started a WAN Optimization trial with some Juniper gear, one of > >> the components of WAN optimization is compression and the puppet > >> client/server communication is compressing pretty well. According to > >> the Juniper reporting the puppet data is being compressed between > >> 25-50%. > >> > >> > >> Is puppet using compression between client and server? > >> > >> If it is not, has it been considered? Considering most of the data > >> exchanged between client and server would be text in the form of > >> manifests and config files i think it could achieve good compression > >> with low overhead. > >> > >> I have nginx in front of my puppetmasters, if i enable gzip > >> compression > >> there will the puppet client still work? > >> > > > If you were going to make code changes in the client you may as well > implement compression into puppet itself as a whole. In other words do > not implement it at the "nginx" level for example but have puppetmaster > and fileserver compress it''s data using Zlib::GzipWriter and > Zlib::GzipReader before serving it to the client and the client can use > the same to do the reverse. > > This way a simple puppet.conf option would enable or disable compression > in any deployment even without a more complicated configuration using > "nginx", "apache" or whatever. > > Ben > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Jason Rojas
2009-Mar-26 07:37 UTC
[Puppet Users] Re: compressing client/server communications
I agree, my puppetmaster running behind passenger averages 40requests per second, and I am serving a lot of files with no splay. So being able to adjust accordingly for servers that get more files might be a good idea. Maybe even an option to just use gzip for file serving instead of normal catalog requests. -Jason On Mar 25, 2009, at 11:54 PM, Ohad Levy <ohadlevy@gmail.com> wrote:> > I would suggest to have an option to enable/disable it per client, > as my puppetmasters are low on cpu...(e.g. too many clients ;)) > did you file a feature request for it? > > On Thu, Mar 26, 2009 at 2:21 PM, Ben > <abnormaliti@clivepeeters.com.au> wrote: > > Larry Ludwig wrote: > > Hmm interesting idea. > > > > While the puppetmaster will work without issue, the issue becomes > the > > client (puppetd) must decode it. > > > > I suspect a code change. > > > > -L > > > > > > On Mar 25, 2009, at 8:56 PM, Ben wrote: > > > > > >> I just started a WAN Optimization trial with some Juniper gear, > one of > >> the components of WAN optimization is compression and the puppet > >> client/server communication is compressing pretty well. > According to > >> the Juniper reporting the puppet data is being compressed between > >> 25-50%. > >> > >> > >> Is puppet using compression between client and server? > >> > >> If it is not, has it been considered? Considering most of the data > >> exchanged between client and server would be text in the form of > >> manifests and config files i think it could achieve good > compression > >> with low overhead. > >> > >> I have nginx in front of my puppetmasters, if i enable gzip > >> compression > >> there will the puppet client still work? > >> > > > If you were going to make code changes in the client you may as well > implement compression into puppet itself as a whole. In other words > do > not implement it at the "nginx" level for example but have > puppetmaster > and fileserver compress it''s data using Zlib::GzipWriter and > Zlib::GzipReader before serving it to the client and the client can > use > the same to do the reverse. > > This way a simple puppet.conf option would enable or disable > compression > in any deployment even without a more complicated configuration using > "nginx", "apache" or whatever. > > Ben > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Brice Figureau
2009-Mar-26 09:54 UTC
[Puppet Users] Re: compressing client/server communications
On 26/03/09 1:56, Ben wrote:> I use, and depend on, puppet extensively and a good portion of the > servers (nodes) are remote, across a WAN. > > I just started a WAN Optimization trial with some Juniper gear, one of > the components of WAN optimization is compression and the puppet > client/server communication is compressing pretty well. According to > the Juniper reporting the puppet data is being compressed between 25-50%. > > > Is puppet using compression between client and server?No, see below.> If it is not, has it been considered? Considering most of the data > exchanged between client and server would be text in the form of > manifests and config files i think it could achieve good compression > with low overhead. > > I have nginx in front of my puppetmasters, if i enable gzip compression > there will the puppet client still work?Puppet uses Net::HTTP on the client to connect to the puppetmaster. This is a part of the standard Ruby lib, and only in version 1.9 did they add content encoding Gzip support: http://redmine.ruby-lang.org/repositories/revision/ruby-19?rev=14187 So if you run your clients with ruby 1.9, then I think you can activate content-encoding compression in nginx. I doubt Luke would agree implementing the compression in the puppet layer of the protocol, because all the transfer use HTTP (especially when we''ll move to a full REST system) and HTTP supports content-encoding since at least a decade. Now one solution to have compression in the client while running an older ruby interpreter version would be to dump Net::HTTP which is horribly slow and start using something "newer" (or simply better designed). -- Brice Figureau Days of Wonder http://www.daysofwonder.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy wrote:> > I would suggest to have an option to enable/disable it per client, as > my puppetmasters are low on cpu...(e.g. too many clients ;)) > did you file a feature request for it? > > On Thu, Mar 26, 2009 at 2:21 PM, Ben <abnormaliti@clivepeeters.com.au > <mailto:abnormaliti@clivepeeters.com.au>> wrote: > > > Larry Ludwig wrote: > > Hmm interesting idea. > > > > While the puppetmaster will work without issue, the issue > becomes the > > client (puppetd) must decode it. > > > > I suspect a code change. > > > > -L > > > > > > On Mar 25, 2009, at 8:56 PM, Ben wrote: > > > > > >> I just started a WAN Optimization trial with some Juniper gear, > one of > >> the components of WAN optimization is compression and the puppet > >> client/server communication is compressing pretty well. > According to > >> the Juniper reporting the puppet data is being compressed between > >> 25-50%. > >> > >> > >> Is puppet using compression between client and server? > >> > >> If it is not, has it been considered? Considering most of the data > >> exchanged between client and server would be text in the form of > >> manifests and config files i think it could achieve good > compression > >> with low overhead. > >> > >> I have nginx in front of my puppetmasters, if i enable gzip > >> compression > >> there will the puppet client still work? > >> > > > If you were going to make code changes in the client you may as well > implement compression into puppet itself as a whole. In other > words do > not implement it at the "nginx" level for example but have > puppetmaster > and fileserver compress it''s data using Zlib::GzipWriter and > Zlib::GzipReader before serving it to the client and the client > can use > the same to do the reverse. > > This way a simple puppet.conf option would enable or disable > compression > in any deployment even without a more complicated configuration using > "nginx", "apache" or whatever. > > Ben >I agree, a per client setting would allow LAN based clients to have it disabled, as there would be little benefit, and WAN based clients get it enabled. Ben --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Brice Figureau wrote:> On 26/03/09 1:56, Ben wrote: > >> I use, and depend on, puppet extensively and a good portion of the >> servers (nodes) are remote, across a WAN. >> >> I just started a WAN Optimization trial with some Juniper gear, one of >> the components of WAN optimization is compression and the puppet >> client/server communication is compressing pretty well. According to >> the Juniper reporting the puppet data is being compressed between 25-50%. >> >> >> Is puppet using compression between client and server? >> > > No, see below. > > >> If it is not, has it been considered? Considering most of the data >> exchanged between client and server would be text in the form of >> manifests and config files i think it could achieve good compression >> with low overhead. >> >> I have nginx in front of my puppetmasters, if i enable gzip compression >> there will the puppet client still work? >> > > Puppet uses Net::HTTP on the client to connect to the puppetmaster. This > is a part of the standard Ruby lib, and only in version 1.9 did they add > content encoding Gzip support: > http://redmine.ruby-lang.org/repositories/revision/ruby-19?rev=14187 > > So if you run your clients with ruby 1.9, then I think you can activate > content-encoding compression in nginx. > > I doubt Luke would agree implementing the compression in the puppet > layer of the protocol, because all the transfer use HTTP (especially > when we''ll move to a full REST system) and HTTP supports > content-encoding since at least a decade. > Now one solution to have compression in the client while running an > older ruby interpreter version would be to dump Net::HTTP which is > horribly slow and start using something "newer" (or simply better designed). >So are you saying that clients running ruby 1.9 would support HTTP compression now without modifications to puppet? All my clients are RHEL and CentOS so i do not see ruby 1.9 being available for them until the next major release, i.e. 6. A solution before then would be helpful. Ben --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Kyle Cordes
2009-Mar-26 23:05 UTC
[Puppet Users] Re: compressing client/server communications
Ben wrote:> All my clients are RHEL and CentOS so i do not see ruby 1.9 being > available for them until the next major release, i.e. 6. A solution > before then would be helpful.I wish there was an option to install Puppet, along with Ruby and other dependencies, standalone in some directory out of the way of the usual package management. Since I might use Puppet to get various other artifacts in place (even those that Puppet depends on, I might want to remove, install, or upgrade Ruby!), ideally Puppet would be quite low in the dependency chain. Perhaps the way to most thoroughly achieve that would be to write it in C, and static-link in all its libraries; but including the dependencies in roughly there current form would yield most of the same benefits yet keep the high productivity of Ruby. Although I wish all that, I don''t wish it badly enough to go make it so myself! :-) -- Kyle Cordes http://kylecordes.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2009-Mar-31 03:24 UTC
[Puppet Users] Re: compressing client/server communications
On Mar 26, 2009, at 4:54 AM, Brice Figureau wrote:> > On 26/03/09 1:56, Ben wrote: >> I use, and depend on, puppet extensively and a good portion of the >> servers (nodes) are remote, across a WAN. >> >> I just started a WAN Optimization trial with some Juniper gear, one >> of >> the components of WAN optimization is compression and the puppet >> client/server communication is compressing pretty well. According to >> the Juniper reporting the puppet data is being compressed between >> 25-50%. >> >> >> Is puppet using compression between client and server? > > No, see below. > >> If it is not, has it been considered? Considering most of the data >> exchanged between client and server would be text in the form of >> manifests and config files i think it could achieve good compression >> with low overhead. >> >> I have nginx in front of my puppetmasters, if i enable gzip >> compression >> there will the puppet client still work? > > Puppet uses Net::HTTP on the client to connect to the puppetmaster. > This > is a part of the standard Ruby lib, and only in version 1.9 did they > add > content encoding Gzip support: > http://redmine.ruby-lang.org/repositories/revision/ruby-19?rev=14187 > > So if you run your clients with ruby 1.9, then I think you can > activate > content-encoding compression in nginx. > > I doubt Luke would agree implementing the compression in the puppet > layer of the protocol, because all the transfer use HTTP (especially > when we''ll move to a full REST system) and HTTP supports > content-encoding since at least a decade. > Now one solution to have compression in the client while running an > older ruby interpreter version would be to dump Net::HTTP which is > horribly slow and start using something "newer" (or simply better > designed).I''m fine with a boolean to enable/disable compression, and it should be able to be done just like browsers, which can specify that they support compression so that the server knows whether it should use it, and even then it''s optional. It''d certainly be nice if the HTTP libraries handled all of this for us, of course. -- Getting caught is the mother of invention. --Robert Byrne --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---