I just talked to Mike McGrath from Fedora Infrastructure, and he told me that they are seeing load spikes (not quite performance problems yet, but definitely a concern) in their setup. As an example, the graph [1] shows a typical client - the spikes from 16:00 to 8:00 are almost exclusively puppetd doing its thing. It seems that the most likely culprit is the fileserver - they serve 500-1500 files to a client, and some dirs with recursion. The files aren''t big (typical config files) Does anybody have suggestions on how client performance can be improved with current puppet, i.e. without adding new file serving backends or hacking up the existing one ? They also rsync some files, but can''t do that for a lot of things, becuase they need to trigger service restarts in a lot of cases. Does anybody have a clever exec for rsync that can be used in this way ? (I don''t think it''s possible though) David [1] https://admin.fedoraproject.org/cacti/graph.php?action=view&rra_id=all&local_graph_id=75
On 6/19/07 12:25 PM, "David Lutterkort" <dlutter@redhat.com> wrote:> I just talked to Mike McGrath from Fedora Infrastructure, and he told me > that they are seeing load spikes (not quite performance problems yet, > but definitely a concern) in their setup. As an example, the graph [1] > shows a typical client - the spikes from 16:00 to 8:00 are almost > exclusively puppetd doing its thing. > > It seems that the most likely culprit is the fileserver - they serve > 500-1500 files to a client, and some dirs with recursion. The files > aren''t big (typical config files) > > Does anybody have suggestions on how client performance can be improved > with current puppet, i.e. without adding new file serving backends or > hacking up the existing one ?I''ve also some interesting behavior as well. In a recent, um, accident, all the ssl certs were blown away from the puppetmaster so all of our clients were told to reinitialize. We have a limited number of hosts currently but they seem to trigger synflooding protection (for the ipv6 interface) on the puppetmaster. Anybody else who uses puppet at a large scale seen this behavior? Cheers, Ryan
> I''ve also some interesting behavior as well. In a recent, um, accident, all > the ssl certs were blown away from the puppetmaster so all of our clients > were told to reinitialize. We have a limited number of hosts currently but > they seem to trigger synflooding protection (for the ipv6 interface) on the > puppetmaster. > > Anybody else who uses puppet at a large scale seen this behavior?So, go linux... The ipv6 flooding protection seems to be shared with the ipv4 limits. After upping both ipv6 and ipv4 icmp.ratelimits, I appear to be in better shape. Cheers, Ryan
On Tue, Jun 19, 2007 at 12:25:35PM -0700, David Lutterkort wrote:> I just talked to Mike McGrath from Fedora Infrastructure, and he told me > that they are seeing load spikes (not quite performance problems yet, > but definitely a concern) in their setup. As an example, the graph [1] > shows a typical client - the spikes from 16:00 to 8:00 are almost > exclusively puppetd doing its thing. > > It seems that the most likely culprit is the fileserver - they serve > 500-1500 files to a client, and some dirs with recursion. The files > aren''t big (typical config files) > > Does anybody have suggestions on how client performance can be improved > with current puppet, i.e. without adding new file serving backends or > hacking up the existing one ?Looking at the network traffic it looks like there is a lot of traffic at the same time as the load/cpu spikes. It does look to me that they are downloading everything (8mbit/sec average for 5mins = 300MB!). Kostas
On Jun 19, 2007, at 2:25 PM, David Lutterkort wrote:> I just talked to Mike McGrath from Fedora Infrastructure, and he > told me > that they are seeing load spikes (not quite performance problems yet, > but definitely a concern) in their setup. As an example, the graph [1] > shows a typical client - the spikes from 16:00 to 8:00 are almost > exclusively puppetd doing its thing. > > It seems that the most likely culprit is the fileserver - they serve > 500-1500 files to a client, and some dirs with recursion. The files > aren''t big (typical config files)Yay. :/> Does anybody have suggestions on how client performance can be > improved > with current puppet, i.e. without adding new file serving backends or > hacking up the existing one ?I can''t think of anything that doesn''t involve at least some modifications to the current system. Since (according to mmcgrath on irc) most of these file interactions don''t actually result in files being copied, it''s the md5 that''s killing the system. One thing we can do is fix the file server so it supports checksums other than full md5; it''s currently broken in that way. Then people could use timestamps or md5lite, which should help a lot. londo on IRC is also seeing if maybe there''s a more efficient way to calculate the md5 sum. Also, I''ve realized I''m not caching these values at all, which I''m embarrassed to realize and will definitely be fixed in the next release. I expect that will have a huge impact and be easy to do. (For the record, I have a big note right where I''ll be doing this, saying there should be caching.)> They also rsync some files, but can''t do that for a lot of things, > becuase they need to trigger service restarts in a lot of cases. Does > anybody have a clever exec for rsync that can be used in this way ? (I > don''t think it''s possible though)I expect caching the md5 sum plus a better md5 algorithm will be wicked easy and have a huge impact (especially the caching). Beyond that, I think the next step is supporting non-md5 summing, and I''ll see if that''s simple enough to slip into this release. -- The first time I see a jogger smiling, I''ll consider it. --Joan Rivers --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> Looking at the network traffic it looks like there is a lot of traffic > at the same time as the load/cpu spikes. It does look to me that they > are downloading everything (8mbit/sec average for 5mins = 300MB!). >wow, I''d be happy to reach that kind of performance on the distant hosts in our network... but on high latency links (300-500ms RTT) this is impossible. It literally takes 2 secs to retrieve a single file - for each and every file. Does puppet pipelines requests? If it doesn''t perhaps pipelining would help... it should also help with the syn flooding issue, I suppose. kind regards Thilo _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Wed, Jun 20, 2007 at 09:27:58AM +0200, Thilo Bangert wrote:> > Looking at the network traffic it looks like there is a lot of traffic > > at the same time as the load/cpu spikes. It does look to me that they > > are downloading everything (8mbit/sec average for 5mins = 300MB!). > > > > wow, I''d be happy to reach that kind of performance on the distant hosts > in our network... but on high latency links (300-500ms RTT) this is > impossible. It literally takes 2 secs to retrieve a single file - for > each and every file. > > Does puppet pipelines requests? If it doesn''t perhaps pipelining would > help... it should also help with the syn flooding issue, I suppose.Well after some discsussion in irc and after looking at the plots for the tenth time I noticed that the load/cpu/network spikes are every hour and so they are not puppet related. There is no pipelining AFAIK, probably the easier way to speed things up at the moment is to change the file related xmlrpc calls to take an array/list of files to check/transfer but since I don''t know the code well enough I could be wrong. Cheers, Kostas Georgiou
On Wed, 2007-06-20 at 10:59 +0100, Kostas Georgiou wrote:> There is no pipelining AFAIK, probably the easier way to speed things > up at the moment is to change the file related xmlrpc calls to take > an array/list of files to check/transfer but since I don''t know the > code well enough I could be wrong.You''d have to change the way puppet goes through the config pretty drastically; right now it talks to the fileserver every time it encoutners a puppet:// URL when executes the config. To pipeline that way, you''d have to pass over the config first to collect all files that are needed. David
David is absolutely correct. However, it would probably significantly increase overall performance since you would only need to negotiate ONE SSL handshake instead of n, n being the number of files that you''re transferring. Trevor On 6/20/07, David Lutterkort <dlutter@redhat.com> wrote:> On Wed, 2007-06-20 at 10:59 +0100, Kostas Georgiou wrote: > > There is no pipelining AFAIK, probably the easier way to speed things > > up at the moment is to change the file related xmlrpc calls to take > > an array/list of files to check/transfer but since I don''t know the > > code well enough I could be wrong. > > You''d have to change the way puppet goes through the config pretty > drastically; right now it talks to the fileserver every time it > encoutners a puppet:// URL when executes the config. To pipeline that > way, you''d have to pass over the config first to collect all files that > are needed. > > David > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
Actually, just as a thought, if this is done, is there any way to collapse all plain text files into a compressed tarball/zipfile/whatever and pop it across in conjunction with any binary files? The ''file'' utility should provide the correct information. Trevor On 6/20/07, Trevor Vaughan <peiriannydd@gmail.com> wrote:> David is absolutely correct. However, it would probably significantly > increase overall performance since you would only need to negotiate > ONE SSL handshake instead of n, n being the number of files that > you''re transferring. > > Trevor > > On 6/20/07, David Lutterkort <dlutter@redhat.com> wrote: > > On Wed, 2007-06-20 at 10:59 +0100, Kostas Georgiou wrote: > > > There is no pipelining AFAIK, probably the easier way to speed things > > > up at the moment is to change the file related xmlrpc calls to take > > > an array/list of files to check/transfer but since I don''t know the > > > code well enough I could be wrong. > > > > You''d have to change the way puppet goes through the config pretty > > drastically; right now it talks to the fileserver every time it > > encoutners a puppet:// URL when executes the config. To pipeline that > > way, you''d have to pass over the config first to collect all files that > > are needed. > > > > David > > > > > > _______________________________________________ > > Puppet-users mailing list > > Puppet-users@madstop.com > > https://mail.madstop.com/mailman/listinfo/puppet-users > > >
On Jun 20, 2007, at 2:55 PM, Trevor Vaughan wrote:> David is absolutely correct. However, it would probably significantly > increase overall performance since you would only need to negotiate > ONE SSL handshake instead of n, n being the number of files that > you''re transferring.I''m all for this, but I think I''m basically at my limit when it comes to optimizing Puppet''s file serving. I think there is a lot of room for improvement -- heck, even just doing all recursive file copies in one go would be a lot better -- but I don''t think I''m the person for that improvement. I''d *love* to see someone tackle this. I doubt I can make it significantly better without spending a solid couple of weeks breaking my brain over it, which means someone''s going to have to sponsor the work or do it themselves if they really want it done. I consider Puppet''s file serving to be sufficient but not great or even necessarily good, but there are enough insufficient areas that I don''t think it makes sense to refactor it entirely right now. -- The first time I see a jogger smiling, I''ll consider it. --Joan Rivers --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jun 20, 2007, at 12:49 PM, David Lutterkort wrote:> On Wed, 2007-06-20 at 10:59 +0100, Kostas Georgiou wrote: >> There is no pipelining AFAIK, probably the easier way to speed things >> up at the moment is to change the file related xmlrpc calls to take >> an array/list of files to check/transfer but since I don''t know the >> code well enough I could be wrong. > > You''d have to change the way puppet goes through the config pretty > drastically; right now it talks to the fileserver every time it > encoutners a puppet:// URL when executes the config. To pipeline that > way, you''d have to pass over the config first to collect all files > that > are needed.You could probably do this with ''prefetch'', but it''d still be a ton of work. If nothing else, prefetch is only called on providers, but files are not yet split into types and providers, so that''d probably be the first step if you wanted to use prefetch, and that would be no mean feat, I expect. I''m not sure that''s really the best approach, though. I dunno. I really think we need a clean-slate review of the file serving. I expect it needs a completely different design. -- This book fills a much-needed gap. -- Moses Hadas --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com