We haven''t been able to use mongrel for testing yet, but in order not to waste time, we did some performance testing with 1, 2 and 4 puppetmasterd/webrick processes. Attached is an HTML page (ripped from our wiki - not pretty, but reasonably readable) which describes the method and results. If someone feels like making some nice graphs, that would be great. If not, I''ll try to bring my camera tomorrow and will post a snapshot of the whiteboard with my rough graph on it. (If you do, it will make more sense to use the numbers in the comment column for the 4 masters/8 clients combination. Also take care with the first table, the numbers of clients are different than in the following two). Attached is also the benchmark class .pp file, mentioned in the page. My basic conclusions are: - the config run phase scales quite nicely, as there are many small file requests - the config fetch phase does not scale AT ALL. The only result is "a little less clients fail by timing out", but we still cannot realistically get over the magical 16 clients limit. Probably a big problem is the method of testing, in which all clients request the configuration at the same point in time, which would not happen in the real life, where config requests tend to happen irregularily. I will now try to use the testing script which luke wrote to create a more realistic load pattern and see how the acceptable load rises with the number of servers. -- Marcin Owsiany Web Systems Integrator - Guardian Unlimited ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Thu, Jun 07, 2007 at 04:41:31PM +0100, Marcin Owsiany wrote:> not, I''ll try to bring my camera tomorrow and will post a snapshot of > the whiteboard with my rough graph on it.Forgot the camera, made graphs instead (attached). Some of the points are interpolated. Anyway, the point about config fetch not being scalable is even more evident with these graphs :-/ Could someone comment on why possibly that is? I cannot find any other explanation for this other than a single config request is so resource-intensive that other requests don''t get a chance to run, or that there is some mutex which makes the requests serialized. -- Marcin Owsiany Web Systems Integrator - Guardian Unlimited ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Jun 8, 2007, at 6:27 AM, Marcin Owsiany wrote:> Forgot the camera, made graphs instead (attached). > Some of the points are interpolated.What are the X and Y and data sets? Looks like the X is number of clients and Y is the time (in seconds? ms?), but then, what are the 1/2/4 things, then?> Anyway, the point about config fetch not being scalable is even more > evident with these graphs :-/I don''t really understand what''s being measured here, but it looks like you get basically linear performance, assuming X is the number of clients.> Could someone comment on why possibly that is? I cannot find any other > explanation for this other than a single config request is so > resource-intensive that other requests don''t get a chance to run, or > that there is some mutex which makes the requests serialized.I believe that webrick (and mongrel) serialize requests. Really, I think all web servers serialize requests, which is why you run lots of them (e.g., you don''t run one Apache process, you run between 4 and 20 of them). I expect mongrel to perform somewhat but not a ton better than webrick; rather, its performance benefit should mostly derive from how easy it is to run multiple processes. -- Somebody has to do something, and it''s just incredibly pathetic that it has to be us. --Jerry Garcia --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On 6/8/07, Luke Kanies <luke@madstop.com> wrote:> I believe that webrick (and mongrel) serialize requests. Really, I > think all web servers serialize requests, which is why you run lots > of them (e.g., you don''t run one Apache process, you run between 4 > and 20 of them).What do you mean by serialized requests? There are multithreaded webservers that certainly handle more than one request at a time. Mongrel, iirc, is multithreaded but due to thread safety issues with Rails blocks for the entirety of each request when handling Rails apps. -- Thomas Lockney | tlockney@gmail.com | http://opposable-thumbs.net
On Jun 9, 2007, at 2:13 PM, Thomas Lockney wrote:> What do you mean by serialized requests? There are multithreaded > webservers that certainly handle more than one request at a time. > Mongrel, iirc, is multithreaded but due to thread safety issues with > Rails blocks for the entirety of each request when handling Rails > apps.Sorry; of course there are multithreaded servers. I know Apache still defaults to pre-fork (and php isn''t thread-safe so I know I had to stick to pre-fork because of SquirrelMail), and webrick can only use Ruby''s green threads which effectively serialize the processes. I don''t remember about webrick, except that I''m basically positive it''s green threads at best, which means it will perform no better than webrick. Ruby''s threads work fine if the threads aren''t each capable fo taking a whole CPU, but Puppet''s compile process is cpu-bound, so there''s no difference between using Ruby threads and serializing, from what I can tell. Note that I haven''t done much in the way of performance optimization of parsing, so if someone wants to spend some time there, we might be able to reduce the cpu overhead of a single compile. -- Ninety-eight percent of the adults in this country are decent, hard-working, honest Americans. It''s the other lousy two percent that get all the publicity. But then--we elected them. --Lily Tomlin --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Fri, Jun 08, 2007 at 11:13:13AM -0500, Luke Kanies wrote:> On Jun 8, 2007, at 6:27 AM, Marcin Owsiany wrote: > > > Forgot the camera, made graphs instead (attached). > > Some of the points are interpolated. > > What are the X and Y and data sets? Looks like the X is number of > clients and Y is the time (in seconds? ms?), but then, what are the > 1/2/4 things, then?Sorry, I assumed these are easily deducible from the html page describing how the test were performed. On all three graphs: - X is the number of concurrent clients. - Y is the processing time in seconds, averaged on all clients - in config-fetch it''s the configuration retrevial time, as reported by puppetd - in config-run it''s the configuration run time (from after retrieving the configuration until finishing checking/applying changes, obviously this includes all file requests), as reported by puppetd - in total-run, it''s the total "real time" of running puppetd --onetime as reported by bash builtin "time" Now, on all three graphs, the line labeled: - "1" is for running clients directly against one puppetmasterd/webrick - "2" is for running clients directly against two hardware load-balanced puppetmasterd/webrick instances (running on the same physical box) - "4" is for running clients directly against four hardware load-balanced puppetmasterd/webrick instances (running on the same physical box) Please note this is all webrick, I''m still trying to get mongrel running. Each client is run on a different physical box.> > Anyway, the point about config fetch not being scalable is even more > > evident with these graphs :-/ > > I don''t really understand what''s being measured here, but it looks > like you get basically linear performance, assuming X is the number > of clients.Right, the linear performance is more or less acceptable. What is bad, is that there is no improvement in fetching config when comparing running against one and four servers. Could you comment on this? -- Marcin Owsiany Web Systems Integrator - Guardian Unlimited ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396
On Jun 11, 2007, at 3:45 AM, Marcin Owsiany wrote:> Right, the linear performance is more or less acceptable. What is bad, > is that there is no improvement in fetching config when comparing > running against one and four servers. > > Could you comment on this?Um, I can only assume that your load balancer isn''t working. Puppetmasterd is certainly not doing any kind of networked lock, so I don''t see how Puppet could do anything to make it scale this poorly. Can you verify that your load balancer is sending traffic to all hosts? -- To have a right to do a thing is not at all the same as to be right in doing it. -- G. K. Chesterton --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
matthew.malthouse@guardian.co.uk
2007-Jun-18 11:06 UTC
Re: Puppet/webrick benchmark Question.
Is there a suggested/recommended way of testing file serving? Been using puppet-test to do config runs. But I want to fetch files to load the web server. Matthew -- ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Jun 18, 2007, at 6:06 AM, matthew.malthouse@guardian.co.uk wrote:> Is there a suggested/recommended way of testing file serving? > > Been using puppet-test to do config runs. But I want to fetch > files to load the web server.The puppet-test that I posted (which I''ve since added to svn in ext/) supports a --file option: luke@culain(1) $ sudo bin/puppet-test --file "/dist/apps/puppet/ puppet.conf" --repeat 5 notice: Retrieved file in 0.09 seconds notice: Retrieved file in 0.09 seconds notice: Retrieved file in 0.08 seconds notice: Retrieved file in 0.08 seconds notice: Retrieved file in 0.08 seconds luke@culain(0) $ File serving is actually a two-stage process (describe and retrieve). puppet-test uses retrieve by default, but you can add -- describe and it will do that instead. -- Westheimer''s Discovery: A couple of months in the laboratory can frequently save a couple of hours in the library. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies wrote:>> Been using puppet-test to do config runs. But I want to fetch >> files to load the web server. > > The puppet-test that I posted (which I''ve since added to svn in ext/) > supports a --file option: > > luke@culain(1) $ sudo bin/puppet-test --file "/dist/apps/puppet/ > puppet.conf" --repeat 5 > notice: Retrieved file in 0.09 seconds > notice: Retrieved file in 0.09 seconds > notice: Retrieved file in 0.08 seconds > notice: Retrieved file in 0.08 seconds > notice: Retrieved file in 0.08 seconds > luke@culain(0) $Does this assume the filserving root is the root of your fileserver? Dave
On Jun 18, 2007, at 10:52 AM, David Campbell wrote:> > Does this assume the filserving root is the root of your fileserver?That isn''t possible with Puppet''s fileserver. Really. There''s no client subversion (the evil thing, not the scm thing) we can do to get around it, with the possible exception of bugs, although I''ve done what I can. This just skips the whole url specification and encoding. The first / \w+ is still the module name, and the rest is the path relative to that module. -- A nation is a society united by delusions about its ancestry and by common hatred of its neighbors. -- William Ralph Inge --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
matthew.malthouse@guardian.co.uk
2007-Jun-19 12:14 UTC
Re: Puppet/webrick benchmark Question.
puppet-users-bounces@madstop.com wrote on 18/06/2007 16:58:15:> On Jun 18, 2007, at 10:52 AM, David Campbell wrote: > > > > Does this assume the filserving root is the root of your fileserver? > > That isn''t possible with Puppet''s fileserver. Really. There''s no > client subversion (the evil thing, not the scm thing) we can do to > get around it, with the possible exception of bugs, although I''ve > done what I can. > > This just skips the whole url specification and encoding. The first / > \w+ is still the module name, and the rest is the path relative to > that module.Okay. Doing: # /opt/bin/ruby puppet-test --debug --file "/public/apache2/conf/magic" --repeat 120 --server pupuppetmaster.domain.tld simultaneously on 10 clients. 8 completed. 2 timed out. The timed out ones just stopped. Is that expected? No attempt to run the subsequent 113 itterations? debug: Conf dir: /home/matthew/.puppet name: puppet-test debug: Puppet::Network::Client::File: defining fileserver.describe debug: Puppet::Network::Client::File: defining fileserver.list debug: Puppet::Network::Client::File: defining fileserver.retrieve debug: Calling fileserver.retrieve notice: Retrieved file in 53.61 seconds debug: Calling fileserver.retrieve notice: Retrieved file in 87.89 seconds debug: Calling fileserver.retrieve notice: Retrieved file in 101.08 seconds debug: Calling fileserver.retrieve notice: Retrieved file in 14.11 seconds debug: Calling fileserver.retrieve notice: Retrieved file in 6.37 seconds debug: Calling fileserver.retrieve notice: Retrieved file in 24.28 seconds debug: Calling fileserver.retrieve err: Could not call fileserver.retrieve: #<Errno::ECONNRESET: Connection reset b y peer> /opt/lib/ruby/1.8/net/http.rb:586:in `connect'': Connection reset by peer (Puppet ::Network::XMLRPCClientError) from /opt/lib/ruby/1.8/net/http.rb:586:in `connect'' from /opt/lib/ruby/1.8/net/http.rb:553:in `do_start'' from /opt/lib/ruby/1.8/net/http.rb:542:in `start'' from /opt/lib/ruby/1.8/net/http.rb:1032:in `request'' from /opt/lib/ruby/1.8/net/http.rb:989:in `post2'' from /opt/lib/ruby/1.8/xmlrpc/client.rb:535:in `do_rpc'' from /opt/lib/ruby/1.8/xmlrpc/client.rb:420:in `call2'' from /opt/lib/ruby/1.8/xmlrpc/client.rb:410:in `call'' ... 7 levels... from /opt/lib/ruby/site_ruby/1.8/puppet/util.rb:211:in `benchmark'' from puppet-test:203 from puppet-test:197:in `times'' Using --describe. (only on one client for this) Help says "describe the file being tested rather than retrieve it" But the time reported for the initial config fetch of 117.43 seconds is not significantly different from the Max=136.78, Average=108.35, Min=79.59 reported for the itterations. Is it doing a config in order to parse the file. eg debug: Retrieved configuration in 84.36 seconds = config run notice: Retrieved configuration in 86.17 seconds = config run + describe file # /opt/bin/ruby puppet-test --debug --describe "/public/apache2/conf/magic" --repeat 10 --server pupuppetmaster.domain.tld debug: Conf dir: /home/matthew/.puppet name: puppet-test debug: /puppetconfig/puppet/File[/home/matthew/.puppet/ssl]: Autorequiring File[/home/matthew/.puppet] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/run]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var]: Autorequiring File[/home/matthew/.puppet] debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/state/puppetdlock]: Autorequiring File[/home/matthew/.puppet/var/state] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/namespaceauth.conf]: Autorequiring File[/home/matthew/.puppet] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/facts]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/state/graphs]: Autorequiring File[/home/matthew/.puppet/var/state] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/templates]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/lib]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/state]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/plugins]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/log/http.log]: Autorequiring File[/home/matthew/.puppet/var/log] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/log]: Autorequiring File[/home/matthew/.puppet/var] debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/log/puppetd.log]: Autorequiring File[/home/matthew/.puppet/var/log] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/state/state.yaml]: Autorequiring File[/home/matthew/.puppet/var/state] debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/state/localconfig]: Autorequiring File[/home/matthew/.puppet/var/state] debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/classes.txt]: Autorequiring File[/home/matthew/.puppet] debug: /puppetconfig/puppet/File[/home/matthew/.puppet/namespaceauth.conf]: File does not exist debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/log/puppetd.log]: File does not exist debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/log/http.log]: File does not exist debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/plugins]: File does not exist debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/state/localconfig]: File does not exist debug: /puppetconfig/puppet/File[/home/matthew/.puppet/var/facts]: File does not exist debug: /puppetconfig/puppetd/File[/home/matthew/.puppet/var/state/puppetdlock]: File does not exist debug: Finishing transaction 4599360 with 0 changes debug: Creating default schedules debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 117.43 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 118.91 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 79.59 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 80.93 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 89.76 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 91.65 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 134.39 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 136.26 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 128.49 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 130.20 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 84.36 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 86.17 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 91.83 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 93.87 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 135.10 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 136.79 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 127.99 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 129.04 seconds debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 90.80 seconds info: Caching configuration at /home/matthew/.puppet/var/state/localconfig.yaml notice: Retrieved configuration in 92.60 seconds Matthew -- ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Jun 19, 2007, at 7:14 AM, matthew.malthouse@guardian.co.uk wrote:> > Okay. > > Doing: > # /opt/bin/ruby puppet-test --debug --file "/public/apache2/conf/ > magic" --repeat 120 --server pupuppetmaster.domain.tld > > simultaneously on 10 clients. 8 completed. 2 timed out. > > The timed out ones just stopped. Is that expected? No attempt to > run the subsequent 113 itterations?Well, I don''t think there''s any error-handling in puppet-test, so yeah, I guess I''d expect failures to kill them, which I suppose is bad.> err: Could not call fileserver.retrieve: #<Errno::ECONNRESET: > Connection reset b > y peer> > /opt/lib/ruby/1.8/net/http.rb:586:in `connect'': Connection reset by > peer (Puppet > ::Network::XMLRPCClientError) > from /opt/lib/ruby/1.8/net/http.rb:586:in `connect'' > from /opt/lib/ruby/1.8/net/http.rb:553:in `do_start'' > from /opt/lib/ruby/1.8/net/http.rb:542:in `start'' > from /opt/lib/ruby/1.8/net/http.rb:1032:in `request'' > from /opt/lib/ruby/1.8/net/http.rb:989:in `post2'' > from /opt/lib/ruby/1.8/xmlrpc/client.rb:535:in `do_rpc'' > from /opt/lib/ruby/1.8/xmlrpc/client.rb:420:in `call2'' > from /opt/lib/ruby/1.8/xmlrpc/client.rb:410:in `call'' > ... 7 levels... > from /opt/lib/ruby/site_ruby/1.8/puppet/util.rb:211:in > `benchmark'' > from puppet-test:203 > from puppet-test:197:in `times''Using Webrick, mongrel, or both? I''m definitely curious how mongrel behaves under load.> Using --describe. (only on one client for this) > > Help says "describe the file being tested rather than retrieve it" > > But the time reported for the initial config fetch of 117.43 > seconds is not significantly different from the Max=136.78, > Average=108.35, Min=79.59 reported for the itterations. > > Is it doing a config in order to parse the file. eg > > debug: Retrieved configuration in 84.36 seconds = config run > notice: Retrieved configuration in 86.17 seconds = config run + > describe fileAh. I''d set the arguments up so that --describe was a modifier to -- file, but I didn''t have any checks to make sure you actually specified --file. Thus, the way you were calling it was doing a compile, rather than a describe. I''ve just updated puppet-test in svn; you now pick between --retrieve <file>, --describe <file>, and --compile (still the default). -- What''s the good of having mastery over cosmic balance and knowing the secrets of fate if you can''t blow something up? -- Terry Pratchett, "Reaper Man" --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
matthew.malthouse@guardian.co.uk
2007-Jun-19 16:33 UTC
Re: Puppet/webrick benchmark Question.
> Using Webrick, mongrel, or both? I''m definitely curious how mongrel > behaves under load.So far webrick behind a hardware load balancer. According to the masterhhtplogs each puppetmaster is being equally loaded so it''s working. I''m doing a test with a puppet config that contains 227 files already run (ie they all exist on the client) 1 delete 69 files 2 random (between 0 and 400 seconds) sleep 3 /opt/bin/puppetd --onetime --debug 4 delete 69 files 5 random sleep 6 /opt/bin/puppetd --onetime --debug then getting times for the config run; the file fetching and total. Averaging the total times 4 client 6 client 8 client 10 client 12 client 14 client 1 server 1212 (0) 1495 (17) 1230 (31) 2485 (25) 2 server 1341 (0) 2485 (25) 4 server Running now big numbers are the average time in seconds to complete 1 run across all those that completed. Number in parenthesis is the percentage of runs that timed out. So far all time outs have been on the config, none of file fetching. Okay, so this is quite intensive and we''re piling it on hard quite deliberately to get to the point where it breaks. But this is just getting a baseline agains which to compare.... Tomorrow Mongrel! :-)> Ah. I''d set the arguments up so that --describe was a modifier to -- > file, but I didn''t have any checks to make sure you actually > specified --file. Thus, the way you were calling it was doing a > compile, rather than a describe.Perfect. That works fine: -bash-3.00$ /opt/bin/ruby puppet-test --debug --file "/public/apache2/conf/magic" --describe --repeat 10 --server pupmst01.gudev.gnl debug: Conf dir: /home/matthew/.puppet name: puppet-test debug: Puppet::Network::Client::File: defining fileserver.describe debug: Puppet::Network::Client::File: defining fileserver.list debug: Puppet::Network::Client::File: defining fileserver.retrieve debug: Calling fileserver.describe notice: Described file in 1.03 seconds debug: Calling fileserver.describe notice: Described file in 0.62 seconds> I''ve just updated puppet-test in svn; you now pick between --retrieve > <file>, --describe <file>, and --compile (still the default).One for Marcin when he gets back. Matthew -- Striving to better, oft we mar what''s well. - Albany, Act 1 Scene iv, King Lear ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
matthew.malthouse@guardian.co.uk a écrit : woot i don''t know which html font you use but your emails are barely readable here , had to switch to plain tex. Is there any reason you do not use text emails but html one ? -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Jun 19, 2007, at 11:33 AM, matthew.malthouse@guardian.co.uk wrote:> > So far webrick behind a hardware load balancer. According to the > masterhhtplogs each puppetmaster is being equally loaded so it''s > working. > > I''m doing a test with a puppet config that contains 227 files > already run (ie they all exist on the client) > > 1 delete 69 files > 2 random (between 0 and 400 seconds) sleep > 3 /opt/bin/puppetd --onetime --debug > 4 delete 69 files > 5 random sleep > 6 /opt/bin/puppetd --onetime --debug > > then getting times for the config run; the file fetching and > total. Averaging the total times > > 4 client 6 client 8 client 10 client 12 client > 14 client > 1 server 1212 (0) 1495 (17) 1230 (31) 2485 (25) > 2 server 1341 (0) 2485 (25) > 4 server > Running nowThat''s, um, very weird. No difference between running with one server or two? With a mongrel back-end and using puppet-test --fork 5 --compile --repeat 5 on a single machine with a small configuration, I get 25s for two servers and 38s for one, which shows imperfect scaling but not bad considering there are only two pieces of hardware involved in the test.> big numbers are the average time in seconds to complete 1 run > across all those that completed. > Number in parenthesis is the percentage of runs that timed out. So > far all time outs have been on the config, none of file fetching. > > Okay, so this is quite intensive and we''re piling it on hard quite > deliberately to get to the point where it breaks. But this is just > getting a baseline agains which to compare....Yeah. Clearly things aren''t behaving well when lots of machines contact at the same time, and the config compile seems to be the biggest burden. It also looks like 6 is around the magical number for webrick, where it can handle that many open connections at once and starts failing beyond that. FWIW, I just ran some tests here and had to go to 30 concurrent clients before I started getting failures on my mongrel node, and even those failures were failed connections to the database (I''m doing both an export and a collect in my configurations, which definitely skews my results). I''ve been considering adding some kind of implicit load-distribution, so that clients could ask the server how long to delay before connecting (the clients already contact the server to figure out if their configuration has changed; this method could be perverted to also indicate whether the client should wait to contact the server).> Tomorrow Mongrel! :-)Great.> > Ah. I''d set the arguments up so that --describe was a modifier > to -- > > file, but I didn''t have any checks to make sure you actually > > specified --file. Thus, the way you were calling it was doing a > > compile, rather than a describe. > > Perfect. That works fine: > > -bash-3.00$ /opt/bin/ruby puppet-test --debug --file "/public/ > apache2/conf/magic" --describe --repeat 10 --server > pupmst01.gudev.gnl > debug: Conf dir: /home/matthew/.puppet name: puppet-test > debug: Puppet::Network::Client::File: defining fileserver.describe > debug: Puppet::Network::Client::File: defining fileserver.list > debug: Puppet::Network::Client::File: defining fileserver.retrieve > debug: Calling fileserver.describe > notice: Described file in 1.03 seconds > debug: Calling fileserver.describe > notice: Described file in 0.62 seconds > > > I''ve just updated puppet-test in svn; you now pick between -- > retrieve > > <file>, --describe <file>, and --compile (still the default).I upgraded puppet-test once again, adding a "fresh" check, which you would need to use "--test fresh" to run. Because configuration is a two step process (check for freshness, then get the config), I figured it was worth having this. My preferred next step with the test is to enable adding multiple tests in a run, so you could have each run be fresh(), compile(), then (describe(), retrieve()) x 5, for instance, which would somewhat mirror a simple configuration. I don''t think I''ll have time to do that today, though. -- oH for Pete''s sake, use a real filter. ;-) -- Tom Christiansen Look, I refuse to install any filter which looks like it''s more highly evolved than homo sapiens, it will just get embarassing when it starts critiquing the literrary style of my friends mail to me. -- Richard Caley, in comp.lang.perl.misc --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
matthew.malthouse@guardian.co.uk
2007-Jun-20 06:08 UTC
Re: Puppet/webrick benchmark Question.
puppet-users-bounces@madstop.com wrote on 19/06/2007 17:57:14:> matthew.malthouse@guardian.co.uk a écrit : > > woot i don''t know which html font you use but your emails are barely > readable here , had to switch to plain tex. Is there any reason you do > not use text emails but html one ?When it leaves my desktop it''s in Courier 12pt plain text. What the corporate Domino mail server does to it after that is sothing I can''t control. Could you mail me off list with a screen shot of what it looks like? Then I can wave it at the Domino Admins and see if something can be done about it. Matthew ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
matthew.malthouse@guardian.co.uk
2007-Jun-20 12:48 UTC
Re: Puppet/webrick benchmark Question.
> then getting times for the config run; the file fetching and total. > Averaging the total times > > 4 client 6 client 8 client 10 client 12 client 14client> 1 server 1212 (0) 1495 (17) 1230 (31) 2485 (25) > 2 server 1341 (0) 2485 (25) > 4 server Running now > > big numbers are the average time in seconds to complete 1 run across > all those that completed. > Number in parenthesis is the percentage of runs that timed out. So > far all time outs have been on the config, none of file fetching.Curio: against 4 (webrick) puppetmaster servers 14, 16, 18 and 20 clients all faultless. With 22 clients I start to get time-outs. But not as above on the config run; these are either describe, retrieve or both. Currently however low level - only 14 errs in 18354 calls. Matthew ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396