are we having problem with defines in puppet 3.0.1/3.0.2? the more and more we use defines in our repo, the more lagish puppetmaster becomes. today i''ve made some synthetic checks with our lb setup, which consists of 121 defines, each of them wraps around 4 another defines (nagios, haproxy, stunnel configs). results are catastrophic. even more, we started to use ruby 1.9.3, as 1.8.7 is becoming obsolete and it''s adding a more lag. i''ve reduced define numbers and observed that compile time reduces significantly each time. additionally i''ve inlined code from each define that we source from our lb define, and it also speeds up compile a lot. i''ve compiled with: time /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby /usr/bin/puppet master --compile lb2 and /usr/bin/ruby /usr/bin/puppet master --compile lb2 actual times are here: https://github.com/asquelt/puppet-defines-wtf/blob/master/BENCHMARK manifest is here: https://github.com/asquelt/puppet-defines-wtf/blob/master/manifests/projects/dc1-lb.pp catalogs are also in repo: https://github.com/asquelt/puppet-defines-wtf you can see that i''ve decorated everything with double colon to make sure this is not related to #17335 is it just me? do you have any workarounds? should i file a bug? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/yJhfP5lqsqYJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thursday, January 24, 2013 1:29:32 PM UTC-6, asq wrote:> > are we having problem with defines in puppet 3.0.1/3.0.2? > the more and more we use defines in our repo, the more lagish puppetmaster > becomes. > > today i''ve made some synthetic checks with our lb setup, which consists of > 121 defines, each of them wraps around 4 another defines (nagios, haproxy, > stunnel configs). > results are catastrophic. > > even more, we started to use ruby 1.9.3, as 1.8.7 is becoming obsolete and > it''s adding a more lag. > > i''ve reduced define numbers and observed that compile time reduces > significantly each time. > additionally i''ve inlined code from each define that we source from our lb > define, and it also speeds up compile a lot. > > i''ve compiled with: > time /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby /usr/bin/puppet master > --compile lb2 > and > /usr/bin/ruby /usr/bin/puppet master --compile lb2 > > actual times are here: > https://github.com/asquelt/puppet-defines-wtf/blob/master/BENCHMARK > manifest is here: > https://github.com/asquelt/puppet-defines-wtf/blob/master/manifests/projects/dc1-lb.pp > catalogs are also in repo: https://github.com/asquelt/puppet-defines-wtf > > you can see that i''ve decorated everything with double colon to make sure > this is not related to #17335 > > is it just me? > do you have any workarounds? > should i file a bug? >To be clear, it looks like you are talking about counts of defined type * instances*, not counts of distinct defined types. Is that right? I am inclined to suspect that the problem is not defined types in general, but rather the specific defined types you are using. Indeed, your data seem to support that inasmuch as inlining definitions doesn''t cut your compilation times very much. I can think of several things might cause long compilation times: - evaluating many templates, especially complex ones - making many hiera lookups if your hierarchy definition uses interpolated non-global variables - repetition of anything requiring network access, such as syncing files with a version-control repository, accessing files on a remote file system, or performing name resolutions. - calling generate() a lot That''s not an exhaustive list. Anyway, I think filing a bug now would be premature. It may be that there is a bona fide performance problem, but if so then I don''t think it has been pinpointed yet. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger napisał:> > > To be clear, it looks like you are talking about counts of defined type * > instances*, not counts of distinct defined types. Is that right? >yes. it''s test::rule() and test::rule_inline() in linked manifest.> I am inclined to suspect that the problem is not defined types in general, > but rather the specific defined types you are using. Indeed, your data > seem to support that inasmuch as inlining definitions doesn''t cut your > compilation times very much. > >we''re talking about 5m33s and 4m13s difference in compile. this *is* a lot.> I can think of several things might cause long compilation times: > > - evaluating many templates, especially complex ones > - making many hiera lookups if your hierarchy definition uses > interpolated non-global variables > - repetition of anything requiring network access, such as syncing > files with a version-control repository, accessing files on a remote file > system, or performing name resolutions. > - calling generate() a lot > > That''s not an exhaustive list. >please check define rule_inline() in linked pp file - it''s nothing fancy. - 3 static files - 1 erb (35 lines with 5 variables substitution and 1 if/else clause) - 1 hiera call - 1 exported resource - 4 calls to custom type (firewall) no generates or other puppetmaster functions, no network stuff, no dns-related stuff or anything i haven''t already hashed-out to test if it will affect compile times significantly. Anyway, I think filing a bug now would be premature. It may be that there> is a bona fide performance problem, but if so then I don''t think it has > been pinpointed yet. > >i filed a bug #18880 anyway, as i sense it might be some design error in puppet - i can''t really see why number of define() calls should relate to compile times. as they are not "real" resources, there should be no reason for them to be included in catalog, which also might relate to apply times. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
W dniu piątek, 25 stycznia 2013 16:29:00 UTC+1 użytkownik asq napisał:> > W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger > napisał: >please check define rule_inline() in linked pp file - it''s nothing fancy.> > - 3 static files > - 1 erb (35 lines with 5 variables substitution and 1 if/else clause) > - 1 hiera call > - 1 exported resource > - 4 calls to custom type (firewall) >disabling hiera i shaved only 11 seconds on ruby rvm (6 seconds on stock ruby). disabling erb template made no measurable difference. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Friday, January 25, 2013 9:29:00 AM UTC-6, asq wrote:> > W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger > napisał: >> >> >> To be clear, it looks like you are talking about counts of defined type * >> instances*, not counts of distinct defined types. Is that right? >> > > yes. it''s test::rule() and test::rule_inline() in linked manifest. > > >> I am inclined to suspect that the problem is not defined types in >> general, but rather the specific defined types you are using. Indeed, your >> data seem to support that inasmuch as inlining definitions doesn''t cut your >> compilation times very much. >> >> > we''re talking about 5m33s and 4m13s difference in compile. this *is* a lot. >The version using more defined-type instances has more total resources. The time per resource is very nearly the same. Your compile times certainly will scale with the total number of resources declared. Do not confuse defined types with macros. They are bona fide resource types, functioning like native types in most ways.> > >> I can think of several things might cause long compilation times: >> >> - evaluating many templates, especially complex ones >> - making many hiera lookups if your hierarchy definition uses >> interpolated non-global variables >> - repetition of anything requiring network access, such as syncing >> files with a version-control repository, accessing files on a remote file >> system, or performing name resolutions. >> - calling generate() a lot >> >> That''s not an exhaustive list. >> > > please check define rule_inline() in linked pp file - it''s nothing fancy. >Fanciness is not the question. Even very simple manifests can make declarations that are costly to compile.> > - 3 static files > - 1 erb (35 lines with 5 variables substitution and 1 if/else clause) >The erb evaluation is probably more costly than everything else about that particular File declaration, combined. Especially so in that it requires (in that particular case) up to four separate probes of the file system just to find the template file.> - 1 hiera call >I believe hiera caches data to the extent that it can do, but depending on how your hierarchy is defined and how your data are arranged, it could still be that each one of those hiera() calls involves filesystem probes and possibly reading and parsing YAML. That''s not particularly cheap even if your data are simple.> - 1 exported resource >I must have overlooked that, but it''s not especially cheap, either, inasmuch as it involves connecting to and querying a DB. It''s practically expensive if the underlying DB is not local, as that then would involve network activity.> - 4 calls to custom type (firewall) >Which are probably cheap(ish), but that''s not certain.> > no generates or other puppetmaster functions, no network stuff, no > dns-related stuff or anything i haven''t already hashed-out to test if it > will affect compile times significantly. > > Anyway, I think filing a bug now would be premature. It may be that there >> is a bona fide performance problem, but if so then I don''t think it has >> been pinpointed yet. >> >> > i filed a bug #18880 anyway, as i sense it might be some design error in > puppet - i can''t really see why number of define() calls should relate to > compile times. as they are not "real" resources, there should be no reason > for them to be included in catalog, which also might relate to apply times. > >Definition instances certainly are real resources in most ways that matter. In any event, your compilation times even for the "inline" version of your manifests seem unreasonably high, which is why I suggested that you look at other possible performance drags. In addition to the manifest-specific issues I discussed above, you should also look at how your master is provisioned and loaded. Puppet is a bit of a memory hog, so if the available physical RAM is low then Puppet may perform a lot of expensive swaps between physical and virtual RAM. If the master is sharing hardware with other services (even other masters in, for example, a passenger setup), then contention for the disk subsystem could slow everything, especially if you have SATA (or worse) disks instead of SCSI. There are many other such possibilities. On the other hand, I don''t mean to suggest that declaration of defined-type instances is free or cheap. As I said, defined types are not macros, but rather bona fide resource types. Declaring a defined-type instance involves more work than simply making all the declarations that appear in the definition body. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
W dniu poniedziałek, 28 stycznia 2013 16:49:38 UTC+1 użytkownik jcbollinger napisał:> > - 1 erb (35 lines with 5 variables substitution and 1 if/else clause) > > The erb evaluation is probably more costly than everything else about that > particular File declaration, combined. Especially so in that it requires > (in that particular case) up to four separate probes of the file system > just to find the template file. > >i replaced this erb with string "foo" and it showed sub-second differences, not really measurable for 2000 resources.> > >> - 1 hiera call >> > > > I believe hiera caches data to the extent that it can do, but depending on > how your hierarchy is defined and how your data are arranged, it could > still be that each one of those hiera() calls involves filesystem probes > and possibly reading and parsing YAML. That''s not particularly cheap even > if your data are simple. >disabling hiera shaves about 10 seconds on 2000 resources. not a big deal either.> - 1 exported resource >> > > > I must have overlooked that, but it''s not especially cheap, either, > inasmuch as it involves connecting to and querying a DB. It''s practically > expensive if the underlying DB is not local, as that then would involve > network activity. >disabling it shaves another 30 seconds on 2000 resources. nothing spectacular, considering that only adding 200-300 resources will make this time as high again.> - 4 calls to custom type (firewall) >> > > Which are probably cheap(ish), but that''s not certain. >yes, hashing them out doesn''t help much.> > In addition to the manifest-specific issues I discussed above, you should > also look at how your master is provisioned and loaded. Puppet is a bit of > a memory hog, so if the available physical RAM is low then Puppet may > perform a lot of expensive swaps between physical and virtual RAM. If the > master is sharing hardware with other services (even other masters in, for > example, a passenger setup), then contention for the disk subsystem could > slow everything, especially if you have SATA (or worse) disks instead of > SCSI. There are many other such possibilities. > >i''ve tried to bind those problems to i/o but putting manifests on ramdisk doesn''t help at all. manifests are still compiling very slow. i know that those times are high. but anyway i can''t find a single "bit" that makes it so. i think puppet is very sensitible to the way its used. when you look at ruby 1.8/1.9 comparisions for the same manifests, you can clearly see that something worrying is happening, as newer ruby should be more optimized, but instead it runs significantly slower. on production we use passenger with no of puppetmaster processes equal to number of threads (and queue of excessive catalog requests on haproxy on front of it). -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
W dniu poniedziałek, 28 stycznia 2013 17:04:51 UTC+1 użytkownik asq napisał:> > > on production we use passenger with no of puppetmaster processes equal to > number of threads (and queue of excessive catalog requests on haproxy on > front of it). >i mean - number of cores (8). -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
The bottom line is that I don''t have an explanation for why your compilations are so slow. Sorry. I don''t think it''s anything specific to defined types, except inasmuch as defined type instances do have overhead comparable to that of native type instances. Perhaps the guys over on puppet-dev would have a suggestion about how you can profile catalog compilation to get an idea of where all the time is spent. On Monday, January 28, 2013 10:04:51 AM UTC-6, asq wrote:> > i know that those times are high. but anyway i can''t find a single "bit" > that makes it so. i think puppet is very sensitible to the way its used. > when you look at ruby 1.8/1.9 comparisions for the same manifests, you can > clearly see that something worrying is happening, as newer ruby should be > more optimized, but instead it runs significantly slower. > >Of course Puppet''s compilation speed is sensitive to the contents of your manifests, and more so than just to their raw size. It may be that you can substantially improve your performance if you can identify the problem area(s). On the other hand, I don''t think it''s at all safe to assume that a higher-version Ruby (or anything else) should yield greater performance. Performance optimization is certainly one development objective, but it is often at odds with new feature implementation and even with bug fixes. Especially with between Ruby 1.8 and Ruby 1.9 there are some deep changes in Ruby''s string handling and character I/O, and these (among other things) might easily exact a performance penalty from applications, such as Puppet, that do a lot of text manipulation. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
W dniu wtorek, 29 stycznia 2013 16:54:46 UTC+1 użytkownik jcbollinger napisał:> > > The bottom line is that I don''t have an explanation for why your > compilations are so slow. Sorry. I don''t think it''s anything specific to > defined types, except inasmuch as defined type instances do have overhead > comparable to that of native type instances. Perhaps the guys over on > puppet-dev would have a suggestion about how you can profile catalog > compilation to get an idea of where all the time is spent. > >it''s surely related to puppetmaster being slowed down by ruby gems, as i mentioned here: https://groups.google.com/d/topic/puppet-users/jpx9fNcFuMk/discussion it might be not only cause, but giving a puppet dedicated rvm gemset (just semver for puppet, json for pdb and gpgme for hiera) gave us some significant boost. another suspicious thing is how catalog is travelling over network. i think it should be compressed (lots of inlined templates) but for some reason it isn''t (there is b64_zlib_yaml type, but there''s no zlib_pson). -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.