Hi everyone, The hiera command line program (version 0.3.0) is currently taking around 2 seconds per invocation. This seems quite slow to me, but is probably not an issue for most users. We have a number of legacy machines that for one reason or another have not been puppetised yet, and probably won''t be anytime soon, so I have a somewhat clever script that, for each of these hosts, uses the hiera command like utility to export a few puppet-templated documents just as puppet+hiera does for our puppeted server. This works very nicely, but involves ~250 hiera lookups per host, which at 2 seconds per lookup, is taking more than 8 minutes per host (and using near 100% of a CPU core the entire time). As we have ~40 such hosts, that''s over 5 hours with no concurrency (we do use some concurrency, but since each invocation uses near 100% of a CPU core each, there''s no point running more simultaneous hiera processes than the number of CPUs, and even then, we need some reserved for other services). So, is there some way I can make this export run faster? The bulk of the time seems to be spent in initialising the hiera command line program, not the actual data lookup (for example, if I fail to provide the necessary identity / scope files, it still takes 2 seconds to get around to returning an error, but if I fail to provide a config file, it returns instantaneously with error). So, I''d either like to make the program load / initialise much faster, or be able to query multiple values for a single hiera command line invocation. Also, is there a more recent version of hiera that would be faster? (I''m a little unclear about hiera versioning - we''re running 0.3.0, but I don''t know if that''s recent or old). Any suggestions / tips would be greatly appreciated :) Thanks. Paul. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
hello, ----- Original Message -----> From: "Paul Colby" <pcolby@gmail.com> > To: puppet-users@googlegroups.com > Sent: Monday, August 13, 2012 6:22:10 PM > Subject: [Puppet Users] hiera command line performance > > Hi everyone, > > > The hiera command line program (version 0.3.0) is currently taking > around 2 seconds per invocation. This seems quite slow to me, but is > probably not an issue for most users.2 seconds seem quite slow though i guess it depends on the size of your yamls and so forth, sounds like yours are big and complex. I''d hope the bulk of the time is spent on starting/stopping ruby each time. A better option might be to consider writing your script in ruby rather than bash so you can then do your lookup that way reusing the single hiera instance and avoid the ruby start/stop cost So a simple test on my setup doing: for i in {1..100} do hiera syslocation ::location=foo done this takes 30 seconds, ages. Here is the same in Ruby reusing the class and using a single interpreter: h = Hiera.new(:config => "/etc/hiera.yaml") Benchmark.measure do 100.times do puts h.lookup("syslocation", "", {"::location" => "foo"}) end end this takes 0.1 second for 100 lookups. If you know a bit of ruby this would be a good approach for you even if you just write your script to configure a single machine in ruby and run it 40 times only> > > We have a number of legacy machines that for one reason or another > have not been puppetised yet, and probably won''t be anytime soon, so > I have a somewhat clever script that, for each of these hosts, uses > the hiera command like utility to export a few puppet-templated > documents just as puppet+hiera does for our puppeted server. This > works very nicely, but involves ~250 hiera lookups per host, which > at 2 seconds per lookup, is taking more than 8 minutes per host (and > using near 100% of a CPU core the entire time). As we have ~40 such > hosts, that''s over 5 hours with no concurrency (we do use some > concurrency, but since each invocation uses near 100% of a CPU core > each, there''s no point running more simultaneous hiera processes > than the number of CPUs, and even then, we need some reserved for > other services). > > > So, is there some way I can make this export run faster? The bulk of > the time seems to be spent in initialising the hiera command line > program, not the actual data lookup (for example, if I fail to > provide the necessary identity / scope files, it still takes 2 > seconds to get around to returning an error, but if I fail to > provide a config file, it returns instantaneously with error). So, > I''d either like to make the program load / initialise much faster, > or be able to query multiple values for a single hiera command line > invocation. > > > Also, is there a more recent version of hiera that would be faster? > (I''m a little unclear about hiera versioning - we''re running 0.3.0, > but I don''t know if that''s recent or old). > > > Any suggestions / tips would be greatly appreciated :) > > > Thanks. > > > Paul. > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ . > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Thanks R.I. Pienaar, that''s exactly the kind of suggestion I was hoping for! :) I haven''t written any Ruby for ~9 years, but as I regularly develop in several other languages (primarily C++), I''ll enjoy getting my hands dirty with Ruby again. And thanks for including a basic single-interpreter example - that will help me get going a lot quicker! Thanks, Paul. On Wednesday, August 15, 2012 2:45:06 AM UTC+10, R.I. Pienaar wrote:> > hello, > > ----- Original Message ----- > > From: "Paul Colby" <pco...@gmail.com <javascript:>> > > To: puppet...@googlegroups.com <javascript:> > > Sent: Monday, August 13, 2012 6:22:10 PM > > Subject: [Puppet Users] hiera command line performance > > > > Hi everyone, > > > > > > The hiera command line program (version 0.3.0) is currently taking > > around 2 seconds per invocation. This seems quite slow to me, but is > > probably not an issue for most users. > > 2 seconds seem quite slow though i guess it depends on the size of your > yamls and so forth, sounds like yours are big and complex. I''d hope the > bulk of the time is spent on starting/stopping ruby each time. > > A better option might be to consider writing your script in ruby rather > than bash so you can then do your lookup that way reusing the single > hiera instance and avoid the ruby start/stop cost > > So a simple test on my setup doing: > > for i in {1..100} > do > hiera syslocation ::location=foo > done > > this takes 30 seconds, ages. Here is the same in Ruby reusing the > class and using a single interpreter: > > h = Hiera.new(:config => "/etc/hiera.yaml") > > Benchmark.measure do > 100.times do > puts h.lookup("syslocation", "", {"::location" => "foo"}) > end > end > > this takes 0.1 second for 100 lookups. If you know a bit of ruby this > would be a good approach for you even if you just write your script to > configure a single machine in ruby and run it 40 times only > > > > > > > We have a number of legacy machines that for one reason or another > > have not been puppetised yet, and probably won''t be anytime soon, so > > I have a somewhat clever script that, for each of these hosts, uses > > the hiera command like utility to export a few puppet-templated > > documents just as puppet+hiera does for our puppeted server. This > > works very nicely, but involves ~250 hiera lookups per host, which > > at 2 seconds per lookup, is taking more than 8 minutes per host (and > > using near 100% of a CPU core the entire time). As we have ~40 such > > hosts, that''s over 5 hours with no concurrency (we do use some > > concurrency, but since each invocation uses near 100% of a CPU core > > each, there''s no point running more simultaneous hiera processes > > than the number of CPUs, and even then, we need some reserved for > > other services). > > > > > > So, is there some way I can make this export run faster? The bulk of > > the time seems to be spent in initialising the hiera command line > > program, not the actual data lookup (for example, if I fail to > > provide the necessary identity / scope files, it still takes 2 > > seconds to get around to returning an error, but if I fail to > > provide a config file, it returns instantaneously with error). So, > > I''d either like to make the program load / initialise much faster, > > or be able to query multiple values for a single hiera command line > > invocation. > > > > > > Also, is there a more recent version of hiera that would be faster? > > (I''m a little unclear about hiera versioning - we''re running 0.3.0, > > but I don''t know if that''s recent or old). > > > > > > Any suggestions / tips would be greatly appreciated :) > > > > > > Thanks. > > > > > > Paul. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Puppet Users" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ . > > To post to this group, send email to puppet...@googlegroups.com<javascript:>. > > > To unsubscribe from this group, send email to > > puppet-users...@googlegroups.com <javascript:>. > > For more options, visit this group at > > http://groups.google.com/group/puppet-users?hl=en. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/WQV-NVyOHXgJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Thanks again R.I. Pienaar.
I just got time to get back to this, I''ve implemented a basic ruby
script
to export the hiera data as you''ve suggested - works extremely well!!
:)
Interestingly, the performance issue seems to be with puppet specifically,
not hiera.
Here''s a slightly cut-down version of my export script:
#!/usr/bin/ruby
require ''rubygems''
require ''hiera''
#require ''puppet''
# Load the "facts" for the host to export settings for.
if ARGV.length != 1
puts "Usage: #{File.basename($PROGRAM_NAME)}
/path/to/host-facts.yaml"
exit
end
scope = YAML.load_file(ARGV.first)
scope[''calling_module''] = ''settings''
# Fetch all settings for this host from hiera.
hiera = Hiera.new(:config => ''/etc/puppet/hiera.yaml'')
settings = hiera.lookup(''settings'', '''',
scope, '''', :array)
settings.each do |setting|
puts "#{setting}=#{hiera.lookup(setting, '''',
scope)}"
end
Note that the "require ''puppet''" line is commented
out... code examples on
the net use that statement, but the scripts runs fine both with and without
it.
The interesting thing is, without that require, the script is nice and
quick, but with it, it is 2.5 times slower.
Example times:
With "require puppet"
real 0m2.448s
user 0m2.123s
sys 0m0.323s
Without "require puppet"
real 0m0.917s
user 0m0.808s
sys 0m0.098s
Anyway, I''m very happy with the result! The inner "*.each"
call at the end
means I''m calling this script ~40 times, instead of the (slower) hiera
command line program ~10,000 times. Much faster, and lower CPU usage too :)
Thanks again.
Paul.
On Wednesday, August 15, 2012 10:55:15 AM UTC+10, Paul Colby
wrote:>
> Thanks R.I. Pienaar, that''s exactly the kind of suggestion I was
hoping
> for! :)
>
> I haven''t written any Ruby for ~9 years, but as I regularly
develop in
> several other languages (primarily C++), I''ll enjoy getting my
hands dirty
> with Ruby again.
>
> And thanks for including a basic single-interpreter example - that will
> help me get going a lot quicker!
>
> Thanks,
>
> Paul.
>
> On Wednesday, August 15, 2012 2:45:06 AM UTC+10, R.I. Pienaar wrote:
>>
>> hello,
>>
>> ----- Original Message -----
>> > From: "Paul Colby" <pco...@gmail.com>
>> > To: puppet...@googlegroups.com
>> > Sent: Monday, August 13, 2012 6:22:10 PM
>> > Subject: [Puppet Users] hiera command line performance
>> >
>> > Hi everyone,
>> >
>> >
>> > The hiera command line program (version 0.3.0) is currently taking
>> > around 2 seconds per invocation. This seems quite slow to me, but
is
>> > probably not an issue for most users.
>>
>> 2 seconds seem quite slow though i guess it depends on the size of your
>> yamls and so forth, sounds like yours are big and complex.
I''d hope the
>> bulk of the time is spent on starting/stopping ruby each time.
>>
>> A better option might be to consider writing your script in ruby rather
>> than bash so you can then do your lookup that way reusing the single
>> hiera instance and avoid the ruby start/stop cost
>>
>> So a simple test on my setup doing:
>>
>> for i in {1..100}
>> do
>> hiera syslocation ::location=foo
>> done
>>
>> this takes 30 seconds, ages. Here is the same in Ruby reusing the
>> class and using a single interpreter:
>>
>> h = Hiera.new(:config => "/etc/hiera.yaml")
>>
>> Benchmark.measure do
>> 100.times do
>> puts h.lookup("syslocation", "",
{"::location" => "foo"})
>> end
>> end
>>
>> this takes 0.1 second for 100 lookups. If you know a bit of ruby this
>> would be a good approach for you even if you just write your script to
>> configure a single machine in ruby and run it 40 times only
>>
>> >
>> >
>> > We have a number of legacy machines that for one reason or another
>> > have not been puppetised yet, and probably won''t be
anytime soon, so
>> > I have a somewhat clever script that, for each of these hosts,
uses
>> > the hiera command like utility to export a few puppet-templated
>> > documents just as puppet+hiera does for our puppeted server. This
>> > works very nicely, but involves ~250 hiera lookups per host, which
>> > at 2 seconds per lookup, is taking more than 8 minutes per host
(and
>> > using near 100% of a CPU core the entire time). As we have ~40
such
>> > hosts, that''s over 5 hours with no concurrency (we do use
some
>> > concurrency, but since each invocation uses near 100% of a CPU
core
>> > each, there''s no point running more simultaneous hiera
processes
>> > than the number of CPUs, and even then, we need some reserved for
>> > other services).
>> >
>> >
>> > So, is there some way I can make this export run faster? The bulk
of
>> > the time seems to be spent in initialising the hiera command line
>> > program, not the actual data lookup (for example, if I fail to
>> > provide the necessary identity / scope files, it still takes 2
>> > seconds to get around to returning an error, but if I fail to
>> > provide a config file, it returns instantaneously with error). So,
>> > I''d either like to make the program load / initialise
much faster,
>> > or be able to query multiple values for a single hiera command
line
>> > invocation.
>> >
>> >
>> > Also, is there a more recent version of hiera that would be
faster?
>> > (I''m a little unclear about hiera versioning -
we''re running 0.3.0,
>> > but I don''t know if that''s recent or old).
>> >
>> >
>> > Any suggestions / tips would be greatly appreciated :)
>> >
>> >
>> > Thanks.
>> >
>> >
>> > Paul.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Puppet Users" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
>> > To post to this group, send email to puppet...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > puppet-users...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/puppet-users?hl=en.
>> >
>>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/zS2noBFW0ZoJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
----- Original Message -----> From: "Paul Colby" <pcolby@gmail.com> > To: puppet-users@googlegroups.com > Sent: Friday, August 24, 2012 11:03:51 AM > Subject: Re: [Puppet Users] hiera command line performance > > Thanks again R.I. Pienaar.awesome, glad you could sort it out :) pretty interesting that just requiring puppet slows it down that much -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.