thr3ads.net - Puppet users - [Puppet Users] Explanation of the metric section of a report [Jul 2011]

If this information is useful, please help other people find it:
Share via:

Haitao Jiang

2011-Jul-08 22:00 UTC

[Puppet Users] Explanation of the metric section of a report

I just wondering if anyone can point me to the documentation on the
metric section of a report. I have a slow agent run which has
following numbers:

Config Retrieval 	10.43 seconds
Exec 	0.00 seconds
File 	157.11 seconds
Filebucket 	0.00 seconds
Package 	0.07 seconds
Schedule 	0.00 seconds
Total 	167.62 seconds

I would like to know meaning of above in order to find out the reasons
of slowness. My guess is it was due to the network latency, but just
want to make sure.

Thanks

Haitao

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel Kersten

2011-Jul-09 16:28 UTC

head link

Re: [Puppet Users] Explanation of the metric section of a report

On Fri, Jul 8, 2011 at 3:00 PM, Haitao Jiang <jianghaitao@gmail.com>
wrote:
> I just wondering if anyone can point me to the documentation on the
> metric section of a report. I have a slow agent run which has
> following numbers:
>
> Config Retrieval        10.43 seconds
> Exec    0.00 seconds
> File    157.11 seconds
> Filebucket      0.00 seconds
> Package         0.07 seconds
> Schedule        0.00 seconds
> Total   167.62 seconds
>
> I would like to know meaning of above in order to find out the reasons
> of slowness. My guess is it was due to the network latency, but just
> want to make sure.
>
It''s telling you that it only took 10 seconds to talk to the server and
retrieve the catalog, and almost all your time was spent in File resources.

File resources impose a different kind of load upon the server, as when they
have a remote source the client needs to request file metadata in order to
compare locally and determine whether or not the file contents need to be
retrieved from the server.

If you have lots of File resources, multiple concurrent clients, and are
still running with the webrick Puppet master, you''ll see performance
degradation like this.

If you have deep recursive directories in a File resource, you''ll run
into
similar issues.

If you have very large files, the default checksumming will take a while.

There are a few parameters that are designed to help with these cases.

http://docs.puppetlabs.com/references/stable/type.html#file

Have a look at recurse and checksum.

-- 
Nigel Kersten
Product Manager, Puppet Labs
Twitter: @nigelkersten

*Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig>
September 22nd and 23rd in Portland, Oregon, USA.
*
*

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Luke Bigum

2011-Jul-11 09:00 UTC

head link

[Puppet Users] Re: Explanation of the metric section of a report

"puppetd --test --evaltrace" may also tell you which resources are
taking longer than others, narrowing down the problem. I don''t have
any large manifests at the moment so I''m not sure if evaluation time
is the same as ''do all the work'' time.

On Jul 9, 5:28 pm, Nigel Kersten <ni...@puppetlabs.com>
wrote:> On Fri, Jul 8, 2011 at 3:00 PM, Haitao Jiang <jianghai...@gmail.com>
wrote:
> > I just wondering if anyone can point me to the documentation on the
> > metric section of a report. I have a slow agent run which has
> > following numbers:
>
> > Config Retrieval        10.43 seconds
> > Exec    0.00 seconds
> > File    157.11 seconds
> > Filebucket      0.00 seconds
> > Package         0.07 seconds
> > Schedule        0.00 seconds
> > Total   167.62 seconds
>
> > I would like to know meaning of above in order to find out the reasons
> > of slowness. My guess is it was due to the network latency, but just
> > want to make sure.
>
> It''s telling you that it only took 10 seconds to talk to the
server and
> retrieve the catalog, and almost all your time was spent in File resources.
>
> File resources impose a different kind of load upon the server, as when
they
> have a remote source the client needs to request file metadata in order to
> compare locally and determine whether or not the file contents need to be
> retrieved from the server.
>
> If you have lots of File resources, multiple concurrent clients, and are
> still running with the webrick Puppet master, you''ll see
performance
> degradation like this.
>
> If you have deep recursive directories in a File resource, you''ll
run into
> similar issues.
>
> If you have very large files, the default checksumming will take a while.
>
> There are a few parameters that are designed to help with these cases.
>
> http://docs.puppetlabs.com/references/stable/type.html#file
>
> Have a look at recurse and checksum.
>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
> Twitter: @nigelkersten
>
> *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig>
> September 22nd and 23rd in Portland, Oregon, USA.
> *
> *
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Haitao Jiang

2011-Jul-11 17:08 UTC

head link

Re: [Puppet Users] Explanation of the metric section of a report

Nigel

Thanks for your reply. I used --evaltrace suggested by Luke and found
that all my time spent on File resources, these are jar files that are
MB in size.

Further question:

- If Puppet just check the file metadata before decided on whether to
download the file from master or not, why it took 10s to do that? I
did not have recursive directories nor checksum in my File resources;

- Can we specify which file metadata to check, say, just the ctime?

- What are the recommended way to improve the performance here? I only
have 3 nodes to manage so far.

Thanks again

Haitao

On Sat, Jul 9, 2011 at 9:28 AM, Nigel Kersten <nigel@puppetlabs.com>
wrote:>
>
> On Fri, Jul 8, 2011 at 3:00 PM, Haitao Jiang <jianghaitao@gmail.com>
wrote:
>>
>> I just wondering if anyone can point me to the documentation on the
>> metric section of a report. I have a slow agent run which has
>> following numbers:
>>
>> Config Retrieval        10.43 seconds
>> Exec    0.00 seconds
>> File    157.11 seconds
>> Filebucket      0.00 seconds
>> Package         0.07 seconds
>> Schedule        0.00 seconds
>> Total   167.62 seconds
>>
>> I would like to know meaning of above in order to find out the reasons
>> of slowness. My guess is it was due to the network latency, but just
>> want to make sure.
>
> It''s telling you that it only took 10 seconds to talk to the
server and
> retrieve the catalog, and almost all your time was spent in File resources.
> File resources impose a different kind of load upon the server, as when
they
> have a remote source the client needs to request file metadata in order to
> compare locally and determine whether or not the file contents need to be
> retrieved from the server.
> If you have lots of File resources, multiple concurrent clients, and are
> still running with the webrick Puppet master, you''ll see
performance
> degradation like this.
> If you have deep recursive directories in a File resource, you''ll
run into
> similar issues.
> If you have very large files, the default checksumming will take a while.
> There are a few parameters that are designed to help with these cases.
> http://docs.puppetlabs.com/references/stable/type.html#file
> Have a look at recurse and checksum.
>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
> Twitter: @nigelkersten
>
> Join us for PuppetConf <http://www.bit.ly/puppetconfsig>
> September 22nd and 23rd in Portland, Oregon, USA.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel Kersten

2011-Jul-11 17:29 UTC

head link

Re: [Puppet Users] Explanation of the metric section of a report

On Mon, Jul 11, 2011 at 10:08 AM, Haitao Jiang
<jianghaitao@gmail.com>wrote:
> Nigel
>
> Thanks for your reply. I used --evaltrace suggested by Luke and found
> that all my time spent on File resources, these are jar files that are
> MB in size.
>
> Further question:
>
> - If Puppet just check the file metadata before decided on whether to
> download the file from master or not, why it took 10s to do that? I
> did not have recursive directories nor checksum in my File resources;
>
Because they''re large files, the checksumming at both ends takes a
reasonable amount of time.

>
> - Can we specify which file metadata to check, say, just the ctime?
>
Absolutely:

http://docs.puppetlabs.com/references/stable/type.html#file
CHECKSUM

The checksum type to use when checksumming a file.

The default checksum parameter, if checksums are enabled, is md5. Valid
values are md5, md5lite, mtime,ctime, none.
depending upon your deployment, md5lite or mtime/ctime might be appropriate
checksums.


>
> - What are the recommended way to improve the performance here? I only
> have 3 nodes to manage so far.
>
I generally recommend packaging large files like jars.

>
> Thanks again
>
> Haitao
>
> On Sat, Jul 9, 2011 at 9:28 AM, Nigel Kersten <nigel@puppetlabs.com>
> wrote:
> >
> >
> > On Fri, Jul 8, 2011 at 3:00 PM, Haitao Jiang
<jianghaitao@gmail.com>
> wrote:
> >>
> >> I just wondering if anyone can point me to the documentation on
the
> >> metric section of a report. I have a slow agent run which has
> >> following numbers:
> >>
> >> Config Retrieval        10.43 seconds
> >> Exec    0.00 seconds
> >> File    157.11 seconds
> >> Filebucket      0.00 seconds
> >> Package         0.07 seconds
> >> Schedule        0.00 seconds
> >> Total   167.62 seconds
> >>
> >> I would like to know meaning of above in order to find out the
reasons
> >> of slowness. My guess is it was due to the network latency, but
just
> >> want to make sure.
> >
> > It''s telling you that it only took 10 seconds to talk to the
server and
> > retrieve the catalog, and almost all your time was spent in File
> resources.
> > File resources impose a different kind of load upon the server, as
when
> they
> > have a remote source the client needs to request file metadata in
order
> to
> > compare locally and determine whether or not the file contents need to
be
> > retrieved from the server.
> > If you have lots of File resources, multiple concurrent clients, and
are
> > still running with the webrick Puppet master, you''ll see
performance
> > degradation like this.
> > If you have deep recursive directories in a File resource,
you''ll run
> into
> > similar issues.
> > If you have very large files, the default checksumming will take a
while.
> > There are a few parameters that are designed to help with these cases.
> > http://docs.puppetlabs.com/references/stable/type.html#file
> > Have a look at recurse and checksum.
> >
> > --
> > Nigel Kersten
> > Product Manager, Puppet Labs
> > Twitter: @nigelkersten
> >
> > Join us for PuppetConf <http://www.bit.ly/puppetconfsig>
> > September 22nd and 23rd in Portland, Oregon, USA.
> >
> > --
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
Nigel Kersten
Product Manager, Puppet Labs
Twitter: @nigelkersten

*Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig>
September 22nd and 23rd in Portland, Oregon, USA.
*
*

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Haitao Jiang

2011-Jul-11 17:44 UTC

head link

Re: [Puppet Users] Explanation of the metric section of a report

Nigel

Thanks a lot! Yes, I did find other options of checksum.

However, after I changed the checksum type to mtime for jetty jar,
Puppet now gave me error and can not recover itself:

info: /Stage[main]/Emb/File[/opt/jetty-distribution-7.4.2.v20110526.tar.gz]:
Filebucketed /opt/jetty-distribution-7.4.2.v20110526.tar.gz to puppet
with sum 3dc95fb231fdabbe161d8773e2a10457
err:
/Stage[main]/Emb/File[/opt/jetty-distribution-7.4.2.v20110526.tar.gz]/content:
change from {mtime}Thu Jul 07 10:44:29 -0700 2011 to
{md5}3dc95fb231fdabbe161d8773e2a10457 failed: Could not rename
temporary file /opt/jetty-distribution-7.4.2.v20110526.tar.gz.puppettmp_7191
to /opt/jetty-distribution-7.4.2.v20110526.tar.gz: File written to
disk did not match checksum; discarding changes ({mtime} vs {mtime}Mon
Jul 11 10:30:50 -0700 2011) at
/etc/puppet/modules/emb/manifests/init.pp:38 at
/etc/puppet/modules/emb/manifests/init.pp:38

Remove the file itself on the client did not help neither. How to
reset the checksum in this case?

Also, I am curious why a 2K small file also took 10s to md5 checksum:
info: /Stage[main]/Emb/File[/opt/jetty/contexts/emb-context.xml]:
Evaluated in 10.06 seconds

Thanks again

Haitao

On Mon, Jul 11, 2011 at 10:29 AM, Nigel Kersten <nigel@puppetlabs.com>
wrote:>
>
> On Mon, Jul 11, 2011 at 10:08 AM, Haitao Jiang
<jianghaitao@gmail.com>
> wrote:
>>
>> Nigel
>>
>> Thanks for your reply. I used --evaltrace suggested by Luke and found
>> that all my time spent on File resources, these are jar files that are
>> MB in size.
>>
>> Further question:
>>
>> - If Puppet just check the file metadata before decided on whether to
>> download the file from master or not, why it took 10s to do that? I
>> did not have recursive directories nor checksum in my File resources;
>
> Because they''re large files, the checksumming at both ends takes a
> reasonable amount of time.
>
>>
>> - Can we specify which file metadata to check, say, just the ctime?
>
> Absolutely:
> http://docs.puppetlabs.com/references/stable/type.html#file
>
> CHECKSUM
>
> The checksum type to use when checksumming a file.
>
> The default checksum parameter, if checksums are enabled, is md5. Valid
> values are md5, md5lite, mtime,ctime, none.
>
> depending upon your deployment, md5lite or mtime/ctime might be appropriate
> checksums.
>
>>
>> - What are the recommended way to improve the performance here? I only
>> have 3 nodes to manage so far.
>
> I generally recommend packaging large files like jars.
>
>>
>> Thanks again
>>
>> Haitao
>>
>> On Sat, Jul 9, 2011 at 9:28 AM, Nigel Kersten
<nigel@puppetlabs.com>
>> wrote:
>> >
>> >
>> > On Fri, Jul 8, 2011 at 3:00 PM, Haitao Jiang
<jianghaitao@gmail.com>
>> > wrote:
>> >>
>> >> I just wondering if anyone can point me to the documentation
on the
>> >> metric section of a report. I have a slow agent run which has
>> >> following numbers:
>> >>
>> >> Config Retrieval        10.43 seconds
>> >> Exec    0.00 seconds
>> >> File    157.11 seconds
>> >> Filebucket      0.00 seconds
>> >> Package         0.07 seconds
>> >> Schedule        0.00 seconds
>> >> Total   167.62 seconds
>> >>
>> >> I would like to know meaning of above in order to find out the
reasons
>> >> of slowness. My guess is it was due to the network latency,
but just
>> >> want to make sure.
>> >
>> > It''s telling you that it only took 10 seconds to talk to
the server and
>> > retrieve the catalog, and almost all your time was spent in File
>> > resources.
>> > File resources impose a different kind of load upon the server, as
when
>> > they
>> > have a remote source the client needs to request file metadata in
order
>> > to
>> > compare locally and determine whether or not the file contents
need to
>> > be
>> > retrieved from the server.
>> > If you have lots of File resources, multiple concurrent clients,
and are
>> > still running with the webrick Puppet master, you''ll see
performance
>> > degradation like this.
>> > If you have deep recursive directories in a File resource,
you''ll run
>> > into
>> > similar issues.
>> > If you have very large files, the default checksumming will take a
>> > while.
>> > There are a few parameters that are designed to help with these
cases.
>> > http://docs.puppetlabs.com/references/stable/type.html#file
>> > Have a look at recurse and checksum.
>> >
>> > --
>> > Nigel Kersten
>> > Product Manager, Puppet Labs
>> > Twitter: @nigelkersten
>> >
>> > Join us for PuppetConf <http://www.bit.ly/puppetconfsig>
>> > September 22nd and 23rd in Portland, Oregon, USA.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Puppet Users" group.
>> > To post to this group, send email to
puppet-users@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > puppet-users+unsubscribe@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/puppet-users?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
>
>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
> Twitter: @nigelkersten
>
> Join us for PuppetConf <http://www.bit.ly/puppetconfsig>
> September 22nd and 23rd in Portland, Oregon, USA.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Puppet users - Jul 2011 - Explanation of the metric section of a report

[Puppet Users] Explanation of the metric section of a report

Re: [Puppet Users] Explanation of the metric section of a report

[Puppet Users] Re: Explanation of the metric section of a report

Re: [Puppet Users] Explanation of the metric section of a report

Re: [Puppet Users] Explanation of the metric section of a report

Re: [Puppet Users] Explanation of the metric section of a report