Puppet users - Dec 2009 - puppet performance and inability to retrieve file metadata

Hey All,
I''m running into a problem whereby some nodes are struggling to
retrieve the metadata for files. I''ve moved all the nodes across to
0.25.1 to try and eliminate version inconsistencies though they still appear.
I''ve also started using asynchronous stored configs as well.

I''m wondering if this has anything to do with performance on the box
and if there is anything to be done to address that issue. Currently there are
roughly 100 nodes and this will end up growing.

Any advice would be greatly appreciated.

Thanks,
Matt Delves

--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

You didnt mention, but are you using webrick with 100 hosts?

Ohad

On Thu, Dec 17, 2009 at 1:12 PM, Matthew Delves
<m.delves@ballarat.edu.au>wrote:
> Hey All,
> I''m running into a problem whereby some nodes are struggling to
retrieve
> the metadata for files. I''ve moved all the nodes across to 0.25.1
to try and
> eliminate version inconsistencies though they still appear. I''ve
also
> started using asynchronous stored configs as well.
>
> I''m wondering if this has anything to do with performance on the
box and if
> there is anything to be done to address that issue. Currently there are
> roughly 100 nodes and this will end up growing.
>
> Any advice would be greatly appreciated.
>
> Thanks,
> Matt Delves
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
>
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

I''m running into a problem whereby some nodes are struggling to
retrieve the
> metadata for files. I''ve moved all the nodes across to 0.25.1 to
try and
> eliminate version inconsistencies though they still appear. I''ve
also
> started using asynchronous stored configs as well.
>
can you be specific about what metadata is not being retrieved? You mean
that parameters are not being set correctly (things like mode/owner)?

Are there errors on the client?

-Dan
>
>
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I''m using mongrel for those 100 hosts.
>>> On 17/12/2009 at 4:17 pm, Ohad Levy <ohadlevy@gmail.com>
wrote:
> You didnt mention, but are you using webrick with 100 hosts?
> 
> Ohad
> 
> On Thu, Dec 17, 2009 at 1:12 PM, Matthew Delves 
> <m.delves@ballarat.edu.au>wrote:
> 
>> Hey All,
>> I''m running into a problem whereby some nodes are struggling
to retrieve
>> the metadata for files. I''ve moved all the nodes across to
0.25.1 to try and
>> eliminate version inconsistencies though they still appear.
I''ve also
>> started using asynchronous stored configs as well.
>>
>> I''m wondering if this has anything to do with performance on
the box and if
>> there is anything to be done to address that issue. Currently there are
>> roughly 100 nodes and this will end up growing.
>>
>> Any advice would be greatly appreciated.
>>
>> Thanks,
>> Matt Delves
>>
>> --
>>
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> 
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroup
> s.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>>
>>
> 
> --
> 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/puppet-users?hl=en.
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>>> On 17/12/2009 at 4:21 pm, Dan Bode <dan@reductivelabs.com>
wrote:
> Hi,
> 
> I''m running into a problem whereby some nodes are struggling to
retrieve the
>> metadata for files. I''ve moved all the nodes across to 0.25.1
to try and
>> eliminate version inconsistencies though they still appear.
I''ve also
>> started using asynchronous stored configs as well.
>>
> 
> can you be specific about what metadata is not being retrieved? You mean
> that parameters are not being set correctly (things like mode/owner)?
> 
> Are there errors on the client?
> 
Hi Dan,
The error I''m getting is:

Failed to retrieve current state of resource: Could not retrieve file metadata
for puppet:///modules/rhnsd/rhnsd: at
/etc/puppet/modules/rhnsd/manifests/init.pp:22


I''m getting the errors for different files and there seems to be no
pattern to them.

Thanks,
Matt Delves

--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Matt,
>
> Hi Dan,
> The error I''m getting is:
>
> Failed to retrieve current state of resource: Could not retrieve file
> metadata for puppet:///modules/rhnsd/rhnsd: at
> /etc/puppet/modules/rhnsd/manifests/init.pp:22
>
> can you show the file resource that this refers to (aka: the code)
How often are the clients checking in with the server?

also, can you capture the --debug and --trace output for this error.

thanks,

Dan

--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Dan,
> How often are the clients checking in with the server?
They check in every half hour (I believe that is the default period).
> also, can you capture the --debug and --trace output for this error.
In the realm of all possibly annoying things to have happen, running it with
--debug and --trace didn''t cause the error to occur.

Thanks,
Matt Delves

--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

In addition to my previous email the output following may be helpful.

debug: Format pson not supported for Puppet::FileServing::Metadata; has not
implemented method ''from_pson''
debug: Format s not supported for Puppet::FileServing::Metadata; has not
implemented method ''from_s''


Is that most likely due to using an earlier version of ruby? Currently there is
version 1.8.1 installed on that box.

Thanks,
Matt Delves
>>> On 17/12/2009 at 4:31 pm, Dan Bode <dan@reductivelabs.com>
wrote:
> Hi Matt,
> 
>>
>> Hi Dan,
>> The error I''m getting is:
>>
>> Failed to retrieve current state of resource: Could not retrieve file
>> metadata for puppet:///modules/rhnsd/rhnsd: at
>> /etc/puppet/modules/rhnsd/manifests/init.pp:22
>>
>> can you show the file resource that this refers to (aka: the code)
> 
> How often are the clients checking in with the server?
> 
> also, can you capture the --debug and --trace output for this error.
> 
> thanks,
> 
> Dan
> 
> --
> 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/puppet-users?hl=en.
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,
>
> debug: Format pson not supported for Puppet::FileServing::Metadata; has not
> implemented method ''from_pson''
> debug: Format s not supported for Puppet::FileServing::Metadata; has not
> implemented method ''from_s''
>
> Can you open a ticket for this?
>
> Is that most likely due to using an earlier version of ruby? Currently
> there is version 1.8.1 installed on that box.
>
>
I know that there have been some issues in the past with ruby versions
earlier than 1.8.2. Is this RH4?

-Dan

--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Yeah, most of them are.
>>> On 17/12/2009 at 4:55 pm, Dan Bode <dan@reductivelabs.com>
wrote:
> Hi,
> 
>>
>> debug: Format pson not supported for Puppet::FileServing::Metadata; has
not
>> implemented method ''from_pson''
>> debug: Format s not supported for Puppet::FileServing::Metadata; has
not
>> implemented method ''from_s''
>>
>> Can you open a ticket for this?
> 
>>
>> Is that most likely due to using an earlier version of ruby? Currently
>> there is version 1.8.1 installed on that box.
>>
>>
> I know that there have been some issues in the past with ruby versions
> earlier than 1.8.2. Is this RH4?
> 
> -Dan
> 
> --
> 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/puppet-users?hl=en.
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

I have a few hundreds RHE4 running (not to mention a few hundreds of RHE3)
without having this issue.

any chance you are restarting your mongrels (e.g. because of memory leaks)?
at the same time?
do you see the request on the apache log correctly etc?

Cheers,
Ohad

On Thu, Dec 17, 2009 at 1:57 PM, Matthew Delves
<m.delves@ballarat.edu.au>wrote:
> Yeah, most of them are.
>
> >>> On 17/12/2009 at 4:55 pm, Dan Bode
<dan@reductivelabs.com> wrote:
> > Hi,
> >
> >>
> >> debug: Format pson not supported for
Puppet::FileServing::Metadata; has
> not
> >> implemented method ''from_pson''
> >> debug: Format s not supported for Puppet::FileServing::Metadata;
has not
> >> implemented method ''from_s''
> >>
> >> Can you open a ticket for this?
> >
> >>
> >> Is that most likely due to using an earlier version of ruby?
Currently
> >> there is version 1.8.1 installed on that box.
> >>
> >>
> > I know that there have been some issues in the past with ruby versions
> > earlier than 1.8.2. Is this RH4?
> >
> > -Dan
> >
> > --
> >
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> >
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
>
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Matt,

On Dec 17, 6:40 am, "Matthew Delves" <m.del...@ballarat.edu.au>
wrote:
> In addition to my previous email the output following may be helpful.
>
> debug: Format pson not supported for Puppet::FileServing::Metadata; has not
implemented method ''from_pson''
> debug: Format s not supported for Puppet::FileServing::Metadata; has not
implemented method ''from_s''
>
> Is that most likely due to using an earlier version of ruby? Currently
there is version 1.8.1 installed on that box.
>
I see the same problem on OpenSolaris with passenger - lots of reports
fail with:
Failed to retrieve current state of resource: Could not retrieve file
metadata for ...

cheers,
/Martin

--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Matt,

I followed up with engineering and they believe that your issue is related
to

http://projects.reductivelabs.com/issues/2757

This issue will be resolved in .25.2.

You may want to try upgrading ruby on one of those machines to see if it
fixes the issue.

On Wed, Dec 16, 2009 at 9:57 PM, Matthew Delves
<m.delves@ballarat.edu.au>wrote:
> Yeah, most of them are.
>
> >>> On 17/12/2009 at 4:55 pm, Dan Bode
<dan@reductivelabs.com> wrote:
> > Hi,
> >
> >>
> >> debug: Format pson not supported for
Puppet::FileServing::Metadata; has
> not
> >> implemented method ''from_pson''
> >> debug: Format s not supported for Puppet::FileServing::Metadata;
has not
> >> implemented method ''from_s''
> >>
> >> Can you open a ticket for this?
> >
> >>
> >> Is that most likely due to using an earlier version of ruby?
Currently
> >> there is version 1.8.1 installed on that box.
> >>
> >>
> > I know that there have been some issues in the past with ruby versions
> > earlier than 1.8.2. Is this RH4?
> >
> > -Dan
> >
> > --
> >
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> >
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
>
--

You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.