thr3ads.net - Puppet users - [Puppet Users] bug: recursive directory copy that re-copies nightly [Oct 2011]

If this information is useful, please help other people find it:
Share via:

Jo Rhett

2011-Oct-26 04:51 UTC

[Puppet Users] bug: recursive directory copy that re-copies nightly

I''m seeing a recurring, nightly copy of files which are not changing. 
I get the following message every night from tagmail.

1. It happens once per day (not every 30 minutes)
2. The files haven''t changed in months
3. The files aren''t changing -- there''s only one md5 in the
clientbucket for each file
4. There''s no schedule on the resource
5. The files aren''t changing on the disk.  I ran a loop which stores
the md5 of the file every minute of the day, and it never varied. For this not
to be caught, there would have to be an update of the file in the 60 seconds
before puppet updated the file, timed exactly across a hundred+ systems.
6. This isn''t affecting a single host.  It''s affecting about a
forty of more than five hundred.
7. No, there''s really nothing running on that host which updates those
files.

The resource is:

	file { ''pluginsdir'':
		ensure  => directory,
		path    => ''/usr/lib64/nagios/plugins'',
		source  => ''puppet:///modules/nagios/plugins'',
		ignore  => ''.svn'',
		recurse => true,
		replace => true,
		owner   => root,
		group   => root,
		purge   => false,
		mode    => 755,
	}

It happens every night around 9pm (4am UDT). EVERY night.

Begin forwarded message:> From: "report@thisorthat.example.com"
<report@thisorthat.example.com>
> Date: October 25, 2011 9:02:50 PM PDT
> To: puppet-info <puppet-info@example.com>
> Subject: Puppet Report for thisorthat.example.com
> 
> Wed Oct 26 04:02:35 +0000 2011
/File[/usr/lib64/nagios/plugins/check_clamd]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:36 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ftp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_http]/content (notice): content changed
''{md5}de26ab80a3fad32aada72004444221cc'' to
''{md5}161aae413e174adbe4bfe2baec5ad366''
> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_imap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_jabber]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ldap]/content (notice): content changed
''{md5}e92e0da9516c7a6243e046afe9cf17b5'' to
''{md5}c09688df1db592242d1053e17666890c''
> Wed Oct 26 04:02:39 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql]/content (notice): content changed
''{md5}477611488115efb04671c93635807047'' to
''{md5}8dfd662eedcab5390b9914fbde0bbf02''
> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql_query]/content (notice): content
changed ''{md5}1313712fb4e85d77f08704453b697a60'' to
''{md5}b82a0f414940818a58b8976f84224b43''
> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntps]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:41 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pgsql]/content (notice): content changed
''{md5}24f334cf41b6b8ec484141a3e7fc7e26'' to
''{md5}4b4a50d661b1f5ffbbba568c654da65c''
> Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_simap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_smtp]/content (notice): content changed
''{md5}32cce97e4d55ded2da4e541ebc4373ee'' to
''{md5}8ccbb28a50706c416889771444b6c084''
> Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_spop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ssmtp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_tcp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:45 +0000 2011
/File[/usr/lib64/nagios/plugins/check_udp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Ken Barber

2011-Oct-26 09:35 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Now that is curious.

What I find really strange about this - is that the checksums are the
same between _some_ of the various files. Original and resultant
checksums as well ...

But this isn''t consistent:
> Wed Oct 26 04:02:35 +0000 2011
/File[/usr/lib64/nagios/plugins/check_clamd]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:36 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ftp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_http]/content (notice): content changed
''{md5}de26ab80a3fad32aada72004444221cc'' to
''{md5}161aae413e174adbe4bfe2baec5ad366''
> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_imap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
Notice check_http is different?

Similar checksums would imply the files are the same.

What is the real md5sum of say these 4 files on the source and
destination? Can we say an ''ls -l'' of those files just for
curiosity
sake? Source and destination? I don''t know what this is - just looking
for more ''data'' :-).

What version of Puppet are you running on the master and agents? I''m
guessing you''re running the agent as a daemon correct?

ken.

On Wed, Oct 26, 2011 at 5:51 AM, Jo Rhett <jrhett@netconsonance.com>
wrote:> I''m seeing a recurring, nightly copy of files which are not
changing.  I get the following message every night from tagmail.
>
> 1. It happens once per day (not every 30 minutes)
> 2. The files haven''t changed in months
> 3. The files aren''t changing -- there''s only one md5 in
the clientbucket for each file
> 4. There''s no schedule on the resource
> 5. The files aren''t changing on the disk.  I ran a loop which
stores the md5 of the file every minute of the day, and it never varied. For
this not to be caught, there would have to be an update of the file in the 60
seconds before puppet updated the file, timed exactly across a hundred+ systems.
> 6. This isn''t affecting a single host.  It''s affecting
about a forty of more than five hundred.
> 7. No, there''s really nothing running on that host which updates
those files.
>
> The resource is:
>
>        file { ''pluginsdir'':
>                ensure  => directory,
>                path    => ''/usr/lib64/nagios/plugins'',
>                source  =>
''puppet:///modules/nagios/plugins'',
>                ignore  => ''.svn'',
>                recurse => true,
>                replace => true,
>                owner   => root,
>                group   => root,
>                purge   => false,
>                mode    => 755,
>        }
>
> It happens every night around 9pm (4am UDT). EVERY night.
>
> Begin forwarded message:
>> From: "report@thisorthat.example.com"
<report@thisorthat.example.com>
>> Date: October 25, 2011 9:02:50 PM PDT
>> To: puppet-info <puppet-info@example.com>
>> Subject: Puppet Report for thisorthat.example.com
>>
>> Wed Oct 26 04:02:35 +0000 2011
/File[/usr/lib64/nagios/plugins/check_clamd]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:36 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ftp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_http]/content (notice): content changed
''{md5}de26ab80a3fad32aada72004444221cc'' to
''{md5}161aae413e174adbe4bfe2baec5ad366''
>> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_imap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_jabber]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ldap]/content (notice): content changed
''{md5}e92e0da9516c7a6243e046afe9cf17b5'' to
''{md5}c09688df1db592242d1053e17666890c''
>> Wed Oct 26 04:02:39 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql]/content (notice): content changed
''{md5}477611488115efb04671c93635807047'' to
''{md5}8dfd662eedcab5390b9914fbde0bbf02''
>> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql_query]/content (notice): content
changed ''{md5}1313712fb4e85d77f08704453b697a60'' to
''{md5}b82a0f414940818a58b8976f84224b43''
>> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntps]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:41 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pgsql]/content (notice): content changed
''{md5}24f334cf41b6b8ec484141a3e7fc7e26'' to
''{md5}4b4a50d661b1f5ffbbba568c654da65c''
>> Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_simap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_smtp]/content (notice): content changed
''{md5}32cce97e4d55ded2da4e541ebc4373ee'' to
''{md5}8ccbb28a50706c416889771444b6c084''
>> Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_spop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ssmtp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_tcp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> Wed Oct 26 04:02:45 +0000 2011
/File[/usr/lib64/nagios/plugins/check_udp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>
> --
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
other randomness
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

madAndroid

2011-Oct-26 12:58 UTC

head link

[Puppet Users] Re: bug: recursive directory copy that re-copies nightly

very odd problem!

is it possible that an nrpe check is being executed at 9pm which is
changing the atime on the plugin directory thereby causing the agent
to check the Md5s? Does your nagios box run any specific checks on
those affected nodes at the same time you''re seeing the puppet
oddness?

Cheers,
Andrew


On Oct 26, 12:51 am, Jo Rhett <jrh...@netconsonance.com>
wrote:> I''m seeing a recurring, nightly copy of files which are not
changing.  I get the following message every night from tagmail.
>
> 1. It happens once per day (not every 30 minutes)
> 2. The files haven''t changed in months
> 3. The files aren''t changing -- there''s only one md5 in
the clientbucket for each file
> 4. There''s no schedule on the resource
> 5. The files aren''t changing on the disk.  I ran a loop which
stores the md5 of the file every minute of the day, and it never varied. For
this not to be caught, there would have to be an update of the file in the 60
seconds before puppet updated the file, timed exactly across a hundred+ systems.
> 6. This isn''t affecting a single host.  It''s affecting
about a forty of more than five hundred.
> 7. No, there''s really nothing running on that host which updates
those files.
>
> The resource is:
>
>         file { ''pluginsdir'':
>                 ensure  => directory,
>                 path    =>
''/usr/lib64/nagios/plugins'',
>                 source  =>
''puppet:///modules/nagios/plugins'',
>                 ignore  => ''.svn'',
>                 recurse => true,
>                 replace => true,
>                 owner   => root,
>                 group   => root,
>                 purge   => false,
>                 mode    => 755,
>         }
>
> It happens every night around 9pm (4am UDT). EVERY night.
>
> Begin forwarded message:
>
>
>
>
>
>
>
>
>
> > From: "rep...@thisorthat.example.com"
<rep...@thisorthat.example.com>
> > Date: October 25, 2011 9:02:50 PM PDT
> > To: puppet-info <puppet-i...@example.com>
> > Subject: Puppet Report for thisorthat.example.com
>
> > Wed Oct 26 04:02:35 +0000 2011
/File[/usr/lib64/nagios/plugins/check_clamd]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:36 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ftp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_http]/content (notice): content changed
''{md5}de26ab80a3fad32aada72004444221cc'' to
''{md5}161aae413e174adbe4bfe2baec5ad366''
> > Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_imap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_jabber]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ldap]/content (notice): content changed
''{md5}e92e0da9516c7a6243e046afe9cf17b5'' to
''{md5}c09688df1db592242d1053e17666890c''
> > Wed Oct 26 04:02:39 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql]/content (notice): content changed
''{md5}477611488115efb04671c93635807047'' to
''{md5}8dfd662eedcab5390b9914fbde0bbf02''
> > Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql_query]/content (notice): content
changed ''{md5}1313712fb4e85d77f08704453b697a60'' to
''{md5}b82a0f414940818a58b8976f84224b43''
> > Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntps]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:41 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pgsql]/content (notice): content changed
''{md5}24f334cf41b6b8ec484141a3e7fc7e26'' to
''{md5}4b4a50d661b1f5ffbbba568c654da65c''
> > Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_simap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_smtp]/content (notice): content changed
''{md5}32cce97e4d55ded2da4e541ebc4373ee'' to
''{md5}8ccbb28a50706c416889771444b6c084''
> > Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_spop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ssmtp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_tcp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
> > Wed Oct 26 04:02:45 +0000 2011
/File[/usr/lib64/nagios/plugins/check_udp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>
> --
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
other randomness
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Doug Warner

2011-Oct-26 14:23 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

On 10/26/2011 12:51 AM, Jo Rhett wrote:> It happens every night around 9pm (4am UDT). EVERY night.
4am, so cron.daily?

prelink?

-Doug

Jo Rhett

2011-Oct-27 04:15 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Given that the check type is md5 by default, the nrpe test would have to change
the md5 of the files involved.  We have lots of NRPE, but every one runs no less
often than every 5 minutes.

On Oct 26, 2011, at 5:58 AM, madAndroid wrote:> very odd problem!
> 
> is it possible that an nrpe check is being executed at 9pm which is
> changing the atime on the plugin directory thereby causing the agent
> to check the Md5s? Does your nagios box run any specific checks on
> those affected nodes at the same time you''re seeing the puppet
> oddness?
> 
> Cheers,
> Andrew
> 
> 
> On Oct 26, 12:51 am, Jo Rhett <jrh...@netconsonance.com> wrote:
>> I''m seeing a recurring, nightly copy of files which are not
changing.  I get the following message every night from tagmail.
>> 
>> 1. It happens once per day (not every 30 minutes)
>> 2. The files haven''t changed in months
>> 3. The files aren''t changing -- there''s only one md5
in the clientbucket for each file
>> 4. There''s no schedule on the resource
>> 5. The files aren''t changing on the disk.  I ran a loop which
stores the md5 of the file every minute of the day, and it never varied. For
this not to be caught, there would have to be an update of the file in the 60
seconds before puppet updated the file, timed exactly across a hundred+ systems.
>> 6. This isn''t affecting a single host.  It''s
affecting about a forty of more than five hundred.
>> 7. No, there''s really nothing running on that host which
updates those files.
>> 
>> The resource is:
>> 
>>         file { ''pluginsdir'':
>>                 ensure  => directory,
>>                 path    =>
''/usr/lib64/nagios/plugins'',
>>                 source  =>
''puppet:///modules/nagios/plugins'',
>>                 ignore  => ''.svn'',
>>                 recurse => true,
>>                 replace => true,
>>                 owner   => root,
>>                 group   => root,
>>                 purge   => false,
>>                 mode    => 755,
>>         }
>> 
>> It happens every night around 9pm (4am UDT). EVERY night.
>> 
>> Begin forwarded message:
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> From: "rep...@thisorthat.example.com"
<rep...@thisorthat.example.com>
>>> Date: October 25, 2011 9:02:50 PM PDT
>>> To: puppet-info <puppet-i...@example.com>
>>> Subject: Puppet Report for thisorthat.example.com
>> 
>>> Wed Oct 26 04:02:35 +0000 2011
/File[/usr/lib64/nagios/plugins/check_clamd]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:36 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ftp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_http]/content (notice): content changed
''{md5}de26ab80a3fad32aada72004444221cc'' to
''{md5}161aae413e174adbe4bfe2baec5ad366''
>>> Wed Oct 26 04:02:37 +0000 2011
/File[/usr/lib64/nagios/plugins/check_imap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_jabber]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:38 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ldap]/content (notice): content changed
''{md5}e92e0da9516c7a6243e046afe9cf17b5'' to
''{md5}c09688df1db592242d1053e17666890c''
>>> Wed Oct 26 04:02:39 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql]/content (notice): content changed
''{md5}477611488115efb04671c93635807047'' to
''{md5}8dfd662eedcab5390b9914fbde0bbf02''
>>> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_mysql_query]/content (notice): content
changed ''{md5}1313712fb4e85d77f08704453b697a60'' to
''{md5}b82a0f414940818a58b8976f84224b43''
>>> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:40 +0000 2011
/File[/usr/lib64/nagios/plugins/check_nntps]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:41 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pgsql]/content (notice): content changed
''{md5}24f334cf41b6b8ec484141a3e7fc7e26'' to
''{md5}4b4a50d661b1f5ffbbba568c654da65c''
>>> Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_pop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:42 +0000 2011
/File[/usr/lib64/nagios/plugins/check_simap]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_smtp]/content (notice): content changed
''{md5}32cce97e4d55ded2da4e541ebc4373ee'' to
''{md5}8ccbb28a50706c416889771444b6c084''
>>> Wed Oct 26 04:02:43 +0000 2011
/File[/usr/lib64/nagios/plugins/check_spop]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_ssmtp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:44 +0000 2011
/File[/usr/lib64/nagios/plugins/check_tcp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>>> Wed Oct 26 04:02:45 +0000 2011
/File[/usr/lib64/nagios/plugins/check_udp]/content (notice): content changed
''{md5}59aae3c31ce1b664af6269e01a321688'' to
''{md5}b17975a3843ca819c0e22df11b8626c5''
>> 
>> --
>> Jo Rhett
>> Net Consonance : consonant endings by net philanthropy, open source and
other randomness
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jo Rhett

2011-Oct-27 04:21 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Thank you Doug, for the clue-by-four.  I overlooked that.  I owe you a beer.

On Oct 26, 2011, at 7:23 AM, Doug Warner wrote:> On 10/26/2011 12:51 AM, Jo Rhett wrote:
>> It happens every night around 9pm (4am UDT). EVERY night.
> 
> 4am, so cron.daily?
> 
> prelink?
> 
> -Doug
> 
-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Felix Frank

2011-Oct-27 11:19 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

On 10/27/2011 06:21 AM, Jo Rhett wrote:> Thank you Doug, for the clue-by-four.  I overlooked that.  I owe you a
beer.
I don''t get it. What''s precisely causing this issue then?
> On Oct 26, 2011, at 7:23 AM, Doug Warner wrote:
>> On 10/26/2011 12:51 AM, Jo Rhett wrote:
>>> It happens every night around 9pm (4am UDT). EVERY night.
>>
>> 4am, so cron.daily?
>>
>> prelink?
>>
>> -Doug
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Ken Barber

2011-Oct-27 11:31 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Yeah - I''m curious to see if/how you solved this as well :-).

On Thu, Oct 27, 2011 at 12:19 PM, Felix Frank
<felix.frank@alumni.tu-berlin.de> wrote:> On 10/27/2011 06:21 AM, Jo Rhett wrote:
>> Thank you Doug, for the clue-by-four.  I overlooked that.  I owe you a
beer.
>
> I don''t get it. What''s precisely causing this issue then?
>
>> On Oct 26, 2011, at 7:23 AM, Doug Warner wrote:
>>> On 10/26/2011 12:51 AM, Jo Rhett wrote:
>>>> It happens every night around 9pm (4am UDT). EVERY night.
>>>
>>> 4am, so cron.daily?
>>>
>>> prelink?
>>>
>>> -Doug
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jo Rhett

2011-Oct-31 17:15 UTC

head link

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

"man pre link" will tell you:
       prelink  is  a program that modifies ELF shared libraries and ELF
dynamically linked binaries in such a
       way that the time needed for the  dynamic  linker  to  perform 
relocations  at  startup  significantly
       decreases.

In short, don''t use recursive copy to duplicate ELF binaries, or exempt
them from pre link in some fashion. In our case it was determined that only an
initial copy was necessary at this time.

On Oct 26, 2011, at 9:21 PM, Jo Rhett wrote:> Thank you Doug, for the clue-by-four.  I overlooked that.  I owe you a
beer.
> 
> On Oct 26, 2011, at 7:23 AM, Doug Warner wrote:
>> On 10/26/2011 12:51 AM, Jo Rhett wrote:
>>> It happens every night around 9pm (4am UDT). EVERY night.
>> 
>> 4am, so cron.daily?
>> 
>> prelink?
>> 
>> -Doug
>> 
> 
> -- 
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
other randomness
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Apparently Analagous Threads

Search for more reasonably related threads

Puppet users - Oct 2011 - bug: recursive directory copy that re-copies nightly

[Puppet Users] bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

[Puppet Users] Re: bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Re: [Puppet Users] bug: recursive directory copy that re-copies nightly

Apparently Analagous Threads