Puppet users - Sep 2006 - Re: strange mtime/md5sum behaviour and constantly changing files with links

On Sep 8, 2006, at 5:58 AM, Thorsten Sandfuchs wrote:
> hio,
> as I have some huge files to distribute and md5-sum-checking takes  
> SOME time
> for them, I''m trying to switch to mtime/timestamps, but got no  
> luck. Regularly
> the mtime seems to change and additionally the md5sums generate  
> themselves
> anyway.
I''m pretty sure that the file copying subsystem always uses checksums  
regardless of what you ask.  This is a bug.  I thought I''d developed  
it to support any of them, but realized recently I had not.

Can you submit this as a bug?  It''ll take a decent amount of work to  
fix, since both the client and server will need to be fixed, but it''s  
definitely worth it.

As to the constantly changing timestamps...  I expect that it''s  
something happening internally because you want to use timestamps but  
Puppet wants to use checksums.  If we could fix the file-copying,  
we''d probably fix this.

-- 
Luke Kanies
http://madstop.com | http://reductivelabs.com | 615-594-8199

On Wed, 13 Sep 2006, Luke Kanies wrote:
> As to the constantly changing timestamps...  I expect that it''s  
> something happening internally because you want to use timestamps but  
> Puppet wants to use checksums.  If we could fix the file-copying,  
> we''d probably fix this.
I don''t know if this is related, but now that there''s
prelinking
enabled on RHEL (and presumably other Linuxes as well), md5sums are no 
longer a reliable measure of file similarity.

[eric@op240-002 ~]$ md5sum /bin/ls
219cefdb72e4b9ae6e543f579a3401b3  /bin/ls
[eric@op240-002 ~]$ ls -l /bin/ls
-rwxr-xr-x  1 root root 87608 Oct  7  2005 /bin/ls

[eric@op240-001 ~]$ md5sum /bin/ls
e2b03dfe637c506ed578ee5ce27ebe69  /bin/ls
[eric@op240-001 ~]$ ls -l /bin/ls
-rwxr-xr-x  1 root root 87608 Oct  7  2005 /bin/ls

Here''s why: 
http://linux.derkeiler.com/Mailing-Lists/RedHat/2006-03/msg00151.html

Again, not sure if it''s applicable but we got bit by this last week, 
and it seems to not be widely known.

-- 
 - Eric Sorenson - N37 17.255 W121 55.738  - http://ahpook.vox.com/   -
 - Personal colo with a professional touch - http://www.explosive.net -

On Sep 13, 2006, at 1:21 PM, Eric Sorenson wrote:
>
> I don''t know if this is related, but now that there''s
prelinking
> enabled on RHEL (and presumably other Linuxes as well), md5sums are no
> longer a reliable measure of file similarity.
[SNIP]
> Again, not sure if it''s applicable but we got bit by this last
week,
> and it seems to not be widely known.
I would think it''s unlikely people would be distributing binaries  
this way, but that certainly doesn''t mean someone won''t run
into it.

Is there any way around the problem?  I assume not, but it''s good to  
know about it.

-- 
Luke Kanies
http://madstop.com | http://reductivelabs.com | 615-594-8199

On Wed, 13 Sep 2006, Luke Kanies wrote:
> I would think it''s unlikely people would be distributing binaries
> this way, but that certainly doesn''t mean someone won''t
run into it.
Yup, my first cfengine site propagated our /usr/local/bin/ tree 
via cfengine 1.5. Not a wonderful solution but it worked well enough.
 
> Is there any way around the problem?  I assume not, but it''s good
to
> know about it.
Check out Section 13 of Jakub Jelínek''s prelink paper:
http://people.redhat.com/jakub/prelink.pdf

I think you''d have to detect if prelink was enabled (/etc/prelink.conf 
exists) and if so, use the output of ''prelink --verify'' as the
basis
for your comparison instead of the in-place binary. 

       -y --verify

        Verifies  a  prelinked binary or library.  This option can be
        used only on a single binary or library. It first applies an --undo
operation
        on the file, then prelinks just that file again and  compares  this 
with  the
        original file. If both are identical, it prints the file after --undo
operation
        on standard output and exit with zero status. Otherwise it exits with 
error
        status.  Thus if --verify operation returns zero exit status and its
standard
        output is equal to the content of the binary or library before
prelinking, you
        can be sure that nobody  modi- fied  the  binaries or libraries after
        prelinking. 

Neat stuff actually, and though I haven''t done any scientifically
valid before-and-afters, it does seem to speed things up.

-- 
 - Eric Sorenson - N37 17.255 W121 55.738  - http://ahpook.vox.com/   -
 - Personal colo with a professional touch - http://www.explosive.net -
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Sep 13, 2006, at 4:37 PM, Eric Sorenson wrote:
>
> Yup, my first cfengine site propagated our /usr/local/bin/ tree
> via cfengine 1.5. Not a wonderful solution but it worked well enough.
Yeah, I''ve done the same, but I think it clearly qualifies as abuse. :)
> Check out Section 13 of Jakub Jelínek''s prelink paper:
> http://people.redhat.com/jakub/prelink.pdf
>
> I think you''d have to detect if prelink was enabled
(/etc/prelink.conf
> exists) and if so, use the output of ''prelink --verify''
as the basis
> for your comparison instead of the in-place binary.
>
>        -y --verify
>
>         Verifies  a  prelinked binary or library.  This option can be
>         used only on a single binary or library. It first applies  
> an --undo operation
>         on the file, then prelinks just that file again and   
> compares  this  with  the
>         original file. If both are identical, it prints the file  
> after --undo operation
>         on standard output and exit with zero status. Otherwise it  
> exits with  error
>         status.  Thus if --verify operation returns zero exit  
> status and its standard
>         output is equal to the content of the binary or library  
> before prelinking, you
>         can be sure that nobody  modi- fied  the  binaries or  
> libraries after
>         prelinking.
>
> Neat stuff actually, and though I haven''t done any scientifically
> valid before-and-afters, it does seem to speed things up.
Until someone complains about this, I''m going to avoid it like the  
plague.  That sounds like a pretty expensive operation to do every  
time I need a simple md5 sum.

-- 
Luke Kanies
http://madstop.com | http://reductivelabs.com | 615-594-8199

On Thu, Sep 14, 2006 at 01:21:25AM -0500, Luke Kanies wrote:
> Until someone complains about this, I''m going to avoid it like the
> plague.  That sounds like a pretty expensive operation to do every  
> time I need a simple md5 sum.
It is, if someone really wants this at some point in the future I
suggest to have something like "md5prelink" as a checksum argument
with
warnings in the docs that maybe they should rethink what they are doing.

I suspect that nobody will care about it though.

Note that for example under Fedora only binaries/libraries in the following
directories are going to be prelinked.
$ cat /etc/prelink.conf
...
-l /bin
-l /usr/bin
-l /sbin
-l /usr/sbin
-l /usr/X11R6/bin
-l /usr/kerberos/bin
-l /usr/games
-l /usr/libexec
-l /var/ftp/bin
-l /lib{,64}
-l /usr/lib{,64}
-l /usr/X11R6/lib{,64}
-l /usr/kerberos/lib{,64}
-l /usr/X11R6/LessTif
-l /var/ftp/lib{,64}

so you can still use /usr/local/bin without problems.

Kostas