Luke Kanies
2006-Sep-13 18:16 UTC
Re: strange mtime/md5sum behaviour and constantly changing files with links
On Sep 8, 2006, at 5:58 AM, Thorsten Sandfuchs wrote:> hio, > as I have some huge files to distribute and md5-sum-checking takes > SOME time > for them, I''m trying to switch to mtime/timestamps, but got no > luck. Regularly > the mtime seems to change and additionally the md5sums generate > themselves > anyway.I''m pretty sure that the file copying subsystem always uses checksums regardless of what you ask. This is a bug. I thought I''d developed it to support any of them, but realized recently I had not. Can you submit this as a bug? It''ll take a decent amount of work to fix, since both the client and server will need to be fixed, but it''s definitely worth it. As to the constantly changing timestamps... I expect that it''s something happening internally because you want to use timestamps but Puppet wants to use checksums. If we could fix the file-copying, we''d probably fix this. -- Luke Kanies http://madstop.com | http://reductivelabs.com | 615-594-8199
Eric Sorenson
2006-Sep-13 18:21 UTC
Re: strange mtime/md5sum behaviour and constantly changing files with links
On Wed, 13 Sep 2006, Luke Kanies wrote:> As to the constantly changing timestamps... I expect that it''s > something happening internally because you want to use timestamps but > Puppet wants to use checksums. If we could fix the file-copying, > we''d probably fix this.I don''t know if this is related, but now that there''s prelinking enabled on RHEL (and presumably other Linuxes as well), md5sums are no longer a reliable measure of file similarity. [eric@op240-002 ~]$ md5sum /bin/ls 219cefdb72e4b9ae6e543f579a3401b3 /bin/ls [eric@op240-002 ~]$ ls -l /bin/ls -rwxr-xr-x 1 root root 87608 Oct 7 2005 /bin/ls [eric@op240-001 ~]$ md5sum /bin/ls e2b03dfe637c506ed578ee5ce27ebe69 /bin/ls [eric@op240-001 ~]$ ls -l /bin/ls -rwxr-xr-x 1 root root 87608 Oct 7 2005 /bin/ls Here''s why: http://linux.derkeiler.com/Mailing-Lists/RedHat/2006-03/msg00151.html Again, not sure if it''s applicable but we got bit by this last week, and it seems to not be widely known. -- - Eric Sorenson - N37 17.255 W121 55.738 - http://ahpook.vox.com/ - - Personal colo with a professional touch - http://www.explosive.net -
Luke Kanies
2006-Sep-13 20:54 UTC
Re: strange mtime/md5sum behaviour and constantly changing files with links
On Sep 13, 2006, at 1:21 PM, Eric Sorenson wrote:> > I don''t know if this is related, but now that there''s prelinking > enabled on RHEL (and presumably other Linuxes as well), md5sums are no > longer a reliable measure of file similarity.[SNIP]> Again, not sure if it''s applicable but we got bit by this last week, > and it seems to not be widely known.I would think it''s unlikely people would be distributing binaries this way, but that certainly doesn''t mean someone won''t run into it. Is there any way around the problem? I assume not, but it''s good to know about it. -- Luke Kanies http://madstop.com | http://reductivelabs.com | 615-594-8199
Eric Sorenson
2006-Sep-13 21:37 UTC
Re: strange mtime/md5sum behaviour and constantly changing files with links
On Wed, 13 Sep 2006, Luke Kanies wrote:> I would think it''s unlikely people would be distributing binaries > this way, but that certainly doesn''t mean someone won''t run into it.Yup, my first cfengine site propagated our /usr/local/bin/ tree via cfengine 1.5. Not a wonderful solution but it worked well enough.> Is there any way around the problem? I assume not, but it''s good to > know about it.Check out Section 13 of Jakub Jelínek''s prelink paper: http://people.redhat.com/jakub/prelink.pdf I think you''d have to detect if prelink was enabled (/etc/prelink.conf exists) and if so, use the output of ''prelink --verify'' as the basis for your comparison instead of the in-place binary. -y --verify Verifies a prelinked binary or library. This option can be used only on a single binary or library. It first applies an --undo operation on the file, then prelinks just that file again and compares this with the original file. If both are identical, it prints the file after --undo operation on standard output and exit with zero status. Otherwise it exits with error status. Thus if --verify operation returns zero exit status and its standard output is equal to the content of the binary or library before prelinking, you can be sure that nobody modi- fied the binaries or libraries after prelinking. Neat stuff actually, and though I haven''t done any scientifically valid before-and-afters, it does seem to speed things up. -- - Eric Sorenson - N37 17.255 W121 55.738 - http://ahpook.vox.com/ - - Personal colo with a professional touch - http://www.explosive.net - _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Luke Kanies
2006-Sep-14 06:21 UTC
Re: strange mtime/md5sum behaviour and constantly changing files with links
On Sep 13, 2006, at 4:37 PM, Eric Sorenson wrote:> > Yup, my first cfengine site propagated our /usr/local/bin/ tree > via cfengine 1.5. Not a wonderful solution but it worked well enough.Yeah, I''ve done the same, but I think it clearly qualifies as abuse. :)> Check out Section 13 of Jakub Jelínek''s prelink paper: > http://people.redhat.com/jakub/prelink.pdf > > I think you''d have to detect if prelink was enabled (/etc/prelink.conf > exists) and if so, use the output of ''prelink --verify'' as the basis > for your comparison instead of the in-place binary. > > -y --verify > > Verifies a prelinked binary or library. This option can be > used only on a single binary or library. It first applies > an --undo operation > on the file, then prelinks just that file again and > compares this with the > original file. If both are identical, it prints the file > after --undo operation > on standard output and exit with zero status. Otherwise it > exits with error > status. Thus if --verify operation returns zero exit > status and its standard > output is equal to the content of the binary or library > before prelinking, you > can be sure that nobody modi- fied the binaries or > libraries after > prelinking. > > Neat stuff actually, and though I haven''t done any scientifically > valid before-and-afters, it does seem to speed things up.Until someone complains about this, I''m going to avoid it like the plague. That sounds like a pretty expensive operation to do every time I need a simple md5 sum. -- Luke Kanies http://madstop.com | http://reductivelabs.com | 615-594-8199
Kostas Georgiou
2006-Sep-14 10:36 UTC
Re: strange mtime/md5sum behaviour and constantly changing files with links
On Thu, Sep 14, 2006 at 01:21:25AM -0500, Luke Kanies wrote:> Until someone complains about this, I''m going to avoid it like the > plague. That sounds like a pretty expensive operation to do every > time I need a simple md5 sum.It is, if someone really wants this at some point in the future I suggest to have something like "md5prelink" as a checksum argument with warnings in the docs that maybe they should rethink what they are doing. I suspect that nobody will care about it though. Note that for example under Fedora only binaries/libraries in the following directories are going to be prelinked. $ cat /etc/prelink.conf ... -l /bin -l /usr/bin -l /sbin -l /usr/sbin -l /usr/X11R6/bin -l /usr/kerberos/bin -l /usr/games -l /usr/libexec -l /var/ftp/bin -l /lib{,64} -l /usr/lib{,64} -l /usr/X11R6/lib{,64} -l /usr/kerberos/lib{,64} -l /usr/X11R6/LessTif -l /var/ftp/lib{,64} so you can still use /usr/local/bin without problems. Kostas