Pavel Gorshkov
2006-Jan-09 16:09 UTC
SHA1_Update() produces wrong results for large buffers
An important detail to begin with is that our libmd.a, as opposed
to libmd.so, provides a faster, assembly-optimized version of the
`SHA1_Update' function (it is not compatible with PIC and is
therefore not used in libmd.so).
The problem is that the asm-optimized version fails on large input
buffers. Attached is a test program, which mmaps a file and then
just feeds its contents to SHA1_Update():
gcc sha1test.c -o sha1test.md-shared -lmd
gcc sha1test.c -o sha1test.md-static -lmd -static
the input files:
dd if=/dev/zero bs=32M count=48 of=test-1.5G
dd if=/dev/zero bs=32M count=32 of=test-1G
dd if=/dev/zero bs=32M count=16 of=test-0.5G
***
# exits immediately, displaying a WRONG hash value
./sha1test.md-static test-1.5G
747cd7172ce7737d1735cf936c0d69ce0f733fcd
# OK
./sha1test.md-shared test-1.5G
a957f01b1a92366c7b72296cb24eb84f42ed06e4
***
./sha1test.md-static test-1G
0d6ee6083bf8b6368cb80d323e82164e5540e296
# ^^^ WRONG
# OK
./sha1test.md-shared test-1G
2a492f15396a6768bcbca016993f4b4c8b0b5307
However, both programs work fine with files less than ~920M:
./sha1test.md-static test-0.5G
5b088492c9f4778f409b7ae61477dec124c99033
./sha1test.md-shared test-0.5G
5b088492c9f4778f409b7ae61477dec124c99033
Everything was tested on a RELENG_6/i386 box (CFLAGS = -O2).
Is this a bug in libmd, or am I missing something?
Thanks in advance,
--
Pavel Gorshkov
-------------- next part --------------
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <sha.h>
int main(int argc, char **argv) {
int fd;
struct stat st;
SHA_CTX ctx;
unsigned char *buf;
char hexdigest[41];
if (argc < 2 || stat(argv[1], &st) < 0 || (fd=open(argv[1],
O_RDONLY)) < 0)
exit(1);
if ((buf = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0)) ==
MAP_FAILED)
exit(1);
SHA1_Init(&ctx);
SHA1_Update(&ctx, buf, st.st_size);
SHA1_End(&ctx, hexdigest);
puts(hexdigest);
if (st.st_size)
munmap(buf, st.st_size);
close(fd);
return 0;
}
Pavel Gorshkov wrote:> Everything was tested on a RELENG_6/i386 box (CFLAGS = -O2).How exactly do you know, that the static version is wrong, and not vice-vera (e.g. caused by the -O2 optimization)? -- Best regards / Viele Gr??e, barner@FreeBSD.org Simon Barner barner@gmx.de -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060110/b70ee556/attachment.bin
Pavel Gorshkov wrote:> ./sha1test.md-static test-1G > 0d6ee6083bf8b6368cb80d323e82164e5540e296 > # ^^^ WRONG > # OK > ./sha1test.md-shared test-1G > 2a492f15396a6768bcbca016993f4b4c8b0b5307Values confirmed for 6.0-STABLE with CFLAGS="-O -pipe". -- Best regards / Viele Gr??e, barner@FreeBSD.org Simon Barner barner@gmx.de -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060110/528c090a/attachment.bin
On Tue, 2006-Jan-10 02:59:53 +0300, Pavel Gorshkov wrote:>The problem is that the asm-optimized version fails on large input >buffers. Attached is a test program, which mmaps a file and then >just feeds its contents to SHA1_Update():"openssl sha1" agrees with the shared version on -current.> # exits immediately, displaying a WRONG hash value > ./sha1test.md-static test-1.5G > 747cd7172ce7737d1735cf936c0d69ce0f733fcdI get this on 7-current as well. Copying the relevant bits from libmd and compiling it myself, I get the same behaviour. The fact that this exits virtually instantly strongly suggests that it is broken (rather than the shared version). My initial guess is that an operation on the length is overflowing 32 bits. Unfortunately, the asm is rather opaque - it was auto-generated by a perl script that doesn't seem to included in the repository. (There is a sha1-586.pl in openssl but it generates different code). As far as I can determine, the asm code (sha1_block_x86) is designed to process an integral number of SHA1 blocks of input, leaving the remainder to be processed in the C code. Using the debugger, the asm code is not looping when passed 1610612736 (1.5G) - which explains the rapid exit and incorrect result. -- Peter Jeremy