Christopher Hall
2017-Jun-08 09:15 UTC
dd is returning different content using same inputs
Hello Dewayne, On Thu, 8 Jun 2017 18:10:11 +1000, Dewayne Geraghty <dewaynegeraghty at gmail.com> wrote:> I'm attempting to skip the first three bytes of a file and then pull > the header out of message. Please note the file size varies between > 105 and 54 bytes, even though the input file and everything else > remains constant, on > > FreeBSD hathor 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #0 r319562M: > Mon Jun 5 03:13:08 AEST 2017 > root at hathor:/110007/D/K8/hqdev-amd64-smp-vga amd64 1100514 1100514 > > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > count=1 of=encheader ; ls -lrth encheader > 0+1 records in > 0+1 records out > 105 bytes transferred in 0.000026 secs (4071661 bytes/sec) > -rw-r--r-- 1 root wheel 105B Jun 8 16:35 encheader > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > count=1 of=encheader ; ls -lrth encheader > 0+1 records in > 0+1 records out > 67 bytes transferred in 0.000026 secs (2554912 bytes/sec) > -rw-r--r-- 1 root wheel 67B Jun 8 16:35 encheader > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > count=1 of=encheader ; ls -lrth encheader > 0+1 records in > 0+1 records out > 68 bytes transferred in 0.000024 secs (2867263 bytes/sec) > -rw-r--r-- 1 root wheel 68B Jun 8 16:35 encheader > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > count=1 of=encheader ; ls -lrth encheader > 0+1 records in > 0+1 records out > 54 bytes transferred in 0.000025 secs (2167717 bytes/sec) > -rw-r--r-- 1 root wheel 54B Jun 8 16:35 encheader > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > count=1 of=encheader ; ls -lrth encheader > 0+1 records in > 0+1 records out > 91 bytes transferred in 0.000024 secs (3751494 bytes/sec) > -rw-r--r-- 1 root wheel 91B Jun 8 16:35 encheader > > I used this around May 8th (from a OS and ports rebuild on 4th May) > and was performing some final testing on an 11.1 Prerelease built on > Jun 5. >It works similarly on TrueOS (FreeBSD 12) I suspect it is because you block sizes are different on the input and output dd commands. I would think that the output block size (obs) on 1st dd needs to be specified the same input block size on receiving dd to make it work correctly.> This works correctly so perhaps I'm not using dd correctly? > dd if=/tmp/tLbdoJ/encmessage skip=1 bs=3 | dd bs=1 count=384 > of=encheader ; hd encheader|head -n1; ls -l encheader > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe at freebsd.org"-- Best Regards. Christopher Hall.
Arrigo Marchiori
2017-Jun-08 09:59 UTC
dd is returning different content using same inputs
Hello, On Thu, Jun 08, 2017 at 05:15:15PM +0800, Christopher Hall wrote:> Hello Dewayne, > > On Thu, 8 Jun 2017 18:10:11 +1000, Dewayne Geraghty > <dewaynegeraghty at gmail.com> wrote: > > > I'm attempting to skip the first three bytes of a file and then pull > > the header out of message. Please note the file size varies between > > 105 and 54 bytes, even though the input file and everything else > > remains constant, on > > > > FreeBSD hathor 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #0 r319562M: > > Mon Jun 5 03:13:08 AEST 2017 > > root at hathor:/110007/D/K8/hqdev-amd64-smp-vga amd64 1100514 1100514 > > > > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > > count=1 of=encheader ; ls -lrth encheader > > 0+1 records in > > 0+1 records out > > 105 bytes transferred in 0.000026 secs (4071661 bytes/sec) > > -rw-r--r-- 1 root wheel 105B Jun 8 16:35 encheader > > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > > count=1 of=encheader ; ls -lrth encheader > > 0+1 records in > > 0+1 records out > > 67 bytes transferred in 0.000026 secs (2554912 bytes/sec) > > -rw-r--r-- 1 root wheel 67B Jun 8 16:35 encheader > > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > > count=1 of=encheader ; ls -lrth encheader > > 0+1 records in > > 0+1 records out > > 68 bytes transferred in 0.000024 secs (2867263 bytes/sec) > > -rw-r--r-- 1 root wheel 68B Jun 8 16:35 encheader > > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > > count=1 of=encheader ; ls -lrth encheader > > 0+1 records in > > 0+1 records out > > 54 bytes transferred in 0.000025 secs (2167717 bytes/sec) > > -rw-r--r-- 1 root wheel 54B Jun 8 16:35 encheader > > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384 > > count=1 of=encheader ; ls -lrth encheader > > 0+1 records in > > 0+1 records out > > 91 bytes transferred in 0.000024 secs (3751494 bytes/sec) > > -rw-r--r-- 1 root wheel 91B Jun 8 16:35 encheader > > > > I used this around May 8th (from a OS and ports rebuild on 4th May) > > and was performing some final testing on an 11.1 Prerelease built on > > Jun 5. > > > It works similarly on TrueOS (FreeBSD 12) I suspect it is because you > block sizes are different on the input and output dd commands. > I would think that the output block size (obs) on 1st dd needs to be > specified the same input block size on receiving dd to make it work > correctly.I also agree; the ``funny'' behavior is probably given by the pipe and the different block sizes. The manual page of dd(1) says about the bs parameter: [...]> bs=n Set both input and output block size to n bytes, superseding the > ibs and obs operands. If no conversion values other than > noerror, notrunc or sync are specified, then each input block is > copied to the output as a single block without any aggregation > of short blocks.The second dd seems to receive non-deterministic amounts of data from the first, and due to the bs parameter, it does not wait until 384 bytes are received, but it outputs its partial block immediately and then quits.> > This works correctly so perhaps I'm not using dd correctly? > > dd if=/tmp/tLbdoJ/encmessage skip=1 bs=3 | dd bs=1 count=384 > > of=encheader ; hd encheader|head -n1; ls -l encheaderIf I understood correctly, you could issue a single command: $ dd if=inputfile bs=1 skip=3 count=384 of=outputfile I hope this helps. -- rigo http://rigo.altervista.org