I forgot to add current/stable to the list TL;DR: there seems(!) to be something(!) unclean about an ssh path between an 8.3-STABLE(r247820) and 10.0-CURRENT(r247826) box such that a zfs send stream is corrupted in transit. below is the thread from -fs about it, with sshd configs from both sides. If I copy the stream it works, but piping through ssh does NOT. -------- Original Message -------- Subject: Re: zfs send/recv invalid data Date: 2013-03-06 04:46 From: Larry Rosenman <ler at lerctr.org> To: Steven Hartland <killing at multiplay.co.uk> Cc: Ronald Klop <ronald-freebsd8 at klop.yi.org>, <freebsd-fs at freebsd.org> On 2013-03-06 02:38, Steven Hartland wrote:> ----- Original Message ----- From: "Larry Rosenman" <ler at lerctr.org> >>>>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs >>>>>>>> recv (to 10.0) of a -R -I stream. >>>>>>>> What data do I need to gather to figure out what side and >>>>>>>> what's wrong? >>>>>>>> I've already started zpool scrubs on both sides. >>>>>>>> I can insert a tee to grab the stream on either/both sides if >>>>>>>> that would help. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Is the problem repeatable or is it just a network glitch? >>>>>>> Ronald. >>>>>> Repeatable....... >>>>> Here is the exact error message: >>>>> receiving incremental stream of vault/home/ctr at 2013-03-05-test3 >>>>> into zroot/backups/TBH/home/ctr at 2013-03-05-test3 >>>>> cannot receive incremental stream: invalid backup stream >>>>> this is the script I'm running: >>>>> #!/bin/sh >>>>> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >>>>> DATE2=`date -v "-1d" "+%Y-%m-%d"` >>>>> # snap the source >>>>> ssh root at tbh.lerctr.org zfs snapshot -r vault@${DATE} >>>>> # zfs copy the source to here. >>>>> ssh root at tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} >>>>> vault@${DATE} | \ >>>>> tee /tmp/backup.stream.send.${DATE} | \ >>>>> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} >>>>> | zfs recv -u -v -d zroot/backups/TBH\"" >>>>> # make sure we NEVER allow the backup stuff to automount. >>>>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >>>>> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >>>>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV >>>> Your send and receive sides differ, which indicates your ssh >>>> shell my not be clean. >>>> Looking at the receive side its got what looks like a mail >>>> message appended. >>>> I suspect if you manually copy the receive copy to the 10 machine >>>> and >>>> the receive it will work fine. >>> we're copying mail files........ >>> and it still fails.... >>> >> I've put more example send/recv files in that directory. >> we're copying home dirs, which include lots of mail. >> (this one is my wife's) >> Ideas? >> I *CAN* give access to both sides via ssh..... > The copy of the data stream on both sides should be identical > though and its not, which leads me to believe something is > corrupting the data on the way. Try the following:- > >> From source:- > zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream > scp test.stream home.lerctr.org:~/ >> From target: > zfs recv -u -v -d zroot/backups/TBH < test.stream > If this works then there is something unclean about your ssh > shell. > Regards > Steve > send side:# zfs send -R -D -I vault at 2013-03-05 vault at 2013-03-06 >/tmp/send.stream # openssl md5 /tmp/send.stream MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 # scp /tmp/send.stream home:/tmp/send.stream send.stream 100% 1180MB 2.5MB/s 07:44 # uname -a FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: Mon Mar 4 18:08:11 CST 2013 root at thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 # Receive side: # uname -a FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon Mar 4 19:59:08 CST 2013 root at borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 # openssl md5 /tmp/send.stream MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 # zfs recv -F -u -v -d zroot/backups/TBH < /tmp/send.stream <success> # So, you are correct that something(tm) is unclean about the ssh path. adding -current and -stable for diagnosing ssh issue. sshd config on the 8.3-STABLE box: # cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $ # $FreeBSD: stable/8/crypto/openssh/sshd_config 247521 2013-03-01 02:06:04Z des $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Note that some of FreeBSD's defaults differ from OpenBSD's, and # FreeBSD has a few additional options. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # Disable legacy (protocol version 1) support in the server for new # installations. In future the default will change to require explicit # activation of protocol 1 Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys #AuthorizedKeysFile .ssh/authorized_keys #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # Change to yes to enable built-in password authentication. #PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable PAM authentication #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'no' to disable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation sandbox #PermitUserEnvironment no #Compression delayed ClientAliveInterval 120 ClientAliveCountMax 200000 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none #VersionAddendum FreeBSD-20120901 # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server # Disable HPN tuning improvements. #HPNDisabled no # Buffer size for HPN to non-HPN connections. #HPNBufferSize 2048 # TCP receive socket buffer polling for HPN. Disable on non autotuning kernels. #TcpRcvBufPoll yes # Allow the use of the NONE cipher. #NoneEnabled no # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server # sshd config on the 10.0-CURRENT: # cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $ # $FreeBSD: head/crypto/openssh/sshd_config 240075 2012-09-03 16:51:41Z des $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Note that some of FreeBSD's defaults differ from OpenBSD's, and # FreeBSD has a few additional options. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # The default requires explicit activation of protocol 1 #Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # Change to yes to enable built-in password authentication. #PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable PAM authentication #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'no' to disable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation sandbox #PermitUserEnvironment no #Compression delayed ClientAliveInterval 120 ClientAliveCountMax 200000 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none #VersionAddendum FreeBSD-20120901 # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server # Disable HPN tuning improvements. #HPNDisabled no # Buffer size for HPN to non-HPN connections. #HPNBufferSize 2048 # TCP receive socket buffer polling for HPN. Disable on non autotuning kernels. #TcpRcvBufPoll yes # Allow the use of the NONE cipher. #NoneEnabled no # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server # Ideas from the ssh folks? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler at lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893