zfs discuss - Mar 2010 - ZFS send and receive corruption across a WAN link?

Can a ZFS send stream become corrupt when piped between two hosts across a WAN
link using ''ssh''?

For example a host in Australia sends a stream to a host in the UK as follows:

# zfs send tank/foo at now | ssh host.uk receive tank/bar
-- 
This message posted from opensolaris.org

On Fri, Mar 19, 2010 at 12:38 PM, Rob <slewbrew at yahoo.com>
wrote:
> Can a ZFS send stream become corrupt when piped between two hosts across a
WAN link using ''ssh''?
unless the end computers are bad (memory problems, etc.), then the
answer should be no. ssh has its own error detection method, and the
zfs send stream itself is checksummed.

-- 
Fajar

On Fri, March 19, 2010 00:38, Rob wrote:
> Can a ZFS send stream become corrupt when piped between two hosts across a
> WAN link using ''ssh''?
>
> For example a host in Australia sends a stream to a host in the UK as
> follows:
>
> # zfs send tank/foo at now | ssh host.uk receive tank/bar
In general, errors would be detected by TCP (or by lower-level hardware
media error-checking), and the packet retransmitted.  I''m not sure what
error-checking ssh does on top of that (if any).

However, these legacy mechanisms aren''t guaranteed to give  you the
less-than-one-wrong-bit-in-10^15 level of accuracy people tend to want for
enterprise backups today (or am I off a couple of orders of magnitude
there?).  They were defined when data rates were much slower and data
volumes much lower.

In addition, memory errors on the receiving host (after the TCP stack
turns the data over to the application), if undetected, could leave you
with corrupted data; not sure what the probability is there.

Every scheme has SOME weak spots.  The well-designed ones at least tell 
you the bit error rate.

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On Fri, 19 Mar 2010, David Dyer-Bennet wrote:
>
> However, these legacy mechanisms aren''t guaranteed to give  you
the
> less-than-one-wrong-bit-in-10^15 level of accuracy people tend to want for
> enterprise backups today (or am I off a couple of orders of magnitude
> there?).  They were defined when data rates were much slower and data
> volumes much lower.
Are you sure?  Have you done any research on this?  You are saying 
that NSA+-grade crypto on the stream is insufficient to detect a 
modification to the data?

It seems that the main failure mode would be disconnect by ssh.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Fri, March 19, 2010 09:49, Bob Friesenhahn wrote:
> On Fri, 19 Mar 2010, David Dyer-Bennet wrote:
>>
>> However, these legacy mechanisms aren''t guaranteed to give 
you the
>> less-than-one-wrong-bit-in-10^15 level of accuracy people tend to want
>> for
>> enterprise backups today (or am I off a couple of orders of magnitude
>> there?).  They were defined when data rates were much slower and data
>> volumes much lower.
>
> Are you sure?  Have you done any research on this?  You are saying
> that NSA+-grade crypto on the stream is insufficient to detect a
> modification to the data?
I was referring to the tcp and hardware-level checksums.  I specifically
said I didn''t know if SSH did anything on top of that (other people
have
since said that it does, and it might well be plenty good enough; also
that ZFS itself has checksums in the send stream).

I don''t think of stream crypto as inherently including validity
checking,
though in practice I suppose it would always be a good idea.
> It seems that the main failure mode would be disconnect by ssh.
Sure, can''t guarantee against aborted connections at whatever level
(actual interruption of IP connectivity).  But those are generally
detected and reported as an error; one shouldn''t be left with the
impression the transfer succeeded.

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On Fri, 19 Mar 2010, David Dyer-Bennet wrote:
>
> I don''t think of stream crypto as inherently including validity
checking,
> though in practice I suppose it would always be a good idea.
This is obviously a vital and necessary function of ssh in order to 
defend against "man in the middle" attacks.  The main requirement is 
to make sure that the transferred data can not be deciphered or 
modified by something other than the two end-points.  I don''t know if 
ssh includes retry logic to request that modified data be 
retransmitted.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

They way we do this here is:


zfs snapshot volname at snapnow
[i]#code to break on error and email not shown.[/i]
zfs send -i volname at snapbefore volname at snapnow | pigz -p4 -1 > file
[i]#code to break on error and email not shown.[/i]
scp /dir/file user at remote:/dir/file
[i]#code to break on error and email not shown.[/i]
shh user at remote "gzip -t /dir/file"
[i]#code to break on error and email not shown.[/i]
shh user at remote "gunzip < /dir/file | zfs receive volname

It works for me and it sends a minimum amount of data across the wire which is
tested to minimize the chance of inflight issues. Excpet on Sundays when we do a
full send.
-- 
This message posted from opensolaris.org

On 03/20/10 09:28 AM, Richard Jahnel wrote:
> They way we do this here is:
>
>
> zfs snapshot volname at snapnow
> [i]#code to break on error and email not shown.[/i]
> zfs send -i volname at snapbefore volname at snapnow | pigz -p4 -1> 
file
> [i]#code to break on error and email not shown.[/i]
> scp /dir/file user at remote:/dir/file
> [i]#code to break on error and email not shown.[/i]
> shh user at remote "gzip -t /dir/file"
> [i]#code to break on error and email not shown.[/i]
> shh user at remote "gunzip<  /dir/file | zfs receive volname
>
> It works for me and it sends a minimum amount of data across the wire which
is tested to minimize the chance of inflight issues. Excpet on Sundays when we
do a full send.
>    
Don''t you trust the stream checksum?

-- 
Ian.

no, but I''m slightly paranoid that way. ;)
-- 
This message posted from opensolaris.org

On Mar 19, 2010, at 1:28 PM, Richard Jahnel wrote:
> They way we do this here is:
> 
> zfs snapshot volname at snapnow
> [i]#code to break on error and email not shown.[/i]
> zfs send -i volname at snapbefore volname at snapnow | pigz -p4 -1 >
file
> [i]#code to break on error and email not shown.[/i]
> scp /dir/file user at remote:/dir/file
> [i]#code to break on error and email not shown.[/i]
> shh user at remote "gzip -t /dir/file"
> [i]#code to break on error and email not shown.[/i]
> shh user at remote "gunzip < /dir/file | zfs receive volname
> 
> It works for me and it sends a minimum amount of data across the wire which
is tested to minimize the chance of inflight issues. Excpet on Sundays when we
do a full send.
NB. deduped streams should further reduce the snapshot size.
 -- richard

ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com

On Mon, Mar 22, 2010 at 10:26 AM, Richard Elling
<richard.elling at gmail.com>wrote:
> NB. deduped streams should further reduce the snapshot size.
>
I haven''t seen a lot of discussion on the list regarding send dedup,
but I
understand it''ll use the DDT if you have dedup enabled on your dataset.
What''s the process and penalty for using it on a dataset that is not
already
deduped? Does it build a DDT for just the data in the send?

-B

-- 
Brandon High : bhigh at freaks.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100322/2c93d1ba/attachment.html>

On 03/22/10 05:04 PM, Brandon High wrote:
> On Mon, Mar 22, 2010 at 10:26 AM, Richard Elling 
> <richard.elling at gmail.com <mailto:richard.elling at
gmail.com>> wrote:
>
>     NB. deduped streams should further reduce the snapshot size.
>
>
> I haven''t seen a lot of discussion on the list regarding send
dedup,
> but I understand it''ll use the DDT if you have dedup enabled on
your
> dataset.
The send code (which is user-level) builds its own DDT no matter what, 
but it will use existing checksums if on-disk dedup is already in effect.
> What''s the process and penalty for using it on a dataset that is
not
> already deduped?
The penalty is the cost of doing the checksums.
> Does it build a DDT for just the data in the send?
Yes, currently limited to 20% of physical memory size.

Lori

>
> -B
>
> -- 
> Brandon High : bhigh at freaks.com <mailto:bhigh at freaks.com>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>    
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100322/01c90b71/attachment.html>

On Thu, Mar 18, 2010 at 10:38:00PM -0700, Rob wrote:
> Can a ZFS send stream become corrupt when piped between two hosts
> across a WAN link using ''ssh''?
No.  SSHv2 uses HMAC-MD5 and/or HMAC-SHA-1, depending on what gets
negotiated, for integrity protection.  The chances of random on the wire
corruption going undetected by link-layer CRCs, TCP''s CRC and
SSHv2''s
MACs is infinitessimally small.  I suspect the chances of local bit
flips due to cosmic rays and what not are higher.

A bigger problem is that SSHv2 connections do not survive corruption on
the wire.  That is, if corruption is detected then the connection gets
aborted.  If you were zfs send''ing 1TB across a long, narrow link and
corruption hit the wire while sending the last block you''d have to
re-send the whole thing (but even then such corruption would still have
to get past link-layer and TCP checksums -- I''ve seen it happen, so it
is possible, but it is also unlikely).

Nico
--

Not quite brave enough to put dedup into prodiction here.

Concerned about the issues some folks have had when releasing large numbers of
blocks in one go.
-- 
This message posted from opensolaris.org

On Mar 23, 2010, at 9:05 AM, Richard Jahnel wrote:
> Not quite brave enough to put dedup into prodiction here.
> 
> Concerned about the issues some folks have had when releasing large numbers
of blocks in one go.
The send/receive dedup is independent of the pool dedup.  You do not
have to dedup the pool to benefit from send/receive dedup.
 -- richard

ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com