Svein Skogen
2010-Mar-10 13:54 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Are there any good options for encapsulating/decapsulating a zfs send stream inside FEC (Forward Error Correction)? This could prove very useful both for backup purposes, and for long-haul transmissions. If there are any good options for simply piping the data trough, feel free to point me in the right direction, because my google-searches didn''t give me any real clues. Maybe my google-fu isn''t up to scratch. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuXpIoACgkQSBMQn1jNM7YPPgCgk40w47g/L2djCvMVEYGiU4zt wOcAoP5CzX/9W7kAJIgLj8H+zZqhSfi+ =SmEf -----END PGP SIGNATURE-----
David Dyer-Bennet
2010-Mar-10 14:50 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
On Wed, March 10, 2010 07:54, Svein Skogen wrote:> Are there any good options for encapsulating/decapsulating a zfs send > stream inside FEC (Forward Error Correction)? This could prove very > useful both for backup purposes, and for long-haul transmissions.I don''t know of anything that would actually function as a filter. In the absence of something better turning up, I''d write the stream to disk, and then use PAR2 to create some redundant data for recover from errors, up to whatever level you choose. Obviously this approach doesn''t work if you don''t have the space; since I think in terms of disks rather than tapes for backups, that''s not my issue. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Wes Felter
2010-Mar-10 19:55 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
Svein Skogen wrote:> Are there any good options for encapsulating/decapsulating a zfs send > stream inside FEC (Forward Error Correction)? This could prove very > useful both for backup purposes, and for long-haul transmissions.http://www.s.netic.de/gfiala/dvbackup.html http://planete-bcast.inrialpes.fr/rubrique.php3?id_rubrique=5 I''m skeptical about the benefit, but there you are. Wes Felter
Daniel Carosone
2010-Mar-11 08:23 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
On Wed, Mar 10, 2010 at 02:54:18PM +0100, Svein Skogen wrote:> Are there any good options for encapsulating/decapsulating a zfs send > stream inside FEC (Forward Error Correction)? This could prove very > useful both for backup purposes, and for long-haul transmissions.I used par2 for this for some time, but it is slow / cpu intensive. There are windows implementations that are rather faster, but the whole codebase seems to be largely abandoned, especially the unix one. I switched to raidz pools in files, as I''ve described here previously. I was interested to look at zfec, which is used within Zooko''s allmydata tahoe filesystem, as an allegedly much faster implementation. I couldn''t get it to work, IIRC because some dependency modules were unreachable at the time. I ran into my lack of python snake-charming skills, and lack of time. You have reminded me to go back and look again, and either find that whatever issue was at fault last time was transient and now gone, or determine what it actually was and get it resolved. In case you want to: http://allmydata.org/trac/zfec> If there are any good options for simply piping the data trough, feel > free to point me in the right direction, because my google-searches > didn''t give me any real clues. Maybe my google-fu isn''t up to scratch.Most of the schemes/tools I''ve seen, including the ones above, don''t work in streaming mode. That''s not been a requirement in my usage (protection against bad dvd media in backup/archives) so far. I haven''t really cared since I have to stage backup copies for other reasons anyway, so there''s no benefit (to me) in streaming. The LDPC ones mentioned in the thread seem to be stream oriented, and I''d like to look at them some more. Note that for streaming over the network, you''d need to use something other than tcp/ssh as transport to gain any advantage. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100311/203b24bf/attachment.bin>
Daniel Carosone
2010-Mar-11 09:00 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
On Thu, Mar 11, 2010 at 07:23:43PM +1100, Daniel Carosone wrote:> You have reminded me to go back and look again, and either find that > whatever issue was at fault last time was transient and now gone, or > determine what it actually was and get it resolved. > > In case you want to: http://allmydata.org/trac/zfecWell, there has been a new release since then - but the same problem. It gets stuck trying to find a dependency "pyutil" from a zooko.com url. A little manual help to grab and install that pkg separately from elsewhere, and things have progressed. It seems to run now, and be well IO bound on my laptop at least - only about 12% cpu in top. So far so good, but I''ve barely started playing with it.. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100311/2c4abe34/attachment.bin>
Svein Skogen
2010-Mar-11 10:00 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
I can''t help but keep wondering if not some sort of FEC wrapper (optional of course) might solve both the "backup" and some of the long-distance-transfer (where retransmissions really isn''t wanted) issues. Reason I''m saying long-distance, is this is where latency-on-the-link starts rearing its ugly head, and where a pure udp stream is beneficial (since that doesn''t depend on so many ack packages in return). Albeit with some earlier discussions using words like "brokenfletcher" to describe things (in a relatively heated discussion I wasn''t part of, see the thread here: http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/032432.html), it seems the zfs code already contain rather good hashing-algorithms for error-detection. This is a good thing. Really it is. Especially if there is and option for adding something rather lightweight (when not correcting errors, that is) like CIRC (Cross-Interleaved Reed-Solomon Codes) wrapped around the send. Now, I''m not saying that we apply CIRC itself, since that has been destroyed for these purposes (some broken countries have software patents, and CIRC has been patented in that country, see US4413340), but the method itself (two layers of reed-solomon encoding) gives rather good options for not only error-detection but also correcting it. This would be beneficial for Sun^wOracle''s enterprise customers as well. But I''m thinking that for snapshotting things like "this is what I need to get enough of the system up and running to run a full restore!" this should probably be applied to the much wanted "zpool send and receive" ideas. ;) (the reasoning being that "enough code to grab the rpool from backup" can be sent over PXE to the box needing restore) //Svein -- This message posted from opensolaris.org
Richard Elling
2010-Mar-11 23:57 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
On Mar 11, 2010, at 2:00 AM, Svein Skogen wrote:> I can''t help but keep wondering if not some sort of FEC wrapper (optional of course) might solve both the "backup" and some of the long-distance-transfer (where retransmissions really isn''t wanted) issues.I don''t think retransmissions of b0rken packets is a problem anymore, most people use ssh which provides good error detection at a fine grain. It is rare that one would need to resend an entire ZFS dump stream when using ssh (or TLS or ...) Archival tape systems are already designed to wrap the payload with ECC. For example, the T10000b is rated at 30 year shelf life and 10^-19 BER -- 4 orders of magnitude better than enterprise class HDDs and 5 orders of magnitude better than consumer class HDDs. Also, many enterprise backup solutions have long-term tape management as part of their core feature set. A well designed, enterprise class backup system should be easily capable of storing ZFS dump images reliably, safely, and securely. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)
Daniel Carosone
2010-Mar-12 01:16 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
On Thu, Mar 11, 2010 at 02:00:41AM -0800, Svein Skogen wrote:> I can''t help but keep wondering if not some sort of FEC wrapper > (optional of course) might solve both the "backup" and some of the > long-distance-transfer (where retransmissions really isn''t wanted) > issues.Retransmissions aren''t really the problem, at least not until you''re dealing with a very lossy path. Old-style TCP stop-and-go-back-N retransmit is the problem, and it gets worse when combined with large windows (needed on long fast links) and nagle slow-start. Modern TCP, particularly SACK, deals well with all that. Default configs that don''t enable large windows and socket buffers sometimes still get in the way, but that''s a once-off setup annoyance at worst. If you wan''t to use FEC in the network transport, you need to avoid TCP (or use many small TCP parallel streams, which has its own issues). Then you''ll have to re-solve many of the other requirements TCP solves in some other way (primarily, flow control so your sender doesn''t outpace the link or receiver and wind up with more packets dropped than your FEC can recover). Stream FEC is of most interest in broad/multicast, where it''s better for clients to reconstruct data than storm the sender with retransmit requests. There could be an interesting application in here for multi-site replication using multicast, avoiding duplication on the transmit path out of the source data centre. There would still need to be a wrapper around this to catch up for periods where a receiver was off-air, but it could be a useful traffic optimisation for the everyday distribution. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100312/7e6df1a4/attachment.bin>
Edward Ned Harvey
2010-Mar-12 13:21 UTC
[zfs-discuss] zfs send and receive ... any ideas for FEC?
> I don''t think retransmissions of b0rken packets is a problem anymore, > most people > use ssh which provides good error detection at a fine grain. It is > rare that one would > need to resend an entire ZFS dump stream when using ssh (or TLS or ...) > > Archival tape systems are already designed to wrap the payload with > ECC. For > example, the T10000b is rated at 30 year shelf life and 10^-19 BER -- 4 > orders of > magnitude better than enterprise class HDDs and 5 orders of magnitude > better > than consumer class HDDs. > > Also, many enterprise backup solutions have long-term tape management > as part of > their core feature set. > > A well designed, enterprise class backup system should be easily > capable of > storing ZFS dump images reliably, safely, and securely.Actually, this is a really good point. We have some FEC experts at work, and I asked them this question. They couldn''t name any software package that does this, because it''s always done in hardware. After conversation, I was shocked to learn how ridiculously everywhere FEC is in every device you use. It''s required in your hard drive, your flash drive, your ram, your motherboard, your Ethernet. Everywhere. And it''s absolutely a design decision how strong you want to build it. So for a commodity hard drive, you can expect lower grade, while an enterprise tape drive and tape media, you can rest assured have much stronger. Whenever a data stream gets corrupted, you can pretty well count on the idea that another layer of FEC wasn''t going to save you; if you get any errors at all in your data, you''re safe assuming the data is *dramatically* altered. There was some good info about it, I read in this article coincidentally yesterday. Read the section that says "Hard disks are unreliable" http://arstechnica.com/microsoft/news/2010/03/why-new-hard-disks-might-not-b e-much-fun-for-xp-users.ars