Brad Diggs
2008-Feb-29 09:20 UTC
[zfs-discuss] RFE: Start with desired end state in mind...
I love the send and receive feature of zfs. However, the one feature that it lacks is that I can''t specify on the receive end how I want the destination zfs filesystem to be be created before receiving the data being sent. For example, lets say that I would like to do a compression study to determine which level of compression of the gzip algorithm would save the most space for my data. One of the easiest ways to do that locally or remotely would be to use send/receive like so. zfs snapshot zpool/data at now gz=1 while [ ${gz} -le 9 ] do zfs send zpool/data at now | \ zfs receive -o compression=gzip-${gz} zpool/gz${gz}data zfs list zpool/gz${gz}data done zfs destroy zpool/data at now Another example. Lets assume that that the zfs encryption feature was available today. Further, lets assume that I have a filesystem that has compression and encryption enabled. I want to duplicate that exact zfs filesystem on another system through send/receive. Today the receive feature does not give me the ability to specify the desired end state configuration of the destination zfs filesystem before receiving the data. I think that would be a great feature. Just some food for thought. Thanks in advance, Brad
Constantin Gonzalez
2008-Feb-29 09:46 UTC
[zfs-discuss] RFE: Start with desired end state in mind...
Hi Brad, this is indeed a good idea. But I assume that it will be difficult to do, due to the low-level nature of zfs send/receive. In your compression example, you''re asking for zfs send/receive to decompress the blocks on the fly. But send/receive operates on a lower level: It doesn''t care much what is actually inside the blocks, it just copies the block structure "as is". So, unless zfs send/receive starts looking inside the blocks it copies, it is probably better to use good old tar or cpio. I would also assume that zfs send/receive doesn''t know (and doesn''t have to) anything about zfs dataset properties. It just starts at the snapshot level (which works independently of whether it''s used for a filesystem, a ZVOL, a whatever) and copies the subtree. But for the sake of implementing the RFE, one could extend the ZFS send/receive framework with a module that permits manipulation of the data on the fly, specifically in order to allow for things like recompression, en/decryption, change of attributes at the dataset level, etc. Best regards, Constantin Brad Diggs wrote:> I love the send and receive feature of zfs. However, the one feature > that it lacks is that I can''t specify on the receive end how I want > the destination zfs filesystem to be be created before receiving the > data being sent. > > For example, lets say that I would like to do a compression study to > determine which level of compression of the gzip algorithm would save > the most space for my data. One of the easiest ways to do that > locally or remotely would be to use send/receive like so. > > zfs snapshot zpool/data at now > gz=1 > while [ ${gz} -le 9 ] > do > zfs send zpool/data at now | \ > zfs receive -o compression=gzip-${gz} zpool/gz${gz}data > zfs list zpool/gz${gz}data > done > zfs destroy zpool/data at now > > Another example. Lets assume that that the zfs encryption feature was > available today. Further, lets assume that I have a filesystem that > has compression and encryption enabled. I want to duplicate that exact > zfs filesystem on another system through send/receive. Today the > receive feature does not give me the ability to specify the desired end > state configuration of the destination zfs filesystem before receiving > the data. I think that would be a great feature. > > Just some food for thought. > > Thanks in advance, > Brad > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Constantin Gonzalez Sun Microsystems GmbH, Germany Platform Technology Group, Global Systems Engineering http://www.sun.de/ Tel.: +49 89/4 60 08-25 91 www.google.com/search?q=constantin+gonzalez Sitz d. Ges.: Sun Microsystems GmbH, Sonnenallee 1, 85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Dr. Roland Boemer Vorsitzender des Aufsichtsrates: Martin Haering
Darren J Moffat
2008-Feb-29 10:47 UTC
[zfs-discuss] RFE: Start with desired end state in mind...
Constantin Gonzalez wrote:> Hi Brad, > > this is indeed a good idea. > > But I assume that it will be difficult to do, due to the low-level nature > of zfs send/receive. > > In your compression example, you''re asking for zfs send/receive to > decompress the blocks on the fly. But send/receive operates on a lower > level: It doesn''t care much what is actually inside the blocks, it > just copies the block structure "as is".zfs send output is uncompressed, it sends the data as seen in the ARC not as seen on disk. Try this: $ zfs create -o compression=gzip-1 tank/gz1/ $ cp hamlet.txt /tank/gz1/ $ zfs snapshot tank/gz1 at s $ zfs send -R tank/gz1 at s > /tmp/foo $ strings /tmp/foo ..... Mar. ''Tis gone and will not answer. Ber. How now, Horatio? You tremble and look pale. Is not this something more than fantasy? What think you on''t? ..... The same is true for encrypted datasets the zfs send stream will be clear text not ciphertext. To protect the data in transit between pools on remote hosts do something like this: $ zfs snapshot -R tank@`date +%F` $ zfs send -R tank@`date +%F` | ssh remotehost ''zfs recv -d rtank` Or it you are storing the stream on tape: $ zfs send -R tank@`date +%F` | encrypt -a aes -k /rmdisk/mykey | dd of=/dev/tape You get the idea.> I would also assume that zfs send/receive doesn''t know (and doesn''t > have to) anything about zfs dataset properties. It just starts at > the snapshot level (which works independently of whether it''s used for > a filesystem, a ZVOL, a whatever) and copies the subtree.See the man page for zfs(1) where the -R options for send is discussed.> But for the sake of implementing the RFE, one could extend the ZFS > send/receive framework with a module that permits manipulation of the > data on the fly, specifically in order to allow for things like > recompression, en/decryption, change of attributes at the dataset level, > etc.No need this already works this way. -- Darren J Moffat
Constantin Gonzalez
2008-Feb-29 11:25 UTC
[zfs-discuss] RFE: Start with desired end state in mind...
Hi Darren, thank you for the clarification, I didn''t know that.> See the man page for zfs(1) where the -R options for send is discussed.oh, this is new. Thank you for bringing us -R. Back to Brad''s RFS, what would one need to do to send a stream from a compressed filesystem to one with a different compression setting, if the source file system has the compression attribute set to a specific algorithm (i.e. not inherited)? Will leaving out -R just create a new, but plain unencrypted fs on the receivig side? What if one wants to replicated a whole package of filesystems via -R, but change properties on the receiving side before it happens? Best regards, Constantin> >> But for the sake of implementing the RFE, one could extend the ZFS >> send/receive framework with a module that permits manipulation of the >> data on the fly, specifically in order to allow for things like >> recompression, en/decryption, change of attributes at the dataset level, >> etc. > > No need this already works this way. >-- Constantin Gonzalez Sun Microsystems GmbH, Germany Platform Technology Group, Global Systems Engineering http://www.sun.de/ Tel.: +49 89/4 60 08-25 91 www.google.com/search?q=constantin+gonzalez Sitz d. Ges.: Sun Microsystems GmbH, Sonnenallee 1, 85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Dr. Roland Boemer Vorsitzender des Aufsichtsrates: Martin Haering
Darren J Moffat
2008-Feb-29 11:31 UTC
[zfs-discuss] RFE: Start with desired end state in mind...
Constantin Gonzalez wrote:> Hi Darren, > > thank you for the clarification, I didn''t know that. > >> See the man page for zfs(1) where the -R options for send is discussed.> Back to Brad''s RFS, what would one need to do to send a stream from a > compressed filesystem to one with a different compression setting, if > the source file system has the compression attribute set to a specific > algorithm (i.e. not inherited)?$ zfs create -o compression=gzip-1 tank/gz1 # put in your data $ zfs snapshot tank/gz1 at s $ zfs create -o compression=gzip-9 tank/gz9 $ zfs send tank/gz1 at s | zfs recv -d tank/gz9> Will leaving out -R just create a new, but plain unencrypted fs on the > receivig side?Depends on inheritance.> What if one wants to replicated a whole package of filesystems via > -R, but change properties on the receiving side before it happens?If they are all getting the same properties use inheritance if they aren''t then you (by the very nature of what you want to do) need to precreate them with the appropriate options. -- Darren J Moffat
Constantin Gonzalez
2008-Feb-29 11:36 UTC
[zfs-discuss] RFE: Start with desired end state in mind...
Hi, great, thank you. So ZFS isn''t picky about finding the target fs already created and attributed when replicating data into it. This is very cool! Best regards, Constantin Darren J Moffat wrote:> Constantin Gonzalez wrote: >> Hi Darren, >> >> thank you for the clarification, I didn''t know that. >> >>> See the man page for zfs(1) where the -R options for send is discussed. > > >> Back to Brad''s RFS, what would one need to do to send a stream from a >> compressed filesystem to one with a different compression setting, if >> the source file system has the compression attribute set to a specific >> algorithm (i.e. not inherited)? > > $ zfs create -o compression=gzip-1 tank/gz1 > # put in your data > $ zfs snapshot tank/gz1 at s > $ zfs create -o compression=gzip-9 tank/gz9 > $ zfs send tank/gz1 at s | zfs recv -d tank/gz9 > >> Will leaving out -R just create a new, but plain unencrypted fs on the >> receivig side? > > Depends on inheritance. > >> What if one wants to replicated a whole package of filesystems via >> -R, but change properties on the receiving side before it happens? > > If they are all getting the same properties use inheritance if they > aren''t then you (by the very nature of what you want to do) need to > precreate them with the appropriate options. >-- Constantin Gonzalez Sun Microsystems GmbH, Germany Platform Technology Group, Global Systems Engineering http://www.sun.de/ Tel.: +49 89/4 60 08-25 91 www.google.com/search?q=constantin+gonzalez Sitz d. Ges.: Sun Microsystems GmbH, Sonnenallee 1, 85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Dr. Roland Boemer Vorsitzender des Aufsichtsrates: Martin Haering