Deduplication was committed last night by Mr. Bonwick:> Log message: > PSARC 2009/571 ZFS Deduplication Properties > 6677093 zfs should have dedup capabilityhttp://mail.opensolaris.org/pipermail/onnv-notify/2009-November/010683.html Via c0t0d0s0.org.
Terrific! Can''t wait to read the man pages / blogs about how to use it... Alex. On Mon, Nov 2, 2009 at 12:21 PM, David Magda <dmagda at ee.ryerson.ca> wrote:> Deduplication was committed last night by Mr. Bonwick: > >> Log message: >> PSARC 2009/571 ZFS Deduplication Properties >> 6677093 zfs should have dedup capability > > > http://mail.opensolaris.org/pipermail/onnv-notify/2009-November/010683.html > > > > Via c0t0d0s0.org. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Marie von Ebner-Eschenbach - "Even a stopped clock is right twice a day." - http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html
Why didn''t one of the developers from green-bytes do the commit? :P </sarcasm>
On Mon, Nov 2, 2009 at 2:25 PM, Alex Lam S.L. <alexlamsl at gmail.com> wrote:> Terrific! Can''t wait to read the man pages / blogs about how to use it...Alex, you may wish to check PSARC 2009/571 materials [1] for a sneak preview :) [1] http://arc.opensolaris.org/caselog/PSARC/2009/571/> > Alex. > > On Mon, Nov 2, 2009 at 12:21 PM, David Magda <dmagda at ee.ryerson.ca> wrote: >> Deduplication was committed last night by Mr. Bonwick: >> >>> Log message: >>> PSARC 2009/571 ZFS Deduplication Properties >>> 6677093 zfs should have dedup capability >> >> >> http://mail.opensolaris.org/pipermail/onnv-notify/2009-November/010683.html >> >> >> >> Via c0t0d0s0.org. >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > > > -- > > Marie von Ebner-Eschenbach ?- "Even a stopped clock is right twice a > day." - http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Regards, Cyril
David Magda wrote:> Deduplication was committed last night by Mr. Bonwick: > >> Log message: >> PSARC 2009/571 ZFS Deduplication Properties >> 6677093 zfs should have dedup capability > > > http://mail.opensolaris.org/pipermail/onnv-notify/2009-November/010683.html >And "PSARC 2009/479 zpool recovery support" is in as well: http://mail.opensolaris.org/pipermail/onnv-notify/2009-October/010682.html
> Terrific! Can''t wait to read the man pages / blogs about how to use it...Just posted one: http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup Enjoy, and let me know if you have any questions or suggestions for follow-on posts. Jeff
On Mon, Nov 2, 2009 at 7:20 AM, Jeff Bonwick <Jeff.Bonwick at sun.com> wrote:>> Terrific! Can''t wait to read the man pages / blogs about how to use it... > > Just posted one: > > http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup > > Enjoy, and let me know if you have any questions or suggestions for > follow-on posts. > > JeffOn systems with crypto accelerators (particularly Niagara 2) does the hash calculation code use the crypto accelerators, so long as a supported hash is used? Assuming the answer is yes, have performance comparisons been done between weaker hash algorithms implemented in software and sha256 implemented in hardware? I''ve been waiting very patiently to see this code go in. Thank you for all your hard work (and the work of those that helped too!). -- Mike Gerdts http://mgerdts.blogspot.com/
This is truly awesome news!
What''s the best way to dedup existing datasets? Will send/recv work, or
do we just cp things around?
Regards,
Tristan
Jeff Bonwick wrote:>> Terrific! Can''t wait to read the man pages / blogs about how
to use it...
>>
>
> Just posted one:
>
> http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup
>
> Enjoy, and let me know if you have any questions or suggestions for
> follow-on posts.
>
> Jeff
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
Double WOHOO! Thanks Victor! -- This message posted from opensolaris.org
On 02.11.09 18:38, Ross wrote:> Double WOHOO! Thanks Victor!Thanks should go to Tim Haley, Jeff Bonwick and George Wilson ;-)
Ok, thanks everyone then (but still thanks to Victor for the heads up) :-) On Mon, Nov 2, 2009 at 4:03 PM, Victor Latushkin <Victor.Latushkin at sun.com> wrote:> On 02.11.09 18:38, Ross wrote: >> >> Double WOHOO! ?Thanks Victor! > > Thanks should go to Tim Haley, Jeff Bonwick and George Wilson ;-) >
>> Terrific! Can''t wait to read the man pages / blogs about how to use >> it... > > Just posted one: > > http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup > > Enjoy, and let me know if you have any questions or suggestions for > follow-on posts.Looking at FIPS-180-3 in sections 4.1.2 and 4.1.3 I was thinking that the major leap from SHA256 to SHA512 was a 32-bit to 64-bit step. If the implementation of the SHA256 ( or possibly SHA512 at some point ) algorithm is well threaded then one would be able to leverage those massively multi-core Niagara T2 servers. The SHA256 hash is based on six 32-bit functions whereas SHA512 is based on six 64-bit functions. The CMT Niagara T2 can easily process those 64-bit hash functions and the multi-core CMT trend is well established. So long as context switch times are very low one would think that IO with a SHA512 based de-dupe implementation would be possible and even realistic. That would solve the hash collision concern I would think. Merely thinking out loud here ... -- Dennis Clarke dclarke at opensolaris.ca <- Email related to the open source Solaris dclarke at blastwave.org <- Email related to open source for Solaris
On Mon, Nov 02, 2009 at 12:58:32PM -0500, Dennis Clarke wrote:> Looking at FIPS-180-3 in sections 4.1.2 and 4.1.3 I was thinking that the > major leap from SHA256 to SHA512 was a 32-bit to 64-bit step.ZFS doesn''t have enough room in blkptr_t for 512-bi hashes. Nico --
On Mon, Nov 2, 2009 at 11:58 AM, Dennis Clarke <dclarke at blastwave.org> wrote:> >>> Terrific! Can''t wait to read the man pages / blogs about how to use >>> it... >> >> Just posted one: >> >> http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup >> >> Enjoy, and let me know if you have any questions or suggestions for >> follow-on posts. > > Looking at FIPS-180-3 in sections 4.1.2 and 4.1.3 I was thinking that the > major leap from SHA256 to SHA512 was a 32-bit to 64-bit step. > > If the implementation of the SHA256 ( or possibly SHA512 at some point ) > algorithm is well threaded then one would be able to leverage those > massively multi-core Niagara T2 servers. The SHA256 hash is based on six > 32-bit functions whereas SHA512 is based on six 64-bit functions. The CMT > Niagara T2 can easily process those 64-bit hash functions and the > multi-core CMT trend is well established. So long as context switch times > are very low one would think that IO with a SHA512 based de-dupe > implementation would be possible and even realistic. That would solve the > hash collision concern I would think. > > Merely thinking out loud here ...And my out loud thinking on this says that the crypto accelerator on a T2 system does hardware acceleration of SHA256. NAME n2cp - Ultra-SPARC T2 crypto provider device driver DESCRIPTION The n2cp device driver is a multi-threaded, loadable hardware driver supporting hardware assisted acceleration of the following cryptographic operations, which are built into the Ultra-SPARC T2 CMT processor: DES: CKM_DES_CBC, CKM_DES_ECB DES3: CKM_DES3_CBC, CKM_DES3_ECB, AES: CKM_AES_CBC, CKM_AES_ECB, CKM_AES_CTR RC4: CKM_RC4 MD5: KM_MD5, CKM_MD5_HMAC, CKM_MD5_HMAC_GENERAL, CKM_SSL3_MD5_MAC SHA-1: CKM_SHA_1, CKM_SHA_1_HMAC, CKM_SHA_1_HMAC_GENERAL, CKM_SSL3_SHA1_MAC SHA-256:CKM_SHA256, CKM_SHA256_HMAC, CKM_SHA256_HMAC_GENERAL According to page 35 of http://www.slideshare.net/ramesh_r_nagappan/wirespeed-cryptographic-acceleration-for-soa-and-java-ee-security, a T2 CPU can do 41 Gb/s of SHA256. The implication here is that this keeps the MAU''s busy but the rest of the core is still idle for things like compression, TCP, etc. -- Mike Gerdts http://mgerdts.blogspot.com/
Okay, nice to hear ZFS can now use dedup. But how can I "update" my current OpenSolaris (2009.06) or Solaris 10 (5/09) to use this. Or have I wait for a new stable release of Solaris 10 / OpenSolaris? -- Daniel -- This message posted from opensolaris.org
On 03/11/2009, at 7:32 AM, Daniel Streicher wrote:> But how can I "update" my current OpenSolaris (2009.06) or Solaris > 10 (5/09) to use this. > Or have I wait for a new stable release of Solaris 10 / OpenSolaris?For OpenSolaris, you change your repository and switch to the development branches - should be available to public in about 3-3.5 weeks time. Plenty of instructions on how to do this on the net and in this list. For Solaris, you need to wait for the next update release. cheers, James
James Lever wrote:> > On 03/11/2009, at 7:32 AM, Daniel Streicher wrote: > >> But how can I "update" my current OpenSolaris (2009.06) or Solaris 10 >> (5/09) to use this. >> Or have I wait for a new stable release of Solaris 10 / OpenSolaris? > > For OpenSolaris, you change your repository and switch to the > development branches - should be available to public in about 3-3.5 > weeks time. Plenty of instructions on how to do this on the net and in > this list. > > For Solaris, you need to wait for the next update release.at which stage a patch ( kernel Patch ) will be released that can be applied to pre update 9 releases to get the latest zpool version, existing pools would require a zpool upgrade. Enda> > cheers, > James > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Looks great - and by the time OpenSolaris build has it, I will have a brand new laptop to put it on ;-) One question though - I have a file server at home with 4x750GB on raidz1. When I upgrade to the latest build and set dedup=on, given that it does not have an offline mode, there is no way to operate on the existing dataset? As a workaround I can move files in and out of the pool through an external 500GB HDD, and with the ZFS snapshots I don''t really risk much about losing data if anything goes (not too horribly, anyway) wrong. Thanks to you guys again for the great work! Alex. On Mon, Nov 2, 2009 at 1:20 PM, Jeff Bonwick <Jeff.Bonwick at sun.com> wrote:>> Terrific! Can''t wait to read the man pages / blogs about how to use it... > > Just posted one: > > http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup > > Enjoy, and let me know if you have any questions or suggestions for > follow-on posts. > > Jeff >-- Mike Ditka - "If God had wanted man to play soccer, he wouldn''t have given us arms." - http://www.brainyquote.com/quotes/authors/m/mike_ditka.html
ZFS dedup will be in snv_128, but putbacks to snv_128 will not likely close till the end of this week. The OpenSolaris dev repository was updated to snv_126 last Thursday: http://mail.opensolaris.org/pipermail/opensolaris-announce/2009-October/001317.html So it looks like about 5 weeks before the dev repository will be updated to snv_128. Then we see if any bugs emerge as we all rush to test it out... Regards Nigel Smith -- This message posted from opensolaris.org
Mike Gerdts wrote:> On Mon, Nov 2, 2009 at 7:20 AM, Jeff Bonwick <Jeff.Bonwick at sun.com> wrote: >>> Terrific! Can''t wait to read the man pages / blogs about how to use it... >> Just posted one: >> >> http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup >> >> Enjoy, and let me know if you have any questions or suggestions for >> follow-on posts. >> >> Jeff > > On systems with crypto accelerators (particularly Niagara 2) does the > hash calculation code use the crypto accelerators, so long as a > supported hash is used?Not yet, it is coming. Currently ZFS has a private copy of SHA256 (for legacy reasons). I have an RTI pending to switch it to the same copy that the crypto framework uses. That is an optimised software implementation (SPARC, Intel and AMD64) but won''t yet use the Niagara 2 or on chip crypto. There is an issue with very early boot and the crypto framework I have to resolve so that will come later. > Assuming the answer is yes, have performance> comparisons been done between weaker hash algorithms implemented in > software and sha256 implemented in hardware?I''ve done some comparisons on that.
Great stuff, Jeff and company. You all rock. =-) A potential topic for the follow-up posts: auto-ditto, and the philosophy behind choosing a default threshold for creating a second copy. -- This message posted from opensolaris.org
I was under the impression that you can create a new zfs dataset and turn on the dedup functionality, and copy your data to it. Or am I wrong? -- This message posted from opensolaris.org
Orvar Korvar wrote:> I was under the impression that you can create a new zfs dataset and turn on the dedup functionality, and copy your data to it. Or am I wrong?you don''t even have to create a new dataset just do: # zfs set dedup=on <dataset> -- Darren J Moffat
Darren J Moffat wrote: Orvar Korvar wrote: I was under the impression that you can create a new zfs dataset and turn on the dedup functionality, and copy your data to it. Or am I wrong? you don''t even have to create a new dataset just do: # zfs set dedup=on But like all ZFS functions will that not only get applied, when you (re)write (old)new data, like compression=on ? Which leads to the question would a scrub activate dedupe? www.eagle.co.nz This email is confidential and may be legally privileged. If received in error please destroy and immediately notify us. _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Trevor Pretty wrote:> > > Darren J Moffat wrote: >> Orvar Korvar wrote: >> >>> I was under the impression that you can create a new zfs dataset and turn on the dedup functionality, and copy your data to it. Or am I wrong? >>> >> >> you don''t even have to create a new dataset just do: >> >> # zfs set dedup=on <dataset> >> > But like all ZFS functions will that not only get applied, when you > (re)write (old)new data, like compression=on ?Correct but if you are creating a new dataset you are writting new data anyway.> Which leads to the question would a scrub activate dedupe?Not at this time now. -- Darren J Moffat