As a Netapp user i have grown very fond of the deduplication feature we have on our filers, and some time ago i heard that dedup was being investigated on zfs aswell. Do you guys know how far this has come? I really like to build a low cost backup appliance using zfs and dedpu.. :) -- This message posted from opensolaris.org
Yes -- dedup is my (and Bill''s) current project. Prototyped in December. Integration this summer. I''ll blog all the details when we integrate, but it''s what you''d expect of ZFS dedup -- synchronous, no limits, etc. Jeff On Tue, Mar 31, 2009 at 02:32:11AM -0700, Joachim Sandvik wrote:> As a Netapp user i have grown very fond of the deduplication feature we have on our filers, and some time ago i heard that dedup was being investigated on zfs aswell. > > Do you guys know how far this has come? > > I really like to build a low cost backup appliance using zfs and dedpu.. :) > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-code mailing list > zfs-code at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-code
Jeff That''s great news. Thanks for sharing that. I''m really looking forward to reading about the details. Best Regards Nigel Smith -- This message posted from opensolaris.org
> Yes -- dedup is my (and Bill''s) current project. > Prototyped in December. > ntegration this summer. I''ll blog all the details > when we integrate, > but it''s what you''d expect of ZFS dedup -- > synchronous, no limits, etc.I am not sure about the "what you''d expect" part. Previous discussions here showed interest for a synchronous version, but also for asynchronous versions (one that works in the background, or one you can run occasionally like a scrub, there are plenty of possibilities). I am not complaining though, the synchronous version is an interesting one, and it should be easier to add the other versions afterwards. -- This message posted from opensolaris.org
Hi, For anyone interested, I have blogged about raidz on-disk layout at: http://mbruning.blogspot.com/2009/04/raidz-on-disk-format.html Comments/corrections are welcome. thanks, max
Awesome news, Jeff. I know you said you''d write about it later, but I want to pose these questions now for several reasons: - I''m excited and eager and can''t wait :-) - There may be things we could do now to prepare existing data and pools for easier dedup later - There may be useful hints in here for documentation, test cases, further RFEs, etc. So, in no particular order: - will it use only the existing checksums, or an additional compare or method? - will it depend on using a particular (eg stronger) checksum? would it help to switch now to that checksum method so blocks written in the meantime are "ready"? (I''m already concerned about the fletcher2 implementation thread and will likely switch anyway) - will it dedup across the entire pool, or only within a dataset? - will it be enable/disable per dataset? (space vs speed) - will it interact with copies=>1? especially where dup blocks exist between datasets that differ in copies= settings? I hope I''d get new ditto blocks for the highest copies= referrer, but then what about when that dataset is destroyed and there are more copies than needed? - will it interact with compression (i.e, does it dedup source blocks or on-disk blocks)? If I write the same files to datasets with differing compression settings, how many copies do I store? - will it detect only whole blocks with the same alignment, or is there something I can do to improve detection of smaller duplicate blocks and split them? - will there be a way for me to examine files for the "dup nature" (I''m thinking of something like seeking for holes) at the app level, to use the information the fs has already discovered? - will it depend on bp-rewrite at all? (for delivery; I presume bp-rewrite will be needed to dedup existing blocks, but is there an implementation dependency that entangles these two somehow, such that we need to wait for both?) - will zfs send be able to avoid sending multiple copies of dup data? -- This message posted from opensolaris.org
Group, Within this long list and am sure an incomplete list... Let me add some thoughts for the future by a non ZFS developer.. The immediate items below are just for thought. Ok,, I assume it at the block level, but easily could be wrong.. Aren''t ZFS file block when modified: read, copied, then inode updated to point to new block. IMO, based on #3, I don''t think you need to support block splitting and if you did, what would prevent heavily modified files/objects degrading to the smallest block supported? Can''t a app find holes now? My (immediate) short list is: 1) How are you going to support backward compatibility to remove existing dups? Where dups are located locally and/or network wide. 2) Other than additional code space and code complication, what level of performance degradation, due to what must be some hash lookup, etc added into the code fastpath? 3) With the storage capacities/density rapidly rising and the ability to mirror data for disaster recovery, load balance, allow single digit/lan ms network access time vs wan access times, then how does a single administrator within the LAN determine the level of support/tradeoffs of this new feature within a global co.. 4) What disk/file objects are considered inappropriate for dedup..? 5) How will you support Direct I/O or will you support wrt Direct I/ O? Mitchell Erblich ------------------------------ On Apr 19, 2009, at 1:15 AM, Daniel Carosone wrote:> Awesome news, Jeff. I know you said you''d write about it later, but > I want to pose these questions now for several reasons: > - I''m excited and eager and can''t wait :-) > - There may be things we could do now to prepare existing data and > pools for easier dedup later > - There may be useful hints in here for documentation, test cases, > further RFEs, etc. > > So, in no particular order: > - will it use only the existing checksums, or an additional compare > or method? > - will it depend on using a particular (eg stronger) checksum? would > it help to switch now to that checksum method so blocks written in > the meantime are "ready"? (I''m already concerned about the > fletcher2 implementation thread and will likely switch anyway) > - will it dedup across the entire pool, or only within a dataset? > - will it be enable/disable per dataset? (space vs speed) > - will it interact with copies=>1? especially where dup blocks exist > between datasets that differ in copies= settings? I hope I''d get > new ditto blocks for the highest copies= referrer, but then what > about when that dataset is destroyed and there are more copies than > needed? > - will it interact with compression (i.e, does it dedup source > blocks or on-disk blocks)? If I write the same files to datasets > with differing compression settings, how many copies do I store? > - will it detect only whole blocks with the same alignment, or is > there something I can do to improve detection of smaller duplicate > blocks and split them? > - will there be a way for me to examine files for the "dup > nature" (I''m thinking of something like seeking for holes) at the > app level, to use the information the fs has already discovered? > - will it depend on bp-rewrite at all? (for delivery; I presume bp- > rewrite will be needed to dedup existing blocks, but is there an > implementation dependency that entangles these two somehow, such > that we need to wait for both?) > - will zfs send be able to avoid sending multiple copies of dup data? > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-code mailing list > zfs-code at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-code
I''ve been trying to keep up with the latest news on ZFS and deduplication, and unfortunately there''s not much news out there and this thread happens to contain the most recent information on the subject. That being said, I found an article that mentioned the following (http://www.technologyandbusiness.com.au/server-hardware-software/News/Sun-shines-more-light-on-Open-Source-Kernels.aspx) - "Sun Microsystems will hold its first-ever Australian Kernel conference in July this year that will examine any open source kernels and technologies within those kernels. ... Sun Microsystems? Sun Fellow and Vice President, Jeff Bonwick and Distinguished Engineer Bill Moore will present the opening keynote for the conference, titled ''Deduplication in ZFS,'' which will likely include a live demonstration." According to http://au.sun.com/sunnews/events/2009/kernel/, the above mentioned conference will be held in Brisbane, Australia, from July 15th to 17th, 2009. Hopefully deduplication will come to SXCE shortly thereafter, and maybe we''ll see it in a second OpenSolaris release this year after 2009.06 is released. -- This message posted from opensolaris.org
On Sat, 16 May 2009 11:43:30 -0700 (PDT) BJ Quinn <bjquinn at seidal.com> wrote:> I''ve been trying to keep up with the latest news on ZFS and deduplication, and unfortunately there''s not much news out there and this thread happens to contain the most recent information on the subject. That being said, I found an article that mentioned the following (http://www.technologyandbusiness.com.au/server-hardware-software/News/Sun-shines-more-light-on-Open-Source-Kernels.aspx) - > > "Sun Microsystems will hold its first-ever Australian Kernel conference in July this year that will examine any open source kernels and technologies within those kernels. ... Sun Microsystems? Sun Fellow and Vice President, Jeff Bonwick and Distinguished Engineer Bill Moore will present the opening keynote for the conference, titled ''Deduplication in ZFS,'' which will likely include a live demonstration." > > According to http://au.sun.com/sunnews/events/2009/kernel/, the above mentioned conference will be held in Brisbane, Australia, from July 15th to 17th, 2009. > > Hopefully deduplication will come to SXCE shortly thereafter, and maybe we''ll see it in a second OpenSolaris release this year after 2009.06 is released.Actually, I''m hoping that Jeff and Bill will have integrated ZFS Deduplication into NV before they get on the plane to come over here for KCA. We''ll just have to wait and see :-) Either way, whenever they integrate it into NV, it''ll be in the next build of SXCE and should show up in OpenSolaris'' dev repo shortly afterwards. There won''t be another OpenSolaris _release_ this year after 2009.06. Cheers, James C. McPherson (chief instigator/agitator for Kernel Conference Australia) -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel
On Sun, May 17, 2009 at 12:39 AM, James C. McPherson <James.McPherson at sun.com> wrote:> There won''t be another OpenSolaris > _release_ this year after 2009.06.What do you mean by that? -- Piotr Jasiukajtis | estibi | SCA OS0072 http://estseg.blogspot.com
On Sun, 17 May 2009 07:58:49 +0200 Piotr Jasiukajtis <estseg at gmail.com> wrote:> On Sun, May 17, 2009 at 12:39 AM, James C. McPherson > <James.McPherson at sun.com> wrote: > > There won''t be another OpenSolaris > > _release_ this year after 2009.06. > What do you mean by that?That we''ve had 2 OpenSolaris binary distro releases so far - 2008.05 and 2008.11. As has been mentioned before, there''s going to be 2009.06, and the next one is planned to be next year some time. Builds of ON are separate to Releases. Yay for taxonomies, and hair-splitting. James -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel
> Either way, whenever they integrate it into NV, it''ll be in > the next build of SXCE and should show up in OpenSolaris'' > dev repo shortly afterwards.Wait, does this mean that you could run the most recent version of OpenSolaris at the time (i.e. 2009.06) and just update to the newest version of ZFS from the dev repository? You''ll have to excuse my Linux background, but filesystems typically aren''t the kind of thing you can update without updating the kernel itself. Never thought of just updating the fs independent of the kernel... or have I misunderstood something altogether? -- This message posted from opensolaris.org
BJ Quinn wrote:>> Either way, whenever they integrate it into NV, it''ll be in >> the next build of SXCE and should show up in OpenSolaris'' >> dev repo shortly afterwards. > > Wait, does this mean that you could run the most recent version of OpenSolaris at the time (i.e. 2009.06) and just update to the newest version of ZFS from the dev repository? > > You''ll have to excuse my Linux background, but filesystems typically aren''t the kind of thing you can update without updating the kernel itself. Never thought of just updating the fs independent of the kernel... or have I misunderstood something altogether?When you ''pkg image-update'' it will update everything consistently so the kernel is updated as well as all the matching commands and libraries. Unlike Linux the OpenSolaris kernel isn''t separate but is part of the same source base as libc and many of the core commands. The builds in the /dev repository carefully lined up collection of all the source bases (consolidations) to form a build. There is on OpenSolaris equivalent to grabbing just the kernel source for Linux. -- Darren J Moffat
> When you ''pkg image-update'' it will update everything consistently so > the kernel is updated as well as all the matching commands and libraries.> Unlike Linux the OpenSolaris kernel isn''t separate but is part of the > same source base as libc and many of the core commands. The builds in > the /dev repository carefully lined up collection of all the source > bases (consolidations) to form a build.Got it. So this means that if I wait for deduplication to show up in the OS dev repository, then I can install 2009.06 and the run pkg image-update. This would update me to build 125 (or whatever) with deduplication, although I''d have to wait for 2010.02 (or whatever) to have that functionality in a "non-dev" version.> There is on OpenSolaris equivalent to grabbing just the kernel source > for Linux.I assume you mean "no" OpenSolaris equivalent. Thanks for the explanation! -- This message posted from opensolaris.org
Update for anyone who''s keeping tabs on this - according to http://au.sun.com/sunnews/events/2009/kernel/speakers.jsp the demonstration will be by both Jeff Bonwick and Bill Moore at 9:15am local time on 7/15. That''s 6:15pm CST / 7:15pm EST on the 14th here in in America. -- This message posted from opensolaris.org
On Tue, 26 May 2009 11:26:49 -0700 (PDT) BJ Quinn <bjquinn at seidal.com> wrote:> Update for anyone who''s keeping tabs on this - according to http://au.sun.com/sunnews/events/2009/kernel/speakers.jsp the demonstration will be by both Jeff Bonwick and Bill Moore at 9:15am local time on 7/15. > > That''s 6:15pm CST / 7:15pm EST on the 14th here in in America.Hi BJ, As the organiser of KCA, I''d just to clarify that while I''m hoping Jeff and Bill demonstrate ZFS deduplication in their keynote, I''m not guaranteeing that they will :-) Registrations for KCA are now open, btw. For the full agenda with abstracts please visit http://wikis.sun.com/display/KCA2009/KCA2009+Conference+Agenda For the conference website please visit http://au.sun.com/sunnews/events/2009/kernel, and for registrations, please go without delay to https://www.conveneit.com/secure/sun/kernel_jul_09. The pricing is very reasonable: students AUD95 earlybird AUD195 (ends 31st May 2009) regular AUD300 cheers, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel
Will the slides by the speakers be available for download? Maybe with a holdback of a couple of months? -- This message posted from opensolaris.org
On Wed, 24 Jun 2009 12:32:02 -0700 (PDT) Orvar Korvar <knatte_fnatte_tjatte at yahoo.com> wrote:> Will the slides by the speakers be available for download? Maybe with a holdback of a couple of months?Yes, the slideware will be available. Hopefully not too long after the conference, but exact dates etc are still to be determined. cheers, James -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel
Any slides, transcript, blog posts, or any other information somewhere on the deduplication keynote? -- This message posted from opensolaris.org
On Sun, 19 Jul 2009 16:34:43 -0700 (PDT) BJ Quinn <bjquinn at seidal.com> wrote:> Any slides, transcript, blog posts, or any other information somewhere > on the deduplication keynote?No, not yet. http://blogs.sun.com/jmcp/entry/kca2009_winding_down_1 James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel
Do we know when dedupe will show up in SXCE and whether it will make it in 2010.02 or not? Also, I''ve noticed that the dedupe presentations still haven''t been posted yet. Is there anywhere else I can go to find out some more details about the dedupe implementation? A google search turns up surprisingly little (nothing, actually), with basically the same search results from before KCA. Not trying to nag, just dying to play around with this feature! -- This message posted from opensolaris.org
> Do we know when dedupe will show up in SXCE and whether it will make it in 2010.02 or not?Well, the first part of the question is actually easy to answer since we know that SXCE will be phased out ~ October. So the answer is "no".> Also, I''ve noticed that the dedupe presentations still haven''t been posted > yet. Is there anywhere else I can go to find out some more details about > the dedupe implementation? A google search turns up surprisingly little > (nothing, actually), with basically the same search results from before > KCA.Nothing more fun than a good conspiracy theory. :-) Regards -- Volker -- ------------------------------------------------------------------------ Volker A. Brandt Consulting and Support for Sun Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim Email: vab at bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgr??e: 45 Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt
Just wanting some reading on dedupe, whether it''s the from the KCA presentations or otherwise. I was "virtually" present at the KCA, but the sound was so bad during the keynote that I couldn''t really understand what was being said. I don''t have a conspiracy theory - there just isn''t any good info on dedupe that I can find. I''d love to learn more about it or try it out. I''ve read the other conspiracy theory posts. That''s not how I think. :) I understand that SXCE will be phased out, but I thought possibly an early version of dedupe would be available before October. Alternatively, I guess I could ask if anyone knows when it will be available to play with in whatever we call the "beta/testing/development/non-release" version of OpenSolaris. Then again, conspicuously, the audio was bad ONLY for the keynote, ALL the presentations have been released EXCEPT for the keynote, Oracle wants to kill OpenSolaris, and GreenBytes has a secret deal with Sun to destroy dedupe, ZFS and open source in general, and to just overall destroy the free world and democracy as we know it. Just kidding. :) -- This message posted from opensolaris.org
Hey James, if the delay in publishing the information on the keynote is a lack of time on your part, if you''d send me the raw video/slides/etc., I''d be more than willing to put together something nice that you could present on the KCA website. -- This message posted from opensolaris.org
I don''t suspect a conspiracy theory, I suspect some legal issues of which I''ve heard rumblings, but is there any update on this? I''m very anxious to see deduplication implemented in ZFS! -- This message posted from opensolaris.org
I would also add a few "End-User" capabilities/Questions for this group. What kind of detailed performance metrics will be available? Will the end-user be able to determine the efficacy of the dedup? Will they be able to break this out by a particular data type, pool or time frame When using compression with Dedup, how will they determine the compression level of the data being written? What tools will help understand throughput performance aspects - Disk, CPU, Network, etc with regards to dedup ingest and replication/mirroring. -- This message posted from opensolaris.org