[zfs-code] ZFS and deduplication?

Daniel Carosone
Sun, 19 Apr 2009 08:15:14 -0000

Awesome news, Jeff.  I know you said you''d write about it later, but I
want to pose these questions now for several reasons:
 - I''m excited and eager and can''t wait :-)
 - There may be things we could do now to prepare existing data and pools for
easier dedup later
 - There may be useful hints in here for documentation, test cases, further
RFEs, etc.

So, in no particular order:
 - will it use only the existing checksums, or an additional compare or method?
 - will it depend on using a particular (eg stronger) checksum? would it help to
switch now to that checksum method so blocks written in the meantime are
"ready"?  (I''m already concerned about the fletcher2
implementation thread and will likely switch anyway)
 - will it dedup across the entire pool, or only within a dataset?
 - will it be enable/disable per dataset? (space vs speed)
 - will it interact with copies=>1? especially where dup blocks exist between
datasets that differ in copies= settings?  I hope I''d get new ditto
blocks for the highest copies= referrer, but then what about when that dataset
is destroyed and there are more copies than needed?
 - will it interact with compression (i.e, does it dedup source blocks or
on-disk blocks)?  If I write the same files to datasets with differing
compression settings, how many copies do I store?
 - will it detect only whole blocks with the same alignment, or is there
something I can do to improve detection of smaller duplicate blocks and split
them?
 - will there be a way for me to examine files for the "dup nature"
(I''m thinking of something like seeking for holes) at the app level, to
use the information the fs has already discovered?
 - will it depend on bp-rewrite at all? (for delivery; I presume bp-rewrite will
be needed to dedup existing blocks, but is there an implementation dependency
that entangles these two somehow, such that we need to wait for both?)
 - will zfs send be able to avoid sending multiple copies of dup data?