zfs discuss - Jul 2012 - encfs on top of zfs

Dear ZFS-Users,

I want to switch to ZFS, but still want to encrypt my data. Native
Encryption for ZFS was added in "ZFS Pool Version Number
30<http://en.wikipedia.org/wiki/ZFS#Release_history>",
but I''m using ZFS on FreeBSD with Version 28. My question is how would
encfs (fuse encryption) affect zfs specific features like data Integrity
and deduplication?

Regards

Tristan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120730/9697f6d8/attachment.html>

On Mon, Jul 30, 2012 at 5:20 AM, Tristan Klocke
<tristan.klocke at googlemail.com> wrote:
> I want to switch to ZFS, but still want to encrypt my data. Native
> Encryption for ZFS was added in "ZFS Pool Version Number 30", but
I''m using
> ZFS on FreeBSD with Version 28. My question is how would encfs (fuse
> encryption) affect zfs specific features like data Integrity and
> deduplication?
If you are using FreeBSD, why not use GELI to provide the block
devices used for the ZFS vdevs?  That''s the "standard" way to
get
encryption and ZFS working on FreeBSD.

-- 
Freddie Cash
fjwcash at gmail.com

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Tristan Klocke
> 
> I want to switch to ZFS, but still want to encrypt my data. Native
Encryption
> for ZFS was added in "ZFS Pool Version Number 30", but
I''m using ZFS on
> FreeBSD with Version 28. My question is how would encfs (fuse encryption)
> affect zfs specific features like data Integrity and deduplication?
Data integrity:  ZFS & encfs will work great together.

Dedup:  First of all, I don''t recommend using dedup under any
circumstance.  Not that it''s unstable or anything, just that the
performance is so horrible, it''s never worth while.  But particularly
with encrypted data, you''re guaranteed to have no duplicate data
anyway, so it would be a pure waste.  Don''t do it.

On 07/31/2012 09:46 AM, opensolarisisdeadlongliveopensolaris
wrote:
> Dedup: First of all, I don''t recommend using dedup under any
> circumstance. Not that it''s unstable or anything, just that the
> performance is so horrible, it''s never worth while. But
particularly
> with encrypted data, you''re guaranteed to have no duplicate data
> anyway, so it would be a pure waste. Don''t do it.
> _______________________________________________ zfs-discuss mailing
> list zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss 
One thing you can do is enable dedup when you copy all your data from
one zpool to another, then, when you''re done, disable dedup.  It will
no
longer waste a ton of memory, and your new volume will have a high dedup
ratio. (Obviously anything you add after you turn dedup off won''t be
deduped.)  You can keep the old pool as a backup, or wipe it or whatever
and later on do the same operation in the other direction.

Anyone know if zfs send | zfs get will maintain the deduped files after
this?  Maybe once deduped you can wipe the old pool, then use get|send
to get a deduped backup?

On Tue, Jul 31, 2012 at 9:36 AM, Ray Arachelian <ray at arachelian.com>
wrote:
> On 07/31/2012 09:46 AM, opensolarisisdeadlongliveopensolaris wrote:
>> Dedup: First of all, I don''t recommend using dedup under any
>> circumstance. Not that it''s unstable or anything, just that
the
>> performance is so horrible, it''s never worth while. But
particularly
>> with encrypted data, you''re guaranteed to have no duplicate
data
>> anyway, so it would be a pure waste. Don''t do it.
>> _______________________________________________ zfs-discuss mailing
>> list zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
> One thing you can do is enable dedup when you copy all your data from
> one zpool to another, then, when you''re done, disable dedup.  It
will no
> longer waste a ton of memory, and your new volume will have a high dedup
> ratio. (Obviously anything you add after you turn dedup off won''t
be
> deduped.)  You can keep the old pool as a backup, or wipe it or whatever
> and later on do the same operation in the other direction.
Once something is written deduped you will always use the memory when
you want to read any files that were written when dedup was enabled,
so you do not save any memory unless you do not normally access most
of your data.

Also don''t let the system crash :D or try to delete too much from the
deduped dataset :D (including snapshots or the dataset itself) because
then you have to reload all (most) of the DDT in order to delete the
files.  This gets a lot of people in trouble (including me at $work
:|) because you need to have the ram available at all times to load
the most (>75% to grab a number out of the air) in case the server
crashes. Otherwise you are stuck with a machine trying to verify its
filesystem for hours. I have one test system that has 4 GB of RAM and
2 TB of deduped data, when it crashes (panic, powerfailure, etc) it
would take 8-12 hours to boot up again.  It now has <1TB of data and
will boot in about 5 minutes or so.
>
> Anyone know if zfs send | zfs get will maintain the deduped files after
> this?  Maybe once deduped you can wipe the old pool, then use get|send
> to get a deduped backup?
No zfs send | zfs recv will not dedup new data unless dedup is on,
however, the existing files remain deduped until the snapshot that is
retaining them is removed.

>From practical work, dedup is only worth is if you have a >10 dedup
ratio or a small deduped dataset because in practice at $work we have
found that 1TB of deduped data require 6-8GB of RAM (using multi
hundred GB files, so 128K record sizes, with smaller record sizes the
ram numbers balloon).  Which means even a low dedup ratio of 3 it is
still cheaper to just buy more disks than find a system that can hold
enough ram to dedup 4-5TB of data (64-128GB of RAM). Of course keep in
mind we at $work only have practical knowledge of systems up to 4 TB
of deduped data.  We are planning some 45+TB and plan on using
compression=gzip-9 and dedup=off.

As far as the OP is concerned, unless you have a dataset that will
dedup well don''t bother with it, use compression instead
(don''t use
both compression and dedup because you will shrink the average record
size and balloon the memory usage).  As mentioned by Ned Harvey, dedup
of encrypted data is probably completely useless (it depends on the
chaining mode, duplicate data in the encrypted dataset may be
encrypted the same way allow for dedup of the encrypted data stream).
However, if the encryption is done below zfs, like what GELI would be
doing by giving the encrypted block device to zfs, then the use of
dedup reverts back the the standard, "are you going to have enough
duplicate data to get a ratio that is high enough to be worth the RAM
requirements?" question.

As I reread the OP, to make sure that the OPs question is actually
answered; use of encryption will not affect zfs integrity beyond
normal issues associated with zfs integrity and disk encryption. That
is, missing encryption keys is like a failed disk (if below zfs) or
any other file on a zfs filesystem (if above zfs).  Zfs integrity
(checksumming and the transactions) are not affected by the underlying
block device (if encryption is below zfs) or the contents of a file on
the system system (if encryption is above zfs).  The "raid" features
of zfs also work no differently, ''n'' encrypted block devices
is no
different that ''n'' plain hard drives as far as ZFS is
concerned.

On Jul 31, 2012, at 10:07 AM, Nigel W wrote:
> On Tue, Jul 31, 2012 at 9:36 AM, Ray Arachelian <ray at
arachelian.com> wrote:
>> On 07/31/2012 09:46 AM, opensolarisisdeadlongliveopensolaris wrote:
>>> Dedup: First of all, I don''t recommend using dedup under
any
>>> circumstance. Not that it''s unstable or anything, just
that the
>>> performance is so horrible, it''s never worth while. But
particularly
>>> with encrypted data, you''re guaranteed to have no
duplicate data
>>> anyway, so it would be a pure waste. Don''t do it.
>>> _______________________________________________ zfs-discuss mailing
>>> list zfs-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>> 
>> One thing you can do is enable dedup when you copy all your data from
>> one zpool to another, then, when you''re done, disable dedup. 
It will no
>> longer waste a ton of memory, and your new volume will have a high
dedup
>> ratio. (Obviously anything you add after you turn dedup off
won''t be
>> deduped.)  You can keep the old pool as a backup, or wipe it or
whatever
>> and later on do the same operation in the other direction.
> 
> Once something is written deduped you will always use the memory when
> you want to read any files that were written when dedup was enabled,
> so you do not save any memory unless you do not normally access most
> of your data.
> 
> Also don''t let the system crash :D or try to delete too much from
the
> deduped dataset :D (including snapshots or the dataset itself) because
> then you have to reload all (most) of the DDT in order to delete the
> files.  This gets a lot of people in trouble (including me at $work
> :|) because you need to have the ram available at all times to load
> the most (>75% to grab a number out of the air) in case the server
> crashes. Otherwise you are stuck with a machine trying to verify its
> filesystem for hours. I have one test system that has 4 GB of RAM and
> 2 TB of deduped data, when it crashes (panic, powerfailure, etc) it
> would take 8-12 hours to boot up again.  It now has <1TB of data and
> will boot in about 5 minutes or so.
I believe what you meant to say was "dedup with HDDs sux." If you had
used fast SSDs instead of HDDs, you will find dedup to be quite fast.
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com
+1-760-896-4422







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120731/5ff54e05/attachment-0001.html>

> Once something is written deduped you will always use the memory when
> you want to read any files that were written when dedup was enabled, so
> you do not save any memory unless you do not normally access most of
> your data.
For reads you don''t need ddt. Also in Solaris 11 (not in Illumos
unfortunately AFAIK) on reads the in-memory ARC will also stay deduped (so
if 10x logical blocks are deduped to 1 and you read all 10 logical copies,
only one block in arc will be allocated). If there are no further
modifications and you only read dedupped data, apart from disk space
savings, there can be very nice improvement in performance as well (less
i/o, more ram for caching, etc.).

> 
> As far as the OP is concerned, unless you have a dataset that will
> dedup well don''t bother with it, use compression instead
(don''t use
> both compression and dedup because you will shrink the average record
> size and balloon the memory usage).
Can you expand a little bit more here?
Dedup+compression works pretty well actually (not counting "standard"
problems with current dedup - compression or not).


-- 
Robert Milkowski
http://milek.blogspot.com

HI

I use GELI with ZFS all the time. Works fine for me so far.


Am 31.07.12 21:54, schrieb Robert Milkowski:
>> Once something is written deduped you will always use the memory when
>> you want to read any files that were written when dedup was enabled, so
>> you do not save any memory unless you do not normally access most of
>> your data.
> 
> For reads you don''t need ddt. Also in Solaris 11 (not in Illumos
> unfortunately AFAIK) on reads the in-memory ARC will also stay deduped (so
> if 10x logical blocks are deduped to 1 and you read all 10 logical copies,
> only one block in arc will be allocated). If there are no further
> modifications and you only read dedupped data, apart from disk space
> savings, there can be very nice improvement in performance as well (less
> i/o, more ram for caching, etc.).
> 
> 
>>
>> As far as the OP is concerned, unless you have a dataset that will
>> dedup well don''t bother with it, use compression instead
(don''t use
>> both compression and dedup because you will shrink the average record
>> size and balloon the memory usage).
> 
> Can you expand a little bit more here?
> Dedup+compression works pretty well actually (not counting
"standard"
> problems with current dedup - compression or not).
> 
>

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Ray Arachelian
> 
> One thing you can do is enable dedup when you copy all your data from
> one zpool to another, then, when you''re done, disable dedup.  It
will no
> longer waste a ton of memory, and your new volume will have a high dedup
> ratio. 
That''s not correct.  It sounds like you are mistakenly believing the
DDT gets held in memory, but actually, it''s held on disk and since it
gets used so much, large portions of it will likely be in ARC/L2ARC. 
Unfortunately, after you dedup a pool and disable dedup, the DDT will still get
used frequently, and still take just as much memory most likely.  But
that''s not the main concern anyway - The main concern is things like
snapshot destroy (or simply rm) which need to unlink blocks.  This requires
decrementing the refcount, which requires finding and writing the DDT entry,
which means a flurry of essentially small random IO.  So the memory &
performance with dedup disabled is just as bad, as long as you previously had
dedup enabled for a signfiicant percentage of your pool.

> Anyone know if zfs send | zfs get will maintain the deduped files after
> this?  Maybe once deduped you can wipe the old pool, then use get|send
> to get a deduped backup?
You can enable the dedup property on a receiving pool, and then the data
received will be dedup''d.  The behavior is dependent on the properties
of the receiving pool.

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Richard Elling
>  
> I believe what you meant to say was "dedup with HDDs sux." If you
had
> used fast SSDs instead of HDDs, you will find dedup to be quite fast.
> ?-- richard
Yes, but this is a linear scale.  Suppose an SSD without dedup is 100x faster
than a HDD without dedup.  And suppose dedup slows down a system by a factor of
10x.  Now your SSD with dedup is only 10x faster than the HDD without dedup.  So
"quite fast" is a relative term.  The SSD with dedup is still faster
than the HDD without dedup, but it''s also slower than the SSD without
dedup.

The extent of fibbing I''m doing is thusly:  In reality, an SSD is about
equally fast with HDD for sequential operations, and about 100x faster for
random IO.  It just so happens that the dedup performance hit is almost purely
random IO, so it''s right in the sweet spot of what SSD''s
handle well.  You can''t use an overly simplified linear model like I
described above - In reality, there''s a grain of truth in what Richard
said, and also a grain of truth in what I said.  The real truth is somewhere in
between what he said and what I said.

No, the SSD will not perform as well with dedup as it does without dedup.  But
the "suppose dedup slows down by 10x" that I described above is not
accurate.  Depending on what you''re doing, dedup might slow down an HDD
by 20x, and it might only slow down SSD by 4x doing the same work load.  Highly
variable, and highly dependent on the specifics of your workload.

On Jul 31, 2012, at 8:05 PM, opensolarisisdeadlongliveopensolaris wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Richard Elling
>> 
>> I believe what you meant to say was "dedup with HDDs sux." If
you had
>> used fast SSDs instead of HDDs, you will find dedup to be quite fast.
>>  -- richard
> 
> Yes, but this is a linear scale.  
No, it is definitely NOT a linear scale. Study Amdahl''s law a little
more carefully.
> Suppose an SSD without dedup is 100x faster than a HDD without dedup.  And
suppose dedup slows down a system by a factor of 10x.  Now your SSD with dedup
is only 10x faster than the HDD without dedup.  So "quite fast" is a
relative term.
Of course it is.
>  The SSD with dedup is still faster than the HDD without dedup, but
it''s also slower than the SSD without dedup.
duh. With dedup you are trading IOPS for space. In general, HDDs have lots of
space and
terrible IOPS. SSDs have less space, but more IOPS. Obviously, as you point out,
the best
solution is lots of space and lots of IOPS.
> The extent of fibbing I''m doing is thusly:  In reality, an SSD is
about equally fast with HDD for sequential operations, and about 100x faster for
random IO.  It just so happens that the dedup performance hit is almost purely
random IO, so it''s right in the sweet spot of what SSD''s
handle well.
In the vast majority of modern systems, there are no sequential I/O workloads.
That is a myth
propagated by people who still think HDDs can be fast.
> You can''t use an overly simplified linear model like I described
above - In reality, there''s a grain of truth in what Richard said, and
also a grain of truth in what I said.  The real truth is somewhere in between
what he said and what I said.
But closer to my truth :-)
> No, the SSD will not perform as well with dedup as it does without dedup. 
But the "suppose dedup slows down by 10x" that I described above is
not accurate.  Depending on what you''re doing, dedup might slow down an
HDD by 20x, and it might only slow down SSD by 4x doing the same work load. 
Highly variable, and highly dependent on the specifics of your workload.
You are making the assumption that the system is not bandwidth limited. This is
a
good assumption for the HDD case, because the media bandwidth is much less 
than the interconnect bandwidth. For SSDs, this assumption is not necessarily
true.
There are SSDs that are bandwidth constrained on the interconnect, and in those
cases, your model fails.
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com
+1-760-896-4422







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120802/27bb937d/attachment-0001.html>