After searching for dm-crypt and ZFS on Linux and finding too little information, I shall ask here. Please keep in mind this in the context of running this in a production environment. We have the need to encypt our data, approximately 30TB on three ZFS volumes under Solaris 10. The volumes currently reside on iscsi sans connected via 10Gb/s ethernet. We have tested Solaris 11 with ZFS encrypted volumes and found the performance to be very poor and have an open bug report with Oracle. We are a Linux shop and since performance is so poor and still no resolution, we are considering ZFS on Linux with dm-crypt. I have read once or twice that if we implemented ZFS + dm-crypt we would loose features, however which features are not specified. We currently mirror the volumes across identical iscsi sans with ZFS and we use hourly ZFS snapshots to update our DR site. Which features of ZFS are lost if we use dm-crypt? My guess would be they are related to raidz but unsure. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121123/b5355841/attachment.html>
On Fri, Nov 23, 2012 at 9:49 AM, John Baxter <johnleebaxter at gmail.com>wrote:> > We have the need to encypt our data, approximately 30TB on three ZFS > volumes under Solaris 10. The volumes currently reside on iscsi sans > connected via 10Gb/s ethernet. We have tested Solaris 11 with ZFS encrypted > volumes and found the performance to be very poor and have an open bug > report with Oracle. > > We are a Linux shop and since performance is so poor and still no > resolution, we are considering ZFS on Linux with dm-crypt. > I have read once or twice that if we implemented ZFS + dm-crypt we would > loose features, however which features are not specified. > We currently mirror the volumes across identical iscsi sans with ZFS and > we use hourly ZFS snapshots to update our DR site. > > Which features of ZFS are lost if we use dm-crypt? My guess would be they > are related to raidz but unsure. > >Why don''t you just use a SAN that supports full drive encryption? There should be basically 0 performance overhead. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121123/e07c7438/attachment.html>
Replacing the SANs is cost prohibitive. On Fri, Nov 23, 2012 at 10:24 AM, Tim Cook <tim at cook.ms> wrote:> > > On Fri, Nov 23, 2012 at 9:49 AM, John Baxter <johnleebaxter at gmail.com>wrote: > >> >> We have the need to encypt our data, approximately 30TB on three ZFS >> volumes under Solaris 10. The volumes currently reside on iscsi sans >> connected via 10Gb/s ethernet. We have tested Solaris 11 with ZFS encrypted >> volumes and found the performance to be very poor and have an open bug >> report with Oracle. >> >> We are a Linux shop and since performance is so poor and still no >> resolution, we are considering ZFS on Linux with dm-crypt. >> I have read once or twice that if we implemented ZFS + dm-crypt we would >> loose features, however which features are not specified. >> We currently mirror the volumes across identical iscsi sans with ZFS and >> we use hourly ZFS snapshots to update our DR site. >> >> Which features of ZFS are lost if we use dm-crypt? My guess would be they >> are related to raidz but unsure. >> >> > > Why don''t you just use a SAN that supports full drive encryption? There > should be basically 0 performance overhead. > > --Tim >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121123/61abbba2/attachment.html>
John Baxter <johnleebaxter at gmail.com> wrote:> After searching for dm-crypt and ZFS on Linux and finding too little > information, I shall ask here. Please keep in mind this in the context of > running this in a production environment. > > We have the need to encypt our data, approximately 30TB on three ZFS > volumes under Solaris 10. The volumes currently reside on iscsi sans > connected via 10Gb/s ethernet. We have tested Solaris 11 with ZFS encrypted > volumes and found the performance to be very poor and have an open bug > report with Oracle.Was the performance acceptable without encryption?> We are a Linux shop and since performance is so poor and still no > resolution, we are considering ZFS on Linux with dm-crypt. > I have read once or twice that if we implemented ZFS + dm-crypt we would > loose features, however which features are not specified. > We currently mirror the volumes across identical iscsi sans with ZFS and we > use hourly ZFS snapshots to update our DR site. > > Which features of ZFS are lost if we use dm-crypt? My guess would be they > are related to raidz but unsure.It depends on where you put the encryption layer. If you put it below ZFS, no ZFS feature has to be lost although bugs in the encryption layer may make the whole setup less reliable. Of course that''s true for Oracle''s ZFS encryption as well. If you put the encryption layer on top of ZFS, features like compression and deduplication should be ineffective. It will not encrypt the ZFS metadata, but it allows you to keep parts of the data on the pool intentionally (or unintentionally) unencrypted. If your application doesn''t work with raw devices, you need a file system on top of the encryption layer again. I''m not aware of anything raidz-related that is lost in either setup. I haven''t used ZFS with dm-crypt on GNU/Linux, but if I had to, I''d put dm-crypt below ZFS and would rather split the pool than put dm-crypt on top of ZFS. My impression is that ext4 on dm-crypt on ZFS is a popular setup (among bloggers), but I have no idea why and certainly wouldn''t want to use it in a production environment. Just in case your GNU/Linux experiments don''t work out, you could also try ZFS on Geli on FreeBSD which works reasonably well. Fabian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121123/9bb77fa0/attachment.bin>
On Nov 23, 2012, at 11:56 AM, Fabian Keil <freebsd-listen at fabiankeil.de> wrote:> > Just in case your GNU/Linux experiments don''t work out, you could > also try ZFS on Geli on FreeBSD which works reasonably well. >For illumos-based distros or Solaris 11, using ZFS with lofi has been well discussed for many years. Prior to the crypto option being integrated as a first class citizen in OpenSolaris, the codename used was "xlofi," so try that in your google searches, or look at the man page for lofiadm -- richard
On 11/23/12 15:49, John Baxter wrote:> After searching for dm-crypt and ZFS on Linux and finding too little > information, I shall ask here. Please keep in mind this in the context > of running this in a production environment. > > We have the need to encypt our data, approximately 30TB on three ZFS > volumes under Solaris 10. The volumes currently reside on iscsi sans > connected via 10Gb/s ethernet. We have tested Solaris 11 with ZFS > encrypted volumes and found the performance to be very poor and have an > open bug report with Oracle.This "bug report" hasn''t reached me yet and I''d really like to be sure if there is a performance bug with ZFS that is unique to encryption I can attempt to resolve it. Can you please provide the bug and/or SR number that Oracle Support gave to you.> We are a Linux shop and since performance is so poor and still no > resolution, we are considering ZFS on Linux with dm-crypt. > I have read once or twice that if we implemented ZFS + dm-crypt we would > loose features, however which features are not specified. > We currently mirror the volumes across identical iscsi sans with ZFS and > we use hourly ZFS snapshots to update our DR site. > > Which features of ZFS are lost if we use dm-crypt? My guess would be > they are related to raidz but unsure. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Darren J Moffat
On 11/30/12 11:41, Darren J Moffat wrote:> > > On 11/23/12 15:49, John Baxter wrote: >> After searching for dm-crypt and ZFS on Linux and finding too little >> information, I shall ask here. Please keep in mind this in the context >> of running this in a production environment. >> >> We have the need to encypt our data, approximately 30TB on three ZFS >> volumes under Solaris 10. The volumes currently reside on iscsi sans >> connected via 10Gb/s ethernet. We have tested Solaris 11 with ZFS >> encrypted volumes and found the performance to be very poor and have an >> open bug report with Oracle. > > This "bug report" hasn''t reached me yet and I''d really like to be sure > if there is a performance bug with ZFS that is unique to encryption I > can attempt to resolve it. > > Can you please provide the bug and/or SR number that Oracle Support gave > to you.For the sake of those on the list, I''ve got these references now. -- Darren J Moffat