Looking for help regaining access to
encrypted ZFS file systems that
stopped accepting the encryption key.
I have a file server with a setup
as follows:
Solaris 11 Express 1010.11/snv_151a
8 x 2-TB disks, each one divided
into three equal size partitions,
three raidz3 pools built from a
"slice" across matching partitions:
Disk 1 Disk 8 zpools
+--+ +--+
|p1| .. |p1| <- slice_0
+--+ +--+
|p2| .. |p2| <- slice_1
+--+ +--+
|p3| .. |p3| <- slice_2
+--+ +--+
zpool status shows:
...
NAME STATE
slice_0 ONLINE
raidz3-0 ONLINE
c7t0d0s0 ONLINE
c7t1d0s0 ONLINE
c7t2d0s0 ONLINE
c7t3d0s0 ONLINE
c7t4d0s0 ONLINE
c7t5d0s0 ONLINE
c7t6d0s0 ONLINE
c7t7d0s0 ONLINE
...
And several file systems on each pool:
zfs list shows:
rpool
...
rpool/export
rpool/export/home
rpool/export/home/user1
...
slice_0
slice_0/base
slice_0/base/fsys_0_1
...
slice_0/base/fsys_0_last
slice_1
slice_1/base
slice_1/base/fsys_1_1
...
slice_1/base/fsys_1_last
...
etc.
The intermediate "base" file systems
are there only to set attributes
to be inherited by all other file
systems in the same pool.
They were created with encryption
on, forcing all others to be encrypted.
The keysource for slice_?/base
was set to
"passphrase,prompt"
while creating the file systems.
Then I stored the keys (one key per
pool) in files in a subdirectory
of home/user1, and set keysource for
slice_0/base to
"passphrase,file:///export/home/user1/keys/key_0"
(Similarly for the other two pools)
So far so good.
Several weeks and several terabytes
of data later, I decided to relocate
the files with the encryption keys
from a subdir of user1 to a subdir
of root. Copied the files and set
slice_0/base keysource to
"passphrase,file:///root/keys/key_0", etc.
That broke it. After doing that, the base
file systems (that contain no data files)
can be mounted, but trying to mount any
other fs fails with the message:
"cannot load key for ''slice_?/base/fsys_?_?'': incorrect
key.
Using "zfs set" I can set the keysource
back and forth to the original directory
and the new one, or to prompt, etc.
I can change the "canmount" attribute,
etc., but not actually mount anything.
Tried changing the files attributes
to readable by all or only by owner.
Tried setting the keysource locally for
each fs with no success (other than not
being able to set it back to inherited
from base.)
Any other thing I can do? Most of the
data is either old junk or things I can
rip again or download again, but there
are some files I can not recover from
anywhere else.
Thanks,
--
Roberto Waltman
On 02/18/12 05:12, Roberto Waltman wrote:> Solaris 11 Express 1010.11/snv_151aI strongly suggest upgrading to Solaris 11 there have been some important ZFS and specifically ZFS encryption related bug fixes.> They were created with encryption > on, forcing all others to be encrypted. > > The keysource for slice_?/base > was set to > "passphrase,prompt" > while creating the file systems. > > Then I stored the keys (one key per > pool) in files in a subdirectory > of home/user1, and set keysource for > slice_0/base to > "passphrase,file:///export/home/user1/keys/key_0" > (Similarly for the other two pools)Did you ever export the slice_0 pool and reimport it or reboot the server ? Basically are you and ZFS both 100% sure you had the correct passphrases stored in those files ?> So far so good. > Several weeks and several terabytes > of data later, I decided to relocate > the files with the encryption keys > from a subdir of user1 to a subdir > of root. Copied the files and set > slice_0/base keysource to > "passphrase,file:///root/keys/key_0", etc.Exactly how did you do that ? zfs key -c -o keysource=passphrase,file:///root/keys/key_0 or zfs set keysource=passphrase,file:///root/keys/key_0 The first does a key change and actually reencryptes the on disk data encryption keys using the newly generated AES wrapping key that is derived from the passphrase. The second only change where to find the passphrase.> That broke it. After doing that, the base > file systems (that contain no data files) > can be mounted, but trying to mount any > other fs fails with the message: > "cannot load key for ''slice_?/base/fsys_?_?'': incorrect key.Can post some sample output of: zfs get -r encryption,keysource slice_0 In particular include a few examples of the filesystems you call ''base'' and the fsys ones. What is important here is understanding where the encryption and keysource properties are set and where they are inherited. -- Darren J Moffat
Darren J Moffat wrote:
Thanks for the reply,
> I strongly suggest upgrading to Solaris 11 there have been some
> important ZFS and specifically ZFS encryption related bug fixes.
Will do. (At least temporarily, until this
problem is solved. Long term plan is
switching to FreeNAS, even if that means
running an older version of ZFS)
>> They were created with encryption
>> on, forcing all others to be encrypted.
>>
>> The keysource for slice_?/base
>> was set to
>> "passphrase,prompt"
>> while creating the file systems.
>>
>> Then I stored the keys (one key per
>> pool) in files in a subdirectory
>> of home/user1, and set keysource for
>> slice_0/base to
>> "passphrase,file:///export/home/user1/keys/key_0"
>> (Similarly for the other two pools)
>
> Did you ever export the slice_0 pool and reimport it or reboot the
> server ? Basically are you and ZFS both 100% sure you had the correct
> passphrases stored in those files ?
The system was rebooted many times
during a period of weeks,without
any problems.
(This is a home file server,
powered up on only on evenings and
weekends, when I am at home.)
The trouble began when I tried to
change the passphrase location as
described below.
>> So far so good.
>> Several weeks and several terabytes
>> of data later, I decided to relocate
>> the files with the encryption keys
>> from a subdir of user1 to a subdir
>> of root. Copied the files and set
>> slice_0/base keysource to
>> "passphrase,file:///root/keys/key_0", etc.
>
> Exactly how did you do that ?
>
> zfs key -c -o keysource=passphrase,file:///root/keys/key_0
>
> or
>
> zfs set keysource=passphrase,file:///root/keys/key_0
>
> The first does a key change and actually reencryptes the on disk data
> encryption keys using the newly generated AES wrapping key that is
> derived from the passphrase. The second only change where to find the
> passphrase.
First, I did the 2nd. (Change location only)
I believe I tried the first form also *after*
things were already broken, but I''m sure the
passphrases were identical: slice_08, slice_18
and slice_28 for each pools 0/1/2. - The ''8''
to bring the length to the minimal
requirement of 8 characters.
( My goal for using encryption was just to
obfuscate the contents if, for example, I
send a disk out for repair; not to hide
anything from the NSA )
Question: I believed the keys generated from a
passphrase depend only on the passphrase, and
not on how it is provided or where it is stored.
Is this a true statement?
>> That broke it. After doing that, the base
>> file systems (that contain no data files)
>> can be mounted, but trying to mount any
>> other fs fails with the message:
>> "cannot load key for ''slice_?/base/fsys_?_?'':
incorrect key.
>
> Can post some sample output of:
>
> zfs get -r encryption,keysource slice_0
Actual commands output for slice_2/... pasted at the end.
The keysource was originally, (in the still working
system,) inherited from slice_2/base. It is now set
locally. At this moment some filesystems use "prompt",
others use "file", neither works.
I did try setting the keysource to "prompt,
to "file" at the original location [ actual path:
file:///export/home/trouser/passphrases/slice_2_passphrase ]
to "file" at the new location [ actual path:
file:///root/passphrases/slice_2_passphrase ]
for all file systems. It always failed.
> In particular include a few examples of the filesystems you call
''base''
> and the fsys ones.
>
> What is important here is understanding where the encryption and
> keysource properties are set and where they are inherited.
========================================== zfs get -r encryption,keysource
slice_2
==========================================
NAME PROPERTY VALUE
SOURCE
slice_2 encryption on
local
slice_2 keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base encryption on
local
slice_2/base keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/bitsavers encryption on
inherited from slice_2/base
slice_2/base/bitsavers keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/codesourcery encryption on
inherited from slice_2/base
slice_2/base/codesourcery keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/embedded encryption on
inherited from slice_2/base
slice_2/base/embedded keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/flightgear encryption on
inherited from slice_2/base
slice_2/base/flightgear keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/forth encryption on
inherited from slice_2/base
slice_2/base/forth keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/functional encryption on
inherited from slice_2/base
slice_2/base/functional keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/linux encryption on
inherited from slice_2/base
slice_2/base/linux keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/minix encryption on
inherited from slice_2/base
slice_2/base/minix keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/modula_n_oberon encryption on
inherited from slice_2/base
slice_2/base/modula_n_oberon keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/trimslice encryption on
inherited from slice_2/base
slice_2/base/trimslice keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/ubuntu encryption on
inherited from slice_2/base
slice_2/base/ubuntu keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/unclassified_2 encryption on
inherited from slice_2/base
slice_2/base/unclassified_2 keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/x-plane encryption on
inherited from slice_2/base
slice_2/base/x-plane keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
======================================= zfs get all slice_2
=======================================
NAME PROPERTY VALUE
SOURCE
slice_2 type filesystem
-
slice_2 creation Sun Aug 14 0:16 2011
-
slice_2 used 136G
-
slice_2 available 2.83T
-
slice_2 referenced 74.5K
-
slice_2 compressratio 1.00x
-
slice_2 mounted yes
-
slice_2 quota none
default
slice_2 reservation none
default
slice_2 recordsize 128K
default
slice_2 mountpoint /slices/slice_2
local
slice_2 sharenfs off
local
slice_2 checksum sha256-mac
local
slice_2 compression off
local
slice_2 atime off
local
slice_2 devices off
local
slice_2 exec on
default
slice_2 setuid on
default
slice_2 readonly off
default
slice_2 zoned off
default
slice_2 snapdir hidden
default
slice_2 aclinherit restricted
default
slice_2 canmount on
default
slice_2 xattr on
default
slice_2 copies 1
default
slice_2 version 5
-
slice_2 utf8only on
-
slice_2 normalization none
-
slice_2 casesensitivity sensitive
-
slice_2 vscan off
default
slice_2 nbmand off
default
slice_2 sharesmb off
default
slice_2 refquota none
default
slice_2 refreservation none
default
slice_2 primarycache all
default
slice_2 secondarycache all
default
slice_2 usedbysnapshots 0
-
slice_2 usedbydataset 74.5K
-
slice_2 usedbychildren 136G
-
slice_2 usedbyrefreservation 0
-
slice_2 logbias latency
default
slice_2 dedup off
default
slice_2 mlslabel none
default
slice_2 sync standard
default
slice_2 encryption on
local
slice_2 keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2 keystatus available
-
slice_2 rekeydate Sun Aug 14 0:16 2011
local
slice_2 rstchown on
default
======================================= zfs get all slice_2/base
=======================================
NAME PROPERTY VALUE
SOURCE
slice_2/base type filesystem
-
slice_2/base creation Sun Aug 14 0:16 2011
-
slice_2/base used 136G
-
slice_2/base available 2.83T
-
slice_2/base referenced 77.0K
-
slice_2/base compressratio 1.00x
-
slice_2/base mounted yes
-
slice_2/base quota none
default
slice_2/base reservation none
default
slice_2/base recordsize 128K
default
slice_2/base mountpoint /slices/slice_2/base
local
slice_2/base sharenfs off
local
slice_2/base checksum sha256-mac
local
slice_2/base compression off
local
slice_2/base atime off
local
slice_2/base devices off
local
slice_2/base exec on
default
slice_2/base setuid on
default
slice_2/base readonly off
default
slice_2/base zoned off
default
slice_2/base snapdir hidden
default
slice_2/base aclinherit restricted
default
slice_2/base canmount on
default
slice_2/base xattr on
default
slice_2/base copies 1
default
slice_2/base version 5
-
slice_2/base utf8only on
-
slice_2/base normalization none
-
slice_2/base casesensitivity sensitive
-
slice_2/base vscan off
default
slice_2/base nbmand off
default
slice_2/base sharesmb off
local
slice_2/base refquota none
default
slice_2/base refreservation none
default
slice_2/base primarycache all
default
slice_2/base secondarycache all
default
slice_2/base usedbysnapshots 0
-
slice_2/base usedbydataset 77.0K
-
slice_2/base usedbychildren 136G
-
slice_2/base usedbyrefreservation 0
-
slice_2/base logbias latency
default
slice_2/base dedup off
default
slice_2/base mlslabel none
default
slice_2/base sync standard
default
slice_2/base encryption on
local
slice_2/base keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base keystatus available
-
slice_2/base rekeydate Sun Aug 14 0:16 2011
local
slice_2/base rstchown on
default
======================================= zfs get all slice_2/base/bitsavers
=======================================
NAME PROPERTY VALUE
SOURCE
slice_2/base/bitsavers type filesystem
-
slice_2/base/bitsavers creation Sun Aug 14 0:16 2011
-
slice_2/base/bitsavers used 79.5K
-
slice_2/base/bitsavers available 2.83T
-
slice_2/base/bitsavers referenced 79.5K
-
slice_2/base/bitsavers compressratio 1.00x
-
slice_2/base/bitsavers mounted no
-
slice_2/base/bitsavers quota none
default
slice_2/base/bitsavers reservation none
default
slice_2/base/bitsavers recordsize 128K
default
slice_2/base/bitsavers mountpoint /global/bitsavers/
local
slice_2/base/bitsavers sharenfs off
local
slice_2/base/bitsavers checksum sha256-mac
inherited from slice_2/base
slice_2/base/bitsavers compression off
inherited from slice_2/base
slice_2/base/bitsavers atime off
inherited from slice_2/base
slice_2/base/bitsavers devices off
inherited from slice_2/base
slice_2/base/bitsavers exec on
default
slice_2/base/bitsavers setuid on
default
slice_2/base/bitsavers readonly off
default
slice_2/base/bitsavers zoned off
default
slice_2/base/bitsavers snapdir hidden
default
slice_2/base/bitsavers aclinherit restricted
default
slice_2/base/bitsavers canmount on
default
slice_2/base/bitsavers xattr on
default
slice_2/base/bitsavers copies 1
default
slice_2/base/bitsavers vscan off
default
slice_2/base/bitsavers nbmand off
default
slice_2/base/bitsavers sharesmb name=bitsavers
local
slice_2/base/bitsavers refquota none
default
slice_2/base/bitsavers refreservation none
default
slice_2/base/bitsavers primarycache all
default
slice_2/base/bitsavers secondarycache all
default
slice_2/base/bitsavers usedbysnapshots 0
-
slice_2/base/bitsavers usedbydataset 79.5K
-
slice_2/base/bitsavers usedbychildren 0
-
slice_2/base/bitsavers usedbyrefreservation 0
-
slice_2/base/bitsavers logbias latency
default
slice_2/base/bitsavers dedup off
default
slice_2/base/bitsavers mlslabel none
default
slice_2/base/bitsavers sync standard
default
slice_2/base/bitsavers encryption on
inherited from slice_2/base
slice_2/base/bitsavers keysource
passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase local
slice_2/base/bitsavers keystatus unavailable
-
slice_2/base/bitsavers rekeydate Sun Aug 14 0:16 2011
local
slice_2/base/bitsavers rstchown on
default
======================================= trying to mount
=======================================
root at turbofan:~# cat /export/home/trouser/passphrases/slice_2_passphrase
slice_28
root at turbofan:~# zfs mount slice_2/base/bitsavers
cannot load key for ''slice_2/base/bitsavers'': incorrect key.
root at turbofan:~# zfs set keysource=passphrase,prompt slice_2/base/bitsavers
root at turbofan:~# zfs mount slice_2/base/bitsavers
Enter passphrase for ''slice_2/base/bitsavers'': [ entering
slice_28 ]
cannot load key for ''slice_2/base/bitsavers'': incorrect key.
root at turbofan:~#
On 02/21/12 01:58, Roberto Waltman wrote:> First, I did the 2nd. (Change location only) > I believe I tried the first form also *after* > things were already broken, but I''m sure the > passphrases were identical: slice_08, slice_18 > and slice_28 for each pools 0/1/2. - The ''8'' > to bring the length to the minimal > requirement of 8 characters.A ''zfs key -c'' won''t work unless a ''zfs key -l'' or ''zfs mount'' has successfully loaded the key first. Can you send the ''zpool history slice_2'' output so I can see what commands have been run.> ( My goal for using encryption was just to > obfuscate the contents if, for example, I > send a disk out for repair; not to hide > anything from the NSA ) > > Question: I believed the keys generated from a > passphrase depend only on the passphrase, and > not on how it is provided or where it is stored. > Is this a true statement?Almost, the passphrase case also depends on a hidden property called "salt" that is updated only when you do ''zfs key -c'' and was set to a random value at the time the dataset was created. Did you ever do a send|recv of these filesystems ? There was a bug with send|recv in 151a that has since been fixed that could cause the salt to be zero''d out in some cases.> slice_2/base/bitsavers keysource > passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase localThis is the interesting part you have set the keysource explicitly on every leaf dataset - you didn''t need to do that it would have been inherited. What this means is that even though you have the same passphrase for each dataset the actual data encryption key is different because the passphrase value plus the hidden salt property are used together to generated the wrapping key. -- Darren J Moffat
On Tue, Feb 21, 2012 at 11:12:14AM +0000, Darren J Moffat wrote:> Did you ever do a send|recv of these filesystems ? There was a bug with > send|recv in 151a that has since been fixed that could cause the salt to > be zero''d out in some cases.Ah, so that''s what that was. I hit this problem some time ago, as was discussed here. Unfortunately, I also wrote more data into the recv''d filesystem before the next reboot, and only after that did the new fs become eunmountable. So, now that the bug is understood and fixed, if i still have the original dataset (source of the send), can I use that to recover the salt and thus the keys and new data? -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120222/74cebd05/attachment.bin>
Darren J Moffat wrote:>... > Can you send the ''zpool history slice_2'' output so I can see what > commands have been run. >Pasted at the end - My recollection of the events was wrong, I started with "keysource=passphrase,file..." not prompt. (Used prompt in earlier experiments and mixed things up) > Did you ever do a send|recv of these filesystems ? There was a bug with> send|recv in 151a that has since been fixed that could cause the salt to > be zero''d out in some cases.No.> This is the interesting part you have set the keysource explicitly on > every leaf dataset - you didn''t need to do that it would have been > inherited. > > What this means is that even though you have the same passphrase for > each dataset the actual data encryption key is different because the > passphrase value plus the hidden salt property are used together to > generated the wrapping key.Are you saying "Abandon all hope?" ################################# slice_2 history ################################# ##### Notes: ##### ##### (a) Some groups of commands are repeated because ##### I was (re)running them from a shell script. ##### ##### (b) A couple filesystems were created, destroyed, then ##### recreated. I copied some data later I did not want ##### there, and/or was checking how long the operation ##### took. ##### ##### (c) I deleted a bunch (100+) of set sharesmb and set ##### sharenfs commands that just added clutter ##### ##### (d) Have a symbolic link /home pointing to /export/home ##### History for ''slice_2'': 2011-08-14.00:16:41 zpool create -O atime=off -O compression=off -O devices=off -O encryption=on -O keysource=passphrase,file:///home/trouser/slice_2_passphrase -O mountpoint=none -O utf8only=on slice_2 raidz3 c7t0d0s2 c7t1d0s2 c7t2d0s2 c7t3d0s2 c7t4d0s2 c7t5d0s2 c7t6d0s2 c7t7d0s2 2011-08-14.00:16:41 zfs create -o atime=off -o compression=off -o devices=off -o encryption=on -o keysource=passphrase,file:///home/trouser/slice_2_passphrase -o mountpoint=none -o utf8only=on slice_2/base 2011-08-14.00:16:43 zfs create -o mountpoint=/global/codesourcery slice_2/base/codesourcery 2011-08-14.00:16:44 zfs create -o mountpoint=/global/x-plane slice_2/base/x-plane 2011-08-14.00:16:46 zfs create -o mountpoint=/global/modula_n_oberon slice_2/base/modula_n_oberon 2011-08-14.00:16:47 zfs create -o mountpoint=/global/filer_2 slice_2/base/filer_2 2011-08-14.00:16:48 zfs create -o mountpoint=/global/forth slice_2/base/forth 2011-08-14.00:16:50 zfs create -o mountpoint=/global/functional slice_2/base/functional 2011-08-14.00:16:51 zfs create -o mountpoint=/global/bitsavers slice_2/base/bitsavers 2011-08-14.00:16:56 zfs create -o mountpoint=/global/flightgear slice_2/base/flightgear 2011-08-14.11:54:31 zfs set mountpoint=/slices/slice_2/base slice_2/base 2011-08-14.11:56:51 zfs mount slice_2/base/bitsavers 2011-08-14.11:56:51 zfs mount slice_2/base/codesourcery 2011-08-14.11:56:51 zfs mount slice_2/base/filer_2 2011-08-14.11:56:51 zfs mount slice_2/base/flightgear 2011-08-14.11:56:51 zfs mount slice_2/base/forth 2011-08-14.11:56:51 zfs mount slice_2/base/functional 2011-08-14.11:56:51 zfs mount slice_2/base/modula_n_oberon 2011-08-14.11:56:51 zfs mount slice_2/base/x-plane 2011-08-16.20:41:11 zfs mount slice_2/base 2011-08-16.20:41:11 zfs mount slice_2/base/bitsavers 2011-08-16.20:41:11 zfs mount slice_2/base/codesourcery 2011-08-16.20:41:11 zfs mount slice_2/base/filer_2 2011-08-16.20:41:11 zfs mount slice_2/base/flightgear 2011-08-16.20:41:11 zfs mount slice_2/base/forth 2011-08-16.20:41:11 zfs mount slice_2/base/functional 2011-08-16.20:41:11 zfs mount slice_2/base/modula_n_oberon 2011-08-16.20:41:11 zfs mount slice_2/base/x-plane 2011-08-16.21:47:37 zfs create -o mountpoint=/global/embedded slice_2/base/embedded 2011-08-18.00:00:53 zfs mount slice_2/base 2011-08-18.00:00:53 zfs mount slice_2/base/bitsavers 2011-08-18.00:00:58 zfs mount slice_2/base/codesourcery 2011-08-18.00:00:58 zfs mount slice_2/base/embedded 2011-08-18.00:00:58 zfs mount slice_2/base/filer_2 2011-08-18.00:00:58 zfs mount slice_2/base/flightgear 2011-08-18.00:00:58 zfs mount slice_2/base/forth 2011-08-18.00:00:58 zfs mount slice_2/base/functional 2011-08-18.00:00:58 zfs mount slice_2/base/modula_n_oberon 2011-08-18.00:00:58 zfs mount slice_2/base/x-plane 2011-08-18.21:21:27 zfs mount slice_2/base 2011-08-18.21:21:27 zfs mount slice_2/base/bitsavers 2011-08-18.21:21:27 zfs mount slice_2/base/codesourcery 2011-08-18.21:21:27 zfs mount slice_2/base/embedded 2011-08-18.21:21:27 zfs mount slice_2/base/filer_2 2011-08-18.21:21:27 zfs mount slice_2/base/flightgear 2011-08-18.21:21:27 zfs mount slice_2/base/forth 2011-08-18.21:21:27 zfs mount slice_2/base/functional 2011-08-18.21:21:27 zfs mount slice_2/base/modula_n_oberon 2011-08-18.21:21:27 zfs mount slice_2/base/x-plane 2011-08-19.22:06:33 zfs mount slice_2/base 2011-08-19.22:06:33 zfs mount slice_2/base/bitsavers 2011-08-19.22:06:33 zfs mount slice_2/base/codesourcery 2011-08-19.22:06:33 zfs mount slice_2/base/embedded 2011-08-19.22:06:33 zfs mount slice_2/base/filer_2 2011-08-19.22:06:33 zfs mount slice_2/base/flightgear 2011-08-19.22:06:33 zfs mount slice_2/base/forth 2011-08-19.22:06:33 zfs mount slice_2/base/functional 2011-08-19.22:06:33 zfs mount slice_2/base/modula_n_oberon 2011-08-19.22:06:33 zfs mount slice_2/base/x-plane 2011-08-19.23:52:32 zfs destroy slice_2/base/filer_2 2011-08-20.13:59:26 zfs mount slice_2/base 2011-08-20.13:59:26 zfs mount slice_2/base/bitsavers 2011-08-20.13:59:26 zfs mount slice_2/base/codesourcery 2011-08-20.13:59:26 zfs mount slice_2/base/embedded 2011-08-20.13:59:26 zfs mount slice_2/base/flightgear 2011-08-20.13:59:26 zfs mount slice_2/base/forth 2011-08-20.13:59:26 zfs mount slice_2/base/functional 2011-08-20.13:59:26 zfs mount slice_2/base/modula_n_oberon 2011-08-20.13:59:26 zfs mount slice_2/base/x-plane 2011-08-20.14:17:08 zfs create -o mountpoint=/global/minix slice_2/base/minix 2011-08-20.15:30:28 zfs create -o mountpoint=/global/unclassified_2 slice_2/base/unclassified_2 2011-08-20.15:37:47 zfs create -o mountpoint=/global/trimslice slice_2/base/trimslice 2011-08-20.15:39:01 zfs destroy slice_2/base/trimslice 2011-08-20.15:39:12 zfs create -o mountpoint=/global/trimslice slice_2/base/trimslice 2011-08-20.20:24:44 zfs create -o mountpoint=/global/linux slice_2/base/linux 2011-08-20.20:25:01 zfs create -o mountpoint=/global/ubuntu slice_2/base/ubuntu ##### everything works ok up to this point, the following commands break it. 2011-08-23.23:48:35 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base 2011-08-23.23:48:35 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-23.23:48:35 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/codesourcery 2011-08-23.23:48:35 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/embedded 2011-08-23.23:48:36 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/flightgear 2011-08-23.23:48:36 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/forth 2011-08-23.23:48:36 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/functional 2011-08-23.23:48:36 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/linux 2011-08-23.23:48:37 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/minix 2011-08-23.23:48:37 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/modula_n_oberon 2011-08-23.23:48:37 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/trimslice 2011-08-23.23:48:37 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/ubuntu 2011-08-23.23:48:37 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/unclassified_2 2011-08-23.23:48:42 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/x-plane ##### Things broken already. got the first "incorrect key" messages - running the shell script again. ##### Don''t know why I don''t see the ''slice_2'' by itself in the previous group 2011-08-23.23:50:04 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2 2011-08-23.23:50:04 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base 2011-08-23.23:50:04 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-23.23:50:04 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/codesourcery 2011-08-23.23:50:04 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/embedded 2011-08-23.23:50:05 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/flightgear 2011-08-23.23:50:05 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/forth 2011-08-23.23:50:05 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/functional 2011-08-23.23:50:05 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/linux 2011-08-23.23:50:05 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/minix 2011-08-23.23:50:06 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/modula_n_oberon 2011-08-23.23:50:06 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/trimslice 2011-08-23.23:50:06 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/ubuntu 2011-08-23.23:50:06 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/unclassified_2 2011-08-23.23:50:11 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/x-plane #### slice_2/base can be mounted, all other filesystems cannot. #### the following commands are futile attempts to get things back in working order, #### all respond with "cannot load key... incorrect key" #### (except, again, slice_2/base ) 2011-08-23.23:50:51 zfs mount slice_2/base 2011-08-23.23:52:31 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2 2011-08-23.23:52:31 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base 2011-08-23.23:52:31 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-23.23:52:32 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/codesourcery 2011-08-23.23:52:32 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/embedded 2011-08-23.23:52:32 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/flightgear 2011-08-23.23:52:32 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/forth 2011-08-23.23:52:32 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/functional 2011-08-23.23:52:32 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/linux 2011-08-23.23:52:33 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/minix 2011-08-23.23:52:33 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/modula_n_oberon 2011-08-23.23:52:33 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/trimslice 2011-08-23.23:52:33 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/ubuntu 2011-08-23.23:52:33 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/unclassified_2 2011-08-23.23:52:38 zfs set keysource=passphrase,file:///root/passphrases/slice_2_passphrase slice_2/base/x-plane 2011-08-24.00:05:00 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2 2011-08-24.00:05:00 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base 2011-08-24.00:05:00 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-24.00:05:00 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/codesourcery 2011-08-24.00:05:00 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/embedded 2011-08-24.00:05:00 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/flightgear 2011-08-24.00:05:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/forth 2011-08-24.00:05:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/functional 2011-08-24.00:05:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/linux 2011-08-24.00:05:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/minix 2011-08-24.00:05:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/modula_n_oberon 2011-08-24.00:05:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/trimslice 2011-08-24.00:05:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/ubuntu 2011-08-24.00:05:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/unclassified_2 2011-08-24.00:05:07 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/x-plane 2011-08-24.22:31:49 zfs mount slice_2/base 2011-08-24.22:41:47 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-24.22:42:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_1_passphrase slice_2/base/bitsavers 2011-08-24.22:42:14 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_0_passphrase slice_2/base/bitsavers 2011-08-24.22:42:23 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-24.22:44:02 zfs set keysource=passphrase,prompt slice_2/base/bitsavers 2011-08-24.22:49:01 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2 2011-08-24.22:49:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base 2011-08-24.22:49:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-24.22:49:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/codesourcery 2011-08-24.22:49:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/embedded 2011-08-24.22:49:02 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/flightgear 2011-08-24.22:49:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/forth 2011-08-24.22:49:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/functional 2011-08-24.22:49:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/linux 2011-08-24.22:49:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/minix 2011-08-24.22:49:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/modula_n_oberon 2011-08-24.22:49:03 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/trimslice 2011-08-24.22:49:04 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/ubuntu 2011-08-24.22:49:04 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/unclassified_2 2011-08-24.22:49:09 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/x-plane 2011-08-24.22:49:53 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2 2011-08-24.22:49:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base 2011-08-24.22:49:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-24.22:49:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/codesourcery 2011-08-24.22:49:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/embedded 2011-08-24.22:49:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/flightgear 2011-08-24.22:49:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/forth 2011-08-24.22:49:55 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/functional 2011-08-24.22:49:55 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/linux 2011-08-24.22:49:55 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/minix 2011-08-24.22:49:55 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/modula_n_oberon 2011-08-24.22:49:55 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/trimslice 2011-08-24.22:49:55 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/ubuntu 2011-08-24.22:49:56 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/unclassified_2 2011-08-24.22:50:01 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/x-plane 2011-08-24.22:51:28 zfs set mountpoint=/slices/slice_2 slice_2 2011-08-24.22:52:30 zfs set mountpoint=/slices/slice_2 slice_2 2011-08-24.22:52:42 zfs set mountpoint=/slices/slice_2/base slice_2/base 2011-08-24.22:53:44 zfs set mountpoint=/global/bitsavers/ slice_2/base/bitsavers 2011-08-24.22:54:54 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/bitsavers 2011-08-24.22:55:02 zfs set mountpoint=/global/bitsavers/ slice_2/base/bitsavers 2011-09-05.21:41:36 zfs set keysource=passphrase,prompt slice_2/base/trimslice 2011-09-05.21:42:17 zfs set mountpoint=/global/trimslice/ slice_2/base/trimslice 2011-09-05.21:42:33 zfs set mountpoint=/global/trimslice/ slice_2/base/trimslice 2011-09-05.21:42:40 zfs set mountpoint=/global/trimslice/ slice_2/base/trimslice 2011-09-05.21:43:18 zfs set mountpoint=/global/trimslice/ slice_2/base/trimslice 2011-09-05.21:44:16 zfs set keysource=passphrase,file:///home/trouser/passphrases/slice_2_passphrase slice_2/base/trimslice 2012-02-05.22:09:06 zpool export slice_2 2012-02-20.20:01:28 zpool import slice_2 2012-02-20.20:35:11 zfs set keysource=passphrase,file:///export/home/trouser/passphrases/slice_2_passphrase slice_2/base/trimslice 2012-02-20.20:42:41 zfs set keysource=passphrase,prompt slice_2/base/bitsavers ################################# slice_2 status I just noticed the following: ################################# root at turbofan:~# root at turbofan:~# zpool status slice_2 pool: slice_2 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM slice_2 ONLINE 20 0 0 raidz3-0 ONLINE 0 0 0 c7t0d0s2 ONLINE 0 0 0 c7t1d0s2 ONLINE 0 0 0 c7t2d0s2 ONLINE 0 0 0 c7t3d0s2 ONLINE 0 0 0 c7t4d0s2 ONLINE 0 0 0 c7t5d0s2 ONLINE 0 0 0 c7t6d0s2 ONLINE 0 0 0 c7t7d0s2 ONLINE 0 0 0 errors: 11 data errors, use ''-v'' for a list root at turbofan:~# root at turbofan:~# zpool status -v slice_2 pool: slice_2 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM slice_2 ONLINE 20 0 0 raidz3-0 ONLINE 0 0 0 c7t0d0s2 ONLINE 0 0 0 c7t1d0s2 ONLINE 0 0 0 c7t2d0s2 ONLINE 0 0 0 c7t3d0s2 ONLINE 0 0 0 c7t4d0s2 ONLINE 0 0 0 c7t5d0s2 ONLINE 0 0 0 c7t6d0s2 ONLINE 0 0 0 c7t7d0s2 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: slice_2/base/trimslice:<0x1> slice_2/base:<0x1> slice_2/base/codesourcery:<0x1> slice_2/base/x-plane:<0x1> slice_2/base/modula_n_oberon:<0x1> slice_2/base/minix:<0x1> slice_2/base/forth:<0x1> slice_2/base/functional:<0x1> slice_2/base/bitsavers:<0x1> slice_2/base/flightgear:<0x1> slice_2/base/embedded:<0x1> root at turbofan:~# #################################
On 02/22/12 06:10, Roberto Waltman wrote:> 2011-08-23.23:48:35 zfs set > keysource=passphrase,file:///root/passphrases/slice_2_passphrase > slice_2/base/bitsaversThat should have failed because the keysource property is inherited from slice_2/base. So you have found a bug and I can reproduce it. The reason that should have failed is the source of where the keysource comes from is used to determine which dataset to look at for the hidden salt property. We know what that salt property should actually be in your case because it is set on slice_2/base. Unfortunately ''zfs set salt'' won''t work because salt is read-only from userland (so it doesn''t accidentally get overridden and cause the very same symptoms you have!). In theory you would assume that you could go back to having the keysource inherited by running: ''zfs inherit keysource slice_2/base/bitsavers'' However that won''t work because of a protection we have in place to again avoid yet another route into these same symptoms. It will fail with an error message something like this: cannot inherit keysource for ''slice_2/base/bitsavers'': use ''zfs key -c -o keysource=...'' Using a hacked up libzfs that removes the check that ''zfs inherit'' does so I can get out of the situation and make the datasets accessible again. So this is fixable so don''t abandon hope yet. -- Darren J Moffat
Seemingly Similar Threads
- [Bug 971] New: zfs key -l fails after unloading (keyscope=dataset)
- ZFS Crypto [Prelim] Codereview
- [Bug 759] New: ''zpool create -o keysource=,'' hanged
- [Bug 752] New: zfs set keysource no longer works on existing pools
- [Bug 1053] New: ''zfs create'' core dumped with keysource=hex, prompt and unmatched entered in