zfs code - Mar 2008 - dsl_dataset_t pointer during ''zfs create'' changes

I''ve noticed that the dsl_dataset_t that points to a given dataset 
changes during the life time of a ''zfs create'' command.  We
start out
with one dsl_dataset_t* during dmu_objset_create_sync() but by the time 
we are later mounting the dataset we have a different in memory 
dsl_dataset_t* referring to the same dataset.

This causes me a big issue with per dataset provided encryption keys, 
but not with a per pool provided encryption key.

When keyscope=dataset we pass the key value used as the wrapping key 
down over the ioctl as part of the dataset creation.  This makes its way 
via dmu_objset_create_sync() to dsl_crypto_key_gen() which generates the 
actual encryption key and wraps it using the key that came over the 
ioctl, it then stores the wrapped key on disk using zap_update.  All 
this part works just fine.  The last thing that dsl_crypto_key_gen() 
does is place the encryption key into ds>ds_key so it can be used later. 
   When keyscope=pool the only difference with the above is that the 
wrapping key is already present in spa->spa_key.

The problem occurs when the userland zfs_create() attempts to mount the 
newly created dataset.  The dsl_dataset_t* that was valid during the 
creation isn''t available anymore and since there is no key we
can''t
mount the dataset.

Have I done something fundamentally wrong by hanging the crypto key off 
the dsl_dataset_t ?  Is there a better choice that is always going to be 
in memory for a given dataset through the whole ''zfs create''
operation ?

-- 
Darren J Moffat

Darren J Moffat wrote:
> I''ve noticed that the dsl_dataset_t that points to a given dataset
> changes during the life time of a ''zfs create'' command. 
We start out
> with one dsl_dataset_t* during dmu_objset_create_sync() but by the time 
> we are later mounting the dataset we have a different in memory 
> dsl_dataset_t* referring to the same dataset.
Yes, the cached copy can come and go, but the underlying dsl_dataset_phys_t
(the dsobj) should be persistent.
> 
> This causes me a big issue with per dataset provided encryption keys, 
> but not with a per pool provided encryption key.
> 
> When keyscope=dataset we pass the key value used as the wrapping key 
> down over the ioctl as part of the dataset creation.  This makes its way 
> via dmu_objset_create_sync() to dsl_crypto_key_gen() which generates the 
> actual encryption key and wraps it using the key that came over the 
> ioctl, it then stores the wrapped key on disk using zap_update.  All 
> this part works just fine.  The last thing that dsl_crypto_key_gen() 
> does is place the encryption key into ds>ds_key so it can be used later.
>    When keyscope=pool the only difference with the above is that the 
> wrapping key is already present in spa->spa_key.
> 
> The problem occurs when the userland zfs_create() attempts to mount the 
> newly created dataset.  The dsl_dataset_t* that was valid during the 
> creation isn''t available anymore and since there is no key we
can''t
> mount the dataset.
This sounds similar to what I had to do for the refquota and refreservation
properties.  You might need to look at pulling in the ds_key zap attribute
during dsl_dataset_open_obj().  Take a look at the code for setting
ds_reserved and ds_quota.

-Chris

Chris Kirby wrote:
> Darren J Moffat wrote:
>> I''ve noticed that the dsl_dataset_t that points to a given
dataset
>> changes during the life time of a ''zfs create''
command.  We start out
>> with one dsl_dataset_t* during dmu_objset_create_sync() but by the 
>> time we are later mounting the dataset we have a different in memory 
>> dsl_dataset_t* referring to the same dataset.
> 
> Yes, the cached copy can come and go, but the underlying dsl_dataset_phys_t
> (the dsobj) should be persistent.
But the unwrapped key isn''t in the dsl_dataset_phys_t - by design 
otherwise the whole crypto is pointless since the key is on disk.

The wrapped key is in the zap and if keyscope=pool the I can unwrap it 
because the unwrapping key is in the spa.  If keyscope=dataset my 
unwrapping key is ?????? - this is the problem case.
> This sounds similar to what I had to do for the refquota and refreservation
> properties.  You might need to look at pulling in the ds_key zap attribute
> during dsl_dataset_open_obj().  Take a look at the code for setting
> ds_reserved and ds_quota.
I see the use of dsl_prop_get_ds_locked() for those properties during a 
dsl_dataset_open_obj().

What I''m looking for is something that is always (during the whole 
create until it is umounted) present in memory where I can place the 
unwrapped key.  Maybe that doesn''t exist and I need to rethink the way 
per dataset keying is done.

-- 
Darren J Moffat

Darren J Moffat wrote:
> Chris Kirby wrote:
>> Darren J Moffat wrote:
>>> I''ve noticed that the dsl_dataset_t that points to a given
dataset
>>> changes during the life time of a ''zfs create''
command.  We start out
>>> with one dsl_dataset_t* during dmu_objset_create_sync() but by the 
>>> time we are later mounting the dataset we have a different in
memory
>>> dsl_dataset_t* referring to the same dataset.
>>
>> Yes, the cached copy can come and go, but the underlying 
>> dsl_dataset_phys_t
>> (the dsobj) should be persistent.
> 
> But the unwrapped key isn''t in the dsl_dataset_phys_t - by design 
> otherwise the whole crypto is pointless since the key is on disk.
> 
> The wrapped key is in the zap and if keyscope=pool the I can unwrap it 
> because the unwrapping key is in the spa.  If keyscope=dataset my 
> unwrapping key is ?????? - this is the problem case.
Ah, OK, I didn''t catch the distinction between the keys.

So is it possible for a snapshot to have a different unwrapping key
than the head dataset?  I would think not, since the blocks in a snapshot
were created as part of the head ds.

I''m wondering whether the dsl_dir_t is the more logical place to keep
the unwrapping key.  Although that doesn''t solve this issue, since the
dsl_dir_t structs come and go from memory too.

Chris Kirby wrote:
> Ah, OK, I didn''t catch the distinction between the keys.
> 
> So is it possible for a snapshot to have a different unwrapping key
> than the head dataset?  I would think not, since the blocks in a snapshot
> were created as part of the head ds.
A snapshot always has the same unwrapping key and same actual encryption 
key as the head dataset.   A clone will have the same actual encryption 
key (and currently has the same wrapped key but I''m looking at changing
that).
> I''m wondering whether the dsl_dir_t is the more logical place to
keep
> the unwrapping key.  Although that doesn''t solve this issue, since
the
> dsl_dir_t structs come and go from memory too.
Indeed - I looked at that as a possible place and objset_t as well.

One possibility is that I create a hash table based on objset id''s that
is hung off the spa.  The advantage this has is that it is the objset id 
I have when I need to look the key up (a zbookmark_t really) - which is 
the performance critical part.

-- 
Darren J Moffat