zfs discuss - Jul 2006 - Enabling compression/encryption on a populated filesystem

When ZFS compression is enabled, although the man page doesn''t
explicitly say this, my guess is that only new data that gets
written out is compressed - in keeping with the COW policy.

This is all well and good, if you enable compression when you
create the ZFS filesystem.  If I enable compression at some
later point, I may have decided that all the data for that
filesystem is to be compressed (or uncompressed.)  But what I
can''t see, currently, is a mechanism to go and compress/uncompress
all of my old data.

The only obvious solution I can see, currently, is to "zfs send"
it all out to some other place and "zfs receive" it back with
compression enabled, with a zfs destroy/create in the middle.

Am I missing something?

Or is there room for a "zfs compress" command here?

Hmmm, well, I suppose the same problem might apply to
encrypting data too...so maybe what I need is a zfs command
that will walk the filesystem''s data tree, read in data and
write it back out according to the current data policy.

This may not work so well for encrypted data if encryption
is disabled, but I''m not sure that is such a good idea.

Darren

On 7/13/06, Darren Reed <Darren.Reed at sun.com>
wrote:
> When ZFS compression is enabled, although the man page doesn''t
> explicitly say this, my guess is that only new data that gets
> written out is compressed - in keeping with the COW policy.
>
> This is all well and good, if you enable compression when you
> create the ZFS filesystem.  If I enable compression at some
> later point, I may have decided that all the data for that
> filesystem is to be compressed (or uncompressed.)  But what I
> can''t see, currently, is a mechanism to go and compress/uncompress
> all of my old data.
         [snip]
> Hmmm, well, I suppose the same problem might apply to
> encrypting data too...so maybe what I need is a zfs command
> that will walk the filesystem''s data tree, read in data and
> write it back out according to the current data policy.
Such a ''rewrite'' command would also solve the expanding RAID-Z
problem
discussed here (assuming small stripe reads are supported):
http://www.opensolaris.org/jive/message.jspa?messageID=47053#47053

Steve

On 7/13/06, Darren Reed <Darren.Reed at sun.com>
wrote:
> When ZFS compression is enabled, although the man page doesn''t
> explicitly say this, my guess is that only new data that gets
> written out is compressed - in keeping with the COW policy.
>
[ ... ]
>
> Hmmm, well, I suppose the same problem might apply to
> encrypting data too...so maybe what I need is a zfs command
> that will walk the filesystem''s data tree, read in data and
> write it back out according to the current data policy.
>
It seems this could be made a function of ''zfs scrub'' --
instead of
simply verifying the data, it could rewrite the data as it goes.

This comes in handy in other situations.  For example, with the
current state of things, if you add disks to a pool that contains
mostly static data, you don''t get the benefit of the additional
spindles when reading old data.  Rewriting the data would gain you
that benefit, plus it would avoid the new disks becoming the hot spot
for all new writes (assuming the old disks were very full.)

Theoretically this could also be useful in a live data migration
situation, where you have both new and old storage connected to a
server.  But this assumes there would be some way to tell ZFS to treat
a subset of disks as read-only.

Chad Mynhier
http://cmynhier.blogspot.com/

Darren Reed wrote:
> Hmmm, well, I suppose the same problem might apply to
> encrypting data too...so maybe what I need is a zfs command
> that will walk the filesystem''s data tree, read in data and
> write it back out according to the current data policy.
And if that file system is multiple terrabytes would you be okay with 
there being a read and write lock while this runs ?
> This may not work so well for encrypted data if encryption
> is disabled, but I''m not sure that is such a good idea.
The current plan is that encryption must be turned on when the file 
system is created and can''t be turned on later.  This means that the 
zfs-crypto work depends on the RFE to set properties at file system 
creation time.

You also won''t be able to turn crypto off for a given filesystem later 
(because you won''t know when all the data is back in the clear again
and
you can safely destroy the key).

-- 
Darren J Moffat

Darren J Moffat wrote:
> Darren Reed wrote:
>
>> Hmmm, well, I suppose the same problem might apply to
>> encrypting data too...so maybe what I need is a zfs command
>> that will walk the filesystem''s data tree, read in data and
>> write it back out according to the current data policy.
>
>
> And if that file system is multiple terrabytes would you be okay with 
> there being a read and write lock while this runs ?

I thought about this and my answer might surprise you:

Yes - so long as it is interruptible.

Granted it could take a very long time to complete, but won''t it be 
quicker than doing anything else to achieve the same result?

i.e. how long would it take to backup those multiple terrabytes to tape 
and then restore them?
Or even to transfer them to a new filesystem?

But...couldn''t you also snapshot/clone a (large) filesystem, turn on 
encryption, start encypting those 10TB of data (the read/write lock 
won''t impact people using the parent) and then use a combination of zfs
send/receieve and swapping snapshots/clones around with the parent to 
achieve an in-place encryption without locking any "live" filesystems?

I suppose this sounds complex, but I don''t think any live migration to 
using encryption (while aiming for 0 down time) will be easy.
>> This may not work so well for encrypted data if encryption
>> is disabled, but I''m not sure that is such a good idea.
>
>
> The current plan is that encryption must be turned on when the file 
> system is created and can''t be turned on later.  This means that
the
> zfs-crypto work depends on the RFE to set properties at file system 
> creation time.
>
> You also won''t be able to turn crypto off for a given filesystem
later
> (because you won''t know when all the data is back in the clear
again
> and you can safely destroy the key).

Ok, I can understand that, both from a security perspective and a "lets 
not let them shoot themselves in the foot".

Darren

The problem with encrypting in place is knowing when the system is 
"safe", ie when it is done.

There are lots of things in this area that are possible but they 
certainly aren''t in the first deliveries for this project and with all 
due respect no amount of pushing by anyone will change my mind on that. 
    I want to first get basic crypto with simple key management working 
and get that out, then look at more complex key management systems, once 
we have done that then we can come back and look at "in place encryption 
enabling".

Thanks for thinking about this and sharing though.

--
Darren J Moffat

Darren J Moffat wrote:
> The problem with encrypting in place is knowing when the system is 
> "safe", ie when it is done.
>
> There are lots of things in this area that are possible but they 
> certainly aren''t in the first deliveries for this project and with
all
> due respect no amount of pushing by anyone will change my mind on 
> that.     I want to first get basic crypto with simple key management 
> working and get that out, then look at more complex key management 
> systems, once we have done that then we can come back and look at "in 
> place encryption enabling".

Sorry, I didn''t mean to imply that the zfs-crypto project should
try and solve this, just that the idea of of compressing or
decompressing an entire filesystem, after changing that attribute,
could possibly be applied to encrypting data on a disk.

Darren

On Fri, 2006-07-14 at 07:03, Darren J Moffat wrote:
> The current plan is that encryption must be turned on when the file 
> system is created and can''t be turned on later.  This means that
the
> zfs-crypto work depends on the RFE to set properties at file system 
> creation time.
> 
> You also won''t be able to turn crypto off for a given filesystem
later
> (because you won''t know when all the data is back in the clear
again and
> you can safely destroy the key).
So, I''d think that, in the fullness of time, you''d want some
sort of
mechanism for graceful key roll-over -- i.e., you''d set a new key,
migrate existing data encrypted using the old key to the new key, then
forget the old key; the whole point of keyed cryptography is that the
key is kept both small (so it can more easily remain secret) AND
changeable.

Perhaps this needs additional infrastructure from ZFS to move/reallocate
already-allocated blocks, but I''d think that infrastructure would also
be useful for the "pool shrink" RFE as well...

						- Bill

On Fri, 2006-07-14 at 07:03, Darren J Moffat wrote:
> The current plan is that encryption must be turned on when the file 
> system is created and can''t be turned on later.  This means that
the
> zfs-crypto work depends on the RFE to set properties at file system 
> creation time.
> 
> You also won''t be able to turn crypto off for a given filesystem
later
> (because you won''t know when all the data is back in the clear
again and
> you can safely destroy the key).
So, I''d think that, in the fullness of time, you''d want some
sort of
mechanism for graceful key roll-over -- i.e., you''d set a new key,
migrate existing data encrypted using the old key to the new key, then
forget the old key; the whole point of keyed cryptography is that the
key is kept both small (so it can more easily remain secret) AND
changeable.

Perhaps this needs additional infrastructure from ZFS to move/reallocate
already-allocated blocks, but I''d think that infrastructure would also
be useful for the "pool shrink" RFE as well...

						- Bill

Darren J Moffat wrote:
> Darren Reed wrote:
> 
>> Hmmm, well, I suppose the same problem might apply to
>> encrypting data too...so maybe what I need is a zfs command
>> that will walk the filesystem''s data tree, read in data and
>> write it back out according to the current data policy.
> 
> And if that file system is multiple terrabytes would you be okay with 
> there being a read and write lock while this runs ?
>
I am only guessing, but when encryption is "important enough" the
answer is "yes."
But the next question is then "is this situation common enough to justify
the effort."
> The current plan is that encryption must be turned on when the file 
> system is created and can''t be turned on later.  This means that
the
> zfs-crypto work depends on the RFE to set properties at file system 
> creation time.
> 
> You also won''t be able to turn crypto off for a given filesystem
later
> (because you won''t know when all the data is back in the clear
again and
> you can safely destroy the key).
Why?  Is the ''data is encrypted'' flag only stored in
filesystem metadata, or is
that flag stored in each data block?  If the latter is true, it would be
possible
(though potentially time-consuming) to determine which files are encrypted, and 
which are not.



--------------------------------------------------------------------------
Jeff VICTOR              Sun Microsystems            jeff.victor @ sun.com
OS Ambassador            Sr. Technical Specialist
Solaris 10 Zones FAQ:    http://www.opensolaris.org/os/community/zones/faq
--------------------------------------------------------------------------

Jeff Victor wrote:
> Why?  Is the ''data is encrypted'' flag only stored in
filesystem
> metadata, or is that flag stored in each data block?  
Like compression and which checksum algorithm it will be stored in
every dmu object.
> If the latter is 
> true, it would be possible (though potentially time-consuming) to 
> determine which files are encrypted, and which are not.
Exactly, time consuming and expensive.

Buth the reason thing is how do you tell the admin "its done now the 
filesystem is safe".   With compression you don''t generally care
if some
old stuff didn''t compress (and with the current implementation it has
to
compress a certain amount or it gets written uncompressed anyway).  With 
encryption the human admin really needs to be told.

So far there really isn''t any time consuming or offline action for ZFS 
data sets that actually impacts your data so I''d rather not introduce 
one now (I wouldn''t really put scrub in this category).


-- 
Darren J Moffat

Bill Sommerfeld wrote:
> On Fri, 2006-07-14 at 07:03, Darren J Moffat wrote:
>> The current plan is that encryption must be turned on when the file 
>> system is created and can''t be turned on later.  This means
that the
>> zfs-crypto work depends on the RFE to set properties at file system 
>> creation time.
>>
>> You also won''t be able to turn crypto off for a given
filesystem later
>> (because you won''t know when all the data is back in the clear
again and
>> you can safely destroy the key).
> 
> So, I''d think that, in the fullness of time, you''d want
some sort of
> mechanism for graceful key roll-over -- i.e., you''d set a new key,
> migrate existing data encrypted using the old key to the new key, then
> forget the old key; the whole point of keyed cryptography is that the
> key is kept both small (so it can more easily remain secret) AND
> changeable.
One way, and the initial way we will deal with this, is to have the key 
change be done on the "master" wrapping key not on the actual per data
set encryption keys.

One of the goals of the ZFS crypto project is to support multiple 
different key management strategies with the same on disk capabilities.

Key roll over is one the agenda for a later phase, as is key expiry 
(manual and time based).

-- 
Darren J Moffat

Joseph Mocker wrote:
> Jeff Victor wrote:
>
>>
>>>
>>> And if that file system is multiple terrabytes would you be okay 
>>> with there being a read and write lock while this runs ?
>>>
>>
>> I am only guessing, but when encryption is "important enough"
the
>> answer is "yes." But the next question is then "is this
situation
>> common enough to justify the effort."
>
> If it is "important enough" wouldn''t you want the data
to start out
> encrypted ?
>
> :-)

Requirements for the security of data storage can change over time.
Both for increasing it and decreasing it.
> On the other hand, if you are migrating your data from a plain 
> datastore  to an encrypted datastore, how much do you trust the plain 
> datastore in the first place?

It''s often not about trusting the unencrypted data but preventing
access
to it - the classic example of this being important is the "stolen
laptop".

The upgrading of Solaris on a laptop (bfu, etc) is a good case in point 
for wanting to be able to migrate a data set from unencrypted to 
encrypted, either all at once or progressively:
- ZFS pool that is strictly data
- upgrade/install the OS
- import the ZFS pool into a ZFS that supports encryption

now how do I get my data encrypted?

Darren

Jeff Victor wrote:
>
>>
>> And if that file system is multiple terrabytes would you be okay with 
>> there being a read and write lock while this runs ?
>>
>
> I am only guessing, but when encryption is "important enough" the
> answer is "yes." But the next question is then "is this
situation
> common enough to justify the effort."
If it is "important enough" wouldn''t you want the data to
start out
encrypted ?

:-)

Ok, so that obviously neglects migration of existing data for "legacy"
apps so that may not be a fair question.

On the other hand, if you are migrating your data from a plain 
datastore  to an encrypted datastore, how much do you trust the plain 
datastore in the first place? If you are interested in migration to an 
encrypted datastore you probably don''t trust the plain datastore all 
that much. In which case you probably should do some 
cleansing/verification on the data during the migration, so your 
original response is probably appropriate.

  --joe

Darren J Moffat wrote:
> Buth the reason thing is how do you tell the admin "its done now the 
> filesystem is safe".   With compression you don''t generally
care if
> some old stuff didn''t compress (and with the current
implementation it
> has to compress a certain amount or it gets written uncompressed 
> anyway).  With encryption the human admin really needs to be told. 
As a sysadmin, I''d be happy with another scrub-type command.  Something
with the following meaning:

    "Reapply all block-level properties such as compression, encryption,
    and checksum to every block in the volume.  Have the admin come back
    tomorrow and run ''zpool status'' too see if it''s
zone."

Mad props if I can do this on a live filesystem (like the other ZFS 
commands, which also get mad props for being good tools).

A natural command for this would be something like "zfs blockscrub 
tank/volume".  Also, "zpool blockscrub tank" would make sense to
me as
well, even though it might touch more data.

Of course, it''s easy for me to just say this, since I''m not
thinking
about the implementation very deeply...

-Luke

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060717/e0aa97b1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3271 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060717/e0aa97b1/attachment.bin>

Hi,

there might be value in a "zpool scrub -r (as in "re-write
blocks") other than
the prior discussion on encryption and compression.

For instance, a bit that is just about to rot might not be detected with a
regular zfs scrub but it would be rewritten with a re-writing scrub.

It would also exercise the writing "muscles" on disks that
don''t see a lot of
writing, such as archives or system disks, thereby detecting any degradation
that affects writing of data.

Of course the re-writing must be 100% safe, but that can be done with COW
quite easily.

Then, admins would for instance run a "zpool scrub" every week and
maybe a
"zpool scrub -r" every month or so.

Just my 2 cents,
  Constantin


Luke Scharf wrote:
> Darren J Moffat wrote:
>> Buth the reason thing is how do you tell the admin "its done now
the
>> filesystem is safe".   With compression you don''t
generally care if
>> some old stuff didn''t compress (and with the current
implementation it
>> has to compress a certain amount or it gets written uncompressed
>> anyway).  With encryption the human admin really needs to be told. 
> As a sysadmin, I''d be happy with another scrub-type command. 
Something
> with the following meaning:
> 
>     "Reapply all block-level properties such as compression,
encryption,
>     and checksum to every block in the volume.  Have the admin come back
>     tomorrow and run ''zpool status'' too see if
it''s zone."
> 
> Mad props if I can do this on a live filesystem (like the other ZFS
> commands, which also get mad props for being good tools).
> 
> A natural command for this would be something like "zfs blockscrub
> tank/volume".  Also, "zpool blockscrub tank" would make
sense to me as
> well, even though it might touch more data.
> 
> Of course, it''s easy for me to just say this, since I''m
not thinking
> about the implementation very deeply...
> 
> -Luke
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
Constantin Gonzalez                            Sun Microsystems GmbH, Germany
Platform Technology Group, Client Solutions                http://www.sun.de/
Tel.: +49 89/4 60 08-25 91                   http://blogs.sun.com/constantin/

> Of course the re-writing must be 100% safe, but that can be done with COW
> quite easily.
Almost.  The hard part is snapshots.  If you modify a data block,
you must also modify every L1 indirect block in every snapshot
that points to it, and every L2 above each L1, all the way up
to the uberblock.  This is certainly possible, but it''s tricky.
But I think we''ll eventually get around to doing it because
the ability to remap blocks would be *so* useful -- it would
enable compression of preexisting data, removing devices from
a pool, automatically moving cold data to slower devices, etc.

Jeff

On Tue, Jul 18, 2006 at 01:27:21AM -0700, Jeff Bonwick
wrote:
> > Of course the re-writing must be 100% safe, but that can be done with
COW
> > quite easily.
> 
> the ability to remap blocks would be *so* useful -- it would
> enable compression of preexisting data, removing devices from
> a pool, automatically moving cold data to slower devices, etc.
Being able to remove devices from a pool would be a good thing.  I
can''t
personally think of any reason that I would ever do it, but a friend of
mine keeps asking me why it can''t do it and that it should be able to.

-brian

On 7/18/06, Brian Hechinger <wonko at 4amlunch.net>
wrote:
> On Tue, Jul 18, 2006 at 01:27:21AM -0700, Jeff Bonwick wrote:
> >
> > the ability to remap blocks would be *so* useful -- it would
> > enable compression of preexisting data, removing devices from
> > a pool, automatically moving cold data to slower devices, etc.
>
> Being able to remove devices from a pool would be a good thing.  I
can''t
> personally think of any reason that I would ever do it, but a friend of
> mine keeps asking me why it can''t do it and that it should be able
to.
>
> -brian
This situation is implicitly included in what Jeff said, but live data
migration is a good example of where this would come in handy.  When
it comes time to replace your big storage array with a newer model (or
your 250GB drive with a 750GB drive), you could simply add the new
storage to an existing pool and then remove the old storage from the
pool.  You need to wait for all of the data to be migrated, but in the
meantime the data is available to users.  You''d be able to do this
with no significant downtime.  And if the data migration happens at a
suitably low priority, there''s not even a performance hit.

Chad Mynhier
http://cmynhier.blogspot.com/

On Tue, Jul 18, 2006 at 09:46:44AM -0400, Chad Mynhier
wrote:
> On 7/18/06, Brian Hechinger <wonko at 4amlunch.net> wrote:
> >On Tue, Jul 18, 2006 at 01:27:21AM -0700, Jeff Bonwick wrote:
> >>
> >> the ability to remap blocks would be *so* useful -- it would
> >> enable compression of preexisting data, removing devices from
> >> a pool, automatically moving cold data to slower devices, etc.
> >
> >Being able to remove devices from a pool would be a good thing.  I
can''t
> >personally think of any reason that I would ever do it, but a friend of
> >mine keeps asking me why it can''t do it and that it should be
able to.
> >
> >-brian
> 
> This situation is implicitly included in what Jeff said, but live data
> migration is a good example of where this would come in handy.  When
> it comes time to replace your big storage array with a newer model (or
> your 250GB drive with a 750GB drive), you could simply add the new
> storage to an existing pool and then remove the old storage from the
> pool.  You need to wait for all of the data to be migrated, but in the
> meantime the data is available to users.  You''d be able to do this
> with no significant downtime.  And if the data migration happens at a
> suitably low priority, there''s not even a performance hit.
Size upgrades you can do in place, and even migrating to a new shelf you
can do in place as well (replace individual disk in old shelf with
individual disk in new chelf).

The only place the removing disks from the pool would be useful in this
scenario would be if the new array had a fewer number of larger disks.

-brian

On 7/18/06, Brian Hechinger <wonko at 4amlunch.net>
wrote:
> On Tue, Jul 18, 2006 at 09:46:44AM -0400, Chad Mynhier wrote:
> > On 7/18/06, Brian Hechinger <wonko at 4amlunch.net> wrote:
> > >
> > >Being able to remove devices from a pool would be a good thing.  I
can''t
> > >personally think of any reason that I would ever do it, but a
friend of
> > >mine keeps asking me why it can''t do it and that it
should be able to.
> > >
> > >-brian
> >
> > This situation is implicitly included in what Jeff said, but live data
> > migration is a good example of where this would come in handy.
>
> Size upgrades you can do in place, and even migrating to a new shelf you
> can do in place as well (replace individual disk in old shelf with
> individual disk in new chelf).
>
> The only place the removing disks from the pool would be useful in this
> scenario would be if the new array had a fewer number of larger disks.
There are conceivable situations in which you''re not able to do a
simple one-to-one device replacement.  One case is the one you give,
where you have an array with fewer, larger disks.  But it''s also
feasible that the zpool structure you want to use on the new storage
doesn''t match what you''re doing on your current storage.

(Although I guess the fewer, larger disk scenario is just a special
case of the situation in which the resultant zpool structure doesn''t
match the original.)

Chad Mynhier
http://cmynhier.blogspot.com/

> On Tue, Jul 18, 2006 at 01:27:21AM -0700, Jeff Bonwick wrote:
> > > Of course the re-writing must be 100% safe, but that can be done
with COW
> > > quite easily.
> > 
> > the ability to remap blocks would be *so* useful -- it would
> > enable compression of preexisting data, removing devices from
> > a pool, automatically moving cold data to slower devices, etc.
> 
> Being able to remove devices from a pool would be a good thing.  I
can''t
> personally think of any reason that I would ever do it, but a friend of
> mine keeps asking me why it can''t do it and that it should be able
to.
Recovery from mistakes.  If you''re perfect, you''ll never need
it.  :-)
I''ve certainly seen messages from people that accidentally added their
last disk into a netapp volume.  Now they have no spares and they can''t
get it back without destroying the volume or adding (significant)
hardware.

Today, the fact that mirrors auto-resize without asking leads to some
similar issues if the components are not the same size.  Being able to
shrink it down would allow recovery in that case.

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

On Sun, 2006-07-16 at 15:21, Darren J Moffat wrote:
> Jeff Victor wrote:
> > Why?  Is the ''data is encrypted'' flag only stored in
filesystem
> > metadata, or is that flag stored in each data block?  
> 
> Like compression and which checksum algorithm it will be stored in
> every dmu object.
I thought the compression alg was stored in the block pointer.  IMHO you
need at least a few bits of key version number in the block pointer to
do graceful key roll-over (you don''t need an alg id in the bp since
that
can be stored with the key..)

						- Bill

>On Sun, 2006-07-16 at 15:21, Darren J Moffat wrote:
>> Jeff Victor wrote:
>> > Why?  Is the ''data is encrypted'' flag only
stored in filesystem
>> > metadata, or is that flag stored in each data block?  
>> 
>> Like compression and which checksum algorithm it will be stored in
>> every dmu object.
>
>I thought the compression alg was stored in the block pointer.  IMHO you
>need at least a few bits of key version number in the block pointer to
>do graceful key roll-over (you don''t need an alg id in the bp since
that
>can be stored with the key..)
So how many of the 128 bits of the blockpointer are used for things
other than to point where the block is?

Casper

On Wed, Jul 19, 2006 at 03:10:00AM +0200, Casper.Dik at sun.com
wrote:
> So how many of the 128 bits of the blockpointer are used for things
> other than to point where the block is?
128 *bits*?  What filesystem have you been using?  :)  We''ve got
luxury-class block pointers that are 128 *bytes*.  We get away with it
because we also have big data blocks to go with them.  In your typical
mixed-data storage pool, about 1-2% total (including all the block
pointers) is ZFS metadata, everything else is real user data.  The
breakdown of a block pointer is roughly:

    3 x 128 bits - DVAs (up to 3-way ditto blocks)
    1 x  64 bits - misc. properties
    1 x  64 bits - block birth
    1 x  64 bits - fill count
    1 x 256 bits - checksum
    3 x  64 bits - future growth (some to be used by encryption)

See this for details:

http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/fs/zfs/sys/spa.h


--Bill

Bill Moore wrote:
>On Wed, Jul 19, 2006 at 03:10:00AM +0200, Casper.Dik at sun.com wrote:
>  
>
>>So how many of the 128 bits of the blockpointer are used for things
>>other than to point where the block is?
>>    
>>
>
>128 *bits*?  What filesystem have you been using?  :)  We''ve got
>luxury-class block pointers that are 128 *bytes*.  We get away with it
>because we also have big data blocks to go with them.  In your typical
>mixed-data storage pool, about 1-2% total (including all the block
>pointers) is ZFS metadata, everything else is real user data.  The
>breakdown of a block pointer is roughly:
>
>    3 x 128 bits - DVAs (up to 3-way ditto blocks)
>    1 x  64 bits - misc. properties
>    1 x  64 bits - block birth
>    1 x  64 bits - fill count
>    1 x 256 bits - checksum
>    3 x  64 bits - future growth (some to be used by encryption)
>  
>
Of course this can change (grow) in the future if the ZFS version number 
changes?

For both the encryption and checksum use, it wouldn''t be unreasonable
to
see the requirements here expand (maybe double?) sometime in the near 
future.

Darren

On Wed, Jul 19, 2006 at 12:38:01PM +0800, Darren Reed
wrote:
> 
> Of course this can change (grow) in the future if the ZFS version number 
> changes?
> 
Possibly.  But it will be difficult.  Unlike current version upgrades,
such a change would require rewriting every block (because each block
isn''t versioned individually).  This would necessitate an offline data
upgrade, which is really nasty and something we try to avoid at all
costs.

Another possibility is to use the block birth time and some pool-wide
metadata to determine when an upgrade took place, and then decode older
versions appropriately.  This avoids the offline data migration, but
adds more complexity elsewhere (and a tax from now and forever whenever
decoding a block pointer).

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Eric Schrock wrote:
>On Wed, Jul 19, 2006 at 12:38:01PM +0800, Darren Reed wrote:
>  
>
>>Of course this can change (grow) in the future if the ZFS version number
>>changes?
>>
...
>Another possibility is to use the block birth time and some pool-wide
>metadata to determine when an upgrade took place, and then decode older
>versions appropriately.  This avoids the offline data migration, but
>adds more complexity elsewhere (and a tax from now and forever whenever
>decoding a block pointer).
>  
>
Can some bits from the block birth time (or elsewhere?) be
sacrificed to represent version or the size of this data?
The size wouldn''t be in bytes, but maybe 128byte chunks?
If the bit used for this is currently 0 (as in a reserved bit(s)),
take it to mean 1+n lots of 128?

Darren

One thing I''ve seen time and again that could easily (a relative term!)
be done here is to have an extended bit.
Just dedicate one of the currently unused bits to be the extended bit. 
When set, the 128 byte pointer is actually 256 bytes long.

Bill

Darren Reed wrote:
> Eric Schrock wrote:
>
>> On Wed, Jul 19, 2006 at 12:38:01PM +0800, Darren Reed wrote:
>>  
>>
>>> Of course this can change (grow) in the future if the ZFS version 
>>> number changes?
>>>
> ...
>
>> Another possibility is to use the block birth time and some pool-wide
>> metadata to determine when an upgrade took place, and then decode older
>> versions appropriately.  This avoids the offline data migration, but
>> adds more complexity elsewhere (and a tax from now and forever whenever
>> decoding a block pointer).
>>  
>>
>
> Can some bits from the block birth time (or elsewhere?) be
> sacrificed to represent version or the size of this data?
> The size wouldn''t be in bytes, but maybe 128byte chunks?
> If the bit used for this is currently 0 (as in a reserved bit(s)),
> take it to mean 1+n lots of 128?
>
> Darren

Bill La Forge wrote:
> One thing I''ve seen time and again that could easily (a relative 
> term!) be done here is to have an extended bit.
> Just dedicate one of the currently unused bits to be the extended bit. 
> When set, the 128 byte pointer is actually 256 bytes long.

And use the same bit in every block to indicate the same thing?

Darren

Darren Reed wrote:
> Bill Moore wrote:
> 
>> On Wed, Jul 19, 2006 at 03:10:00AM +0200, Casper.Dik at sun.com wrote:
>>  
>>
>>> So how many of the 128 bits of the blockpointer are used for things
>>> other than to point where the block is?
>>>   
>>
>> 128 *bits*?  What filesystem have you been using?  :)  We''ve
got
>> luxury-class block pointers that are 128 *bytes*.  We get away with it
> For both the encryption and checksum use, it wouldn''t be
unreasonable to
> see the requirements here expand (maybe double?) sometime in the near 
> future.
I don''t believe it does need to grow at all.  Certainly not for
checksum
or compression and at this stage I don''t seem to need any more space
for
  crypto either.

-- 
Darren J Moffat