zfs discuss - Nov 2009 - heads-up: dedup=fletcher4,verify was broken

If you did not do "zfs set dedup=fletcher4,verify <fs>" (which
is available
in build 128 and nightly bits since then), you can ignore this message.

We have changed the on-disk format of the pool when using 
dedup=fletcher4,verify with the integration of:

	6903705 dedup=fletcher4,verify doesn''t byteswap correctly, has lots of
hash
collisions

This is not the default dedup setting; pools that only used "zfs set 
dedup=on" (or =sha256, or =verify, or =sha256,verify) are unaffected.

Before installing bits with this fix, you will need to destroy any 
filesystems that have had dedup=fletcher4,verify set on them.  You can 
preserve your existing data by running:

	zfs set dedup=<any other setting> <old fs>
	zfs snapshot -r <old fs>@snap
	zfs create <new fs>
	zfs send -R <old fs>@snap | zfs recv -d <new fs>
	zfs destroy -r <old fs>

Simply changing the setting from dedup=fletcher4,verify to another setting is 
not sufficient, as this does not modify existing data.

You can verify that your pool isn''t using dedup=fletcher4,verify by
running
	zdb -D <pool> | grep DDT-fletcher4
If there are no matches, your pool is not using dedup=fletcher4,verify, and 
it is safe to install bits with this fix.

Build 128 will be respun to include this fix.

Sorry for the inconvenience,

-- team zfs

We discovered another, more fundamental problem with dedup=fletcher4,verify. 
  I''ve just putback the fix for:

     6904243 zpool scrub/resilver doesn''t work with cross-endian 
dedup=fletcher4,verify blocks

The same instructions as below apply, but in addition, the 
dedup=fletcher4,verify functionality has been removed.  We will investigate 
whether it''s possible to fix these isses and re-enable this
functionality.

--matt


Matthew Ahrens wrote:
> If you did not do "zfs set dedup=fletcher4,verify <fs>"
(which is
> available in build 128 and nightly bits since then), you can ignore this 
> message.
> 
> We have changed the on-disk format of the pool when using 
> dedup=fletcher4,verify with the integration of:
> 
>     6903705 dedup=fletcher4,verify doesn''t byteswap correctly, has
lots
> of hash collisions
> 
> This is not the default dedup setting; pools that only used "zfs set 
> dedup=on" (or =sha256, or =verify, or =sha256,verify) are unaffected.
> 
> Before installing bits with this fix, you will need to destroy any 
> filesystems that have had dedup=fletcher4,verify set on them.  You can 
> preserve your existing data by running:
> 
>     zfs set dedup=<any other setting> <old fs>
>     zfs snapshot -r <old fs>@snap
>     zfs create <new fs>
>     zfs send -R <old fs>@snap | zfs recv -d <new fs>
>     zfs destroy -r <old fs>
> 
> Simply changing the setting from dedup=fletcher4,verify to another 
> setting is not sufficient, as this does not modify existing data.
> 
> You can verify that your pool isn''t using dedup=fletcher4,verify
by running
>     zdb -D <pool> | grep DDT-fletcher4
> If there are no matches, your pool is not using dedup=fletcher4,verify, 
> and it is safe to install bits with this fix.
> 
> Build 128 will be respun to include this fix.
> 
> Sorry for the inconvenience,
> 
> -- team zfs
>

And, for the record, this is my fault.  There is an aspect of endianness
that I simply hadn''t thought of.  When I have a little more time I will
blog about the whole thing, because there are many useful lessons here.

Thank you, Matt, for all your help with this.  And my apologies to
everyone else for the disruption.

Jeff

On Mon, Nov 23, 2009 at 09:15:48PM -0800, Matthew Ahrens
wrote:
> We discovered another, more fundamental problem with 
> dedup=fletcher4,verify. I''ve just putback the fix for:
> 
>     6904243 zpool scrub/resilver doesn''t work with cross-endian 
> dedup=fletcher4,verify blocks
> 
> The same instructions as below apply, but in addition, the 
> dedup=fletcher4,verify functionality has been removed.  We will investigate
> whether it''s possible to fix these isses and re-enable this
functionality.
> 
> --matt
> 
> 
> Matthew Ahrens wrote:
> >If you did not do "zfs set dedup=fletcher4,verify <fs>"
(which is
> >available in build 128 and nightly bits since then), you can ignore
this
> >message.
> >
> >We have changed the on-disk format of the pool when using 
> >dedup=fletcher4,verify with the integration of:
> >
> >    6903705 dedup=fletcher4,verify doesn''t byteswap correctly,
has lots
> >of hash collisions
> >
> >This is not the default dedup setting; pools that only used "zfs
set
> >dedup=on" (or =sha256, or =verify, or =sha256,verify) are
unaffected.
> >
> >Before installing bits with this fix, you will need to destroy any 
> >filesystems that have had dedup=fletcher4,verify set on them.  You can 
> >preserve your existing data by running:
> >
> >    zfs set dedup=<any other setting> <old fs>
> >    zfs snapshot -r <old fs>@snap
> >    zfs create <new fs>
> >    zfs send -R <old fs>@snap | zfs recv -d <new fs>
> >    zfs destroy -r <old fs>
> >
> >Simply changing the setting from dedup=fletcher4,verify to another 
> >setting is not sufficient, as this does not modify existing data.
> >
> >You can verify that your pool isn''t using
dedup=fletcher4,verify by running
> >    zdb -D <pool> | grep DDT-fletcher4
> >If there are no matches, your pool is not using dedup=fletcher4,verify,
> >and it is safe to install bits with this fix.
> >
> >Build 128 will be respun to include this fix.
> >
> >Sorry for the inconvenience,
> >
> >-- team zfs
> >
>

Finally, just to be clear, one last point:  the two fixes integrated
today only affect you if you''ve explicitly set dedup=fletcher4,verify.
To quote Matt:
> This is not the default dedup setting; pools that only used "zfs set 
> dedup=on" (or =sha256, or =verify, or =sha256,verify) are unaffected.
Jeff

On Mon, Nov 23, 2009 at 09:44:41PM -0800, Jeff Bonwick
wrote:
> And, for the record, this is my fault.  There is an aspect of endianness
> that I simply hadn''t thought of.  When I have a little more time I
will
> blog about the whole thing, because there are many useful lessons here.
> 
> Thank you, Matt, for all your help with this.  And my apologies to
> everyone else for the disruption.
> 
> Jeff
> 
> On Mon, Nov 23, 2009 at 09:15:48PM -0800, Matthew Ahrens wrote:
> > We discovered another, more fundamental problem with 
> > dedup=fletcher4,verify. I''ve just putback the fix for:
> > 
> >     6904243 zpool scrub/resilver doesn''t work with
cross-endian
> > dedup=fletcher4,verify blocks
> > 
> > The same instructions as below apply, but in addition, the 
> > dedup=fletcher4,verify functionality has been removed.  We will
investigate
> > whether it''s possible to fix these isses and re-enable this
functionality.
> > 
> > --matt
> > 
> > 
> > Matthew Ahrens wrote:
> > >If you did not do "zfs set dedup=fletcher4,verify
<fs>" (which is
> > >available in build 128 and nightly bits since then), you can
ignore this
> > >message.
> > >
> > >We have changed the on-disk format of the pool when using 
> > >dedup=fletcher4,verify with the integration of:
> > >
> > >    6903705 dedup=fletcher4,verify doesn''t byteswap
correctly, has lots
> > >of hash collisions
> > >
> > >This is not the default dedup setting; pools that only used
"zfs set
> > >dedup=on" (or =sha256, or =verify, or =sha256,verify) are
unaffected.
> > >
> > >Before installing bits with this fix, you will need to destroy any
> > >filesystems that have had dedup=fletcher4,verify set on them.  You
can
> > >preserve your existing data by running:
> > >
> > >    zfs set dedup=<any other setting> <old fs>
> > >    zfs snapshot -r <old fs>@snap
> > >    zfs create <new fs>
> > >    zfs send -R <old fs>@snap | zfs recv -d <new fs>
> > >    zfs destroy -r <old fs>
> > >
> > >Simply changing the setting from dedup=fletcher4,verify to another
> > >setting is not sufficient, as this does not modify existing data.
> > >
> > >You can verify that your pool isn''t using
dedup=fletcher4,verify by running
> > >    zdb -D <pool> | grep DDT-fletcher4
> > >If there are no matches, your pool is not using
dedup=fletcher4,verify,
> > >and it is safe to install bits with this fix.
> > >
> > >Build 128 will be respun to include this fix.
> > >
> > >Sorry for the inconvenience,
> > >
> > >-- team zfs
> > >
> >

When will SXCE 129 be released since 128 was passed over? There used to 
be a release calendar on opensolaris.org but I can''t find it anymore.


Jeff Bonwick wrote:
> And, for the record, this is my fault.  There is an aspect of endianness
> that I simply hadn''t thought of.  When I have a little more time I
will
> blog about the whole thing, because there are many useful lessons here.
>
> Thank you, Matt, for all your help with this.  And my apologies to
> everyone else for the disruption.
>
> Jeff
>
> On Mon, Nov 23, 2009 at 09:15:48PM -0800, Matthew Ahrens wrote:
>   
>> We discovered another, more fundamental problem with 
>> dedup=fletcher4,verify. I''ve just putback the fix for:
>>
>>     6904243 zpool scrub/resilver doesn''t work with
cross-endian
>> dedup=fletcher4,verify blocks
>>
>> The same instructions as below apply, but in addition, the 
>> dedup=fletcher4,verify functionality has been removed.  We will
investigate
>> whether it''s possible to fix these isses and re-enable this
functionality.
>>
>> --matt
>>
>>
>> Matthew Ahrens wrote:
>>     
>>> If you did not do "zfs set dedup=fletcher4,verify
<fs>" (which is
>>> available in build 128 and nightly bits since then), you can ignore
this
>>> message.
>>>
>>> We have changed the on-disk format of the pool when using 
>>> dedup=fletcher4,verify with the integration of:
>>>
>>>    6903705 dedup=fletcher4,verify doesn''t byteswap
correctly, has lots
>>> of hash collisions
>>>
>>> This is not the default dedup setting; pools that only used
"zfs set
>>> dedup=on" (or =sha256, or =verify, or =sha256,verify) are
unaffected.
>>>
>>> Before installing bits with this fix, you will need to destroy any 
>>> filesystems that have had dedup=fletcher4,verify set on them.  You
can
>>> preserve your existing data by running:
>>>
>>>    zfs set dedup=<any other setting> <old fs>
>>>    zfs snapshot -r <old fs>@snap
>>>    zfs create <new fs>
>>>    zfs send -R <old fs>@snap | zfs recv -d <new fs>
>>>    zfs destroy -r <old fs>
>>>
>>> Simply changing the setting from dedup=fletcher4,verify to another 
>>> setting is not sufficient, as this does not modify existing data.
>>>
>>> You can verify that your pool isn''t using
dedup=fletcher4,verify by running
>>>    zdb -D <pool> | grep DDT-fletcher4
>>> If there are no matches, your pool is not using
dedup=fletcher4,verify,
>>> and it is safe to install bits with this fix.
>>>
>>> Build 128 will be respun to include this fix.
>>>
>>> Sorry for the inconvenience,
>>>
>>> -- team zfs
>>>
>>>       
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>   


CONFIDENTIALITY NOTICE:  This communication (including all attachments) is
confidential and is intended for the use of the named addressee(s) only and
may contain information that is private, confidential, privileged, and
exempt from disclosure under law.  All rights to privilege are expressly
claimed and reserved and are not waived.  Any use, dissemination,
distribution, copying or disclosure of this message and any attachments, in
whole or in part, by anyone other than the intended recipient(s) is strictly
prohibited.  If you have received this communication in error, please notify
the sender immediately, delete this communication from all data storage
devices and destroy all hard copies.

Maybe 11/30/2009 ?

According to
http://hub.opensolaris.org/bin/view/Community+Group+on/schedule. we have
onnv_129 11/23/2009 11/30/2009

But..as far as i know those release dates are in a "best effort
basis".

Bruno

Karl Rossing wrote:
> When will SXCE 129 be released since 128 was passed over? There used
> to be a release calendar on opensolaris.org but I can''t find it
anymore.
>
>
> Jeff Bonwick wrote:
>> And, for the record, this is my fault.  There is an aspect of
endianness
>> that I simply hadn''t thought of.  When I have a little more
time I will
>> blog about the whole thing, because there are many useful lessons here.
>>
>> Thank you, Matt, for all your help with this.  And my apologies to
>> everyone else for the disruption.
>>
>> Jeff
>>
>> On Mon, Nov 23, 2009 at 09:15:48PM -0800, Matthew Ahrens wrote:
>>  
>>> We discovered another, more fundamental problem with
>>> dedup=fletcher4,verify. I''ve just putback the fix for:
>>>
>>>     6904243 zpool scrub/resilver doesn''t work with
cross-endian
>>> dedup=fletcher4,verify blocks
>>>
>>> The same instructions as below apply, but in addition, the
>>> dedup=fletcher4,verify functionality has been removed.  We will
>>> investigate whether it''s possible to fix these isses and
re-enable
>>> this functionality.
>>>
>>> --matt
>>>
>>>
>>> Matthew Ahrens wrote:
>>>    
>>>> If you did not do "zfs set dedup=fletcher4,verify
<fs>" (which is
>>>> available in build 128 and nightly bits since then), you can
ignore
>>>> this message.
>>>>
>>>> We have changed the on-disk format of the pool when using
>>>> dedup=fletcher4,verify with the integration of:
>>>>
>>>>    6903705 dedup=fletcher4,verify doesn''t byteswap
correctly, has
>>>> lots of hash collisions
>>>>
>>>> This is not the default dedup setting; pools that only used
"zfs
>>>> set dedup=on" (or =sha256, or =verify, or =sha256,verify)
are
>>>> unaffected.
>>>>
>>>> Before installing bits with this fix, you will need to destroy
any
>>>> filesystems that have had dedup=fletcher4,verify set on them. 
You
>>>> can preserve your existing data by running:
>>>>
>>>>    zfs set dedup=<any other setting> <old fs>
>>>>    zfs snapshot -r <old fs>@snap
>>>>    zfs create <new fs>
>>>>    zfs send -R <old fs>@snap | zfs recv -d <new fs>
>>>>    zfs destroy -r <old fs>
>>>>
>>>> Simply changing the setting from dedup=fletcher4,verify to
another
>>>> setting is not sufficient, as this does not modify existing
data.
>>>>
>>>> You can verify that your pool isn''t using
dedup=fletcher4,verify by
>>>> running
>>>>    zdb -D <pool> | grep DDT-fletcher4
>>>> If there are no matches, your pool is not using
>>>> dedup=fletcher4,verify, and it is safe to install bits with
this fix.
>>>>
>>>> Build 128 will be respun to include this fix.
>>>>
>>>> Sorry for the inconvenience,
>>>>
>>>> -- team zfs
>>>>
>>>>       
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>   
>
>
>
> CONFIDENTIALITY NOTICE:  This communication (including all
> attachments) is
> confidential and is intended for the use of the named addressee(s)
> only and
> may contain information that is private, confidential, privileged, and
> exempt from disclosure under law.  All rights to privilege are expressly
> claimed and reserved and are not waived.  Any use, dissemination,
> distribution, copying or disclosure of this message and any
> attachments, in
> whole or in part, by anyone other than the intended recipient(s) is
> strictly
> prohibited.  If you have received this communication in error, please
> notify
> the sender immediately, delete this communication from all data storage
> devices and destroy all hard copies.
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3656 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091125/6229bb74/attachment.bin>

It seems b128 will be re-spun for IPS, and was canceled only for SXCE.
-- 
This message posted from opensolaris.org