zfs discuss - Sep 2011 - aclmode gone in S10u10?

I recently saw a message posted to the sunmanagers list complaining
about installing a kernel patch and suddenly having his ACL''s disappear
completely whenever a chmod occurred. I replied and asked him to check
if the aclmode attribute was gone, as it sounded like the default
discard that was (questionably) implemented in OpenSolaris/Solaris 11.
He confirmed it was, so it looks like the removal of aclmode was
backported to Solaris 10? I don''t know exactly what kernel patch he
installed; it doesn''t look like update 10 is out yet.

Can somebody in the know confirm whether or not aclmode is gone in
update 10? I didn''t think they''d backport such a feature
disabling
change to Solaris 10, seems to not line up with the "long term stability
and compatibility" that''s supposed to be the benefit there...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Mon, Sep 12, 2011 at 8:50 PM, Paul B. Henson <henson at acm.org>
wrote:
> I recently saw a message posted to the sunmanagers list complaining
> about installing a kernel patch and suddenly having his ACL''s
disappear
> completely whenever a chmod occurred. I replied and asked him to check
> if the aclmode attribute was gone, as it sounded like the default
> discard that was (questionably) implemented in OpenSolaris/Solaris 11.
> He confirmed it was, so it looks like the removal of aclmode was
> backported to Solaris 10? I don''t know exactly what kernel patch
he
> installed; it doesn''t look like update 10 is out yet.
Patch-ID# 144500-19
is the kernel update that is the kernel from 10U10

Thank you for flagging this, as Oracle support is telling me I have to
update to this release to get zpool 26 which fixes a zfs bug we are
running into, but if it breaks the ACL inheritance we have been using,
then it is non-starter.
> Can somebody in the know confirm whether or not aclmode is gone in
> update 10? I didn''t think they''d backport such a feature
disabling
> change to Solaris 10, seems to not line up with the "long term
stability
> and compatibility" that''s supposed to be the benefit there...
I cannot confirm this. So far I have one system at this kernel release
and have not yet been able to import the zpool with the problem. The
bug I am running into is that destruction of a snapshot runs the
system out of RAM. The snapshot is a partial zfs recv. I am told this
is a known bug (destruction of large snapshots can run the system out
of RAM as the destroy operation commits as one TXG). There is a fix,
but it in the on-disk format, so just zpool upgrading to version 26
will not fix existing snapshots. We only have 32 GB in this system and
the faulty snapshot *should* be about 2.5 TB.

-- 
{--------1---------2---------3---------4---------5---------6---------7---------}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Designer: Frankenstein, A New Musical
(http://www.facebook.com/event.php?eid=123170297765140)
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, RPI Players

On Tue, Sep 13, 2011 at 1:50 AM, Paul B. Henson <henson at acm.org>
wrote:
> I recently saw a message posted to the sunmanagers list complaining
> about installing a kernel patch and suddenly having his ACL''s
disappear
> completely whenever a chmod occurred. I replied and asked him to check
> if the aclmode attribute was gone, as it sounded like the default
> discard that was (questionably) implemented in OpenSolaris/Solaris 11.
> He confirmed it was, so it looks like the removal of aclmode was
> backported to Solaris 10? I don''t know exactly what kernel patch
he
> installed; it doesn''t look like update 10 is out yet.
Update 10 has been out for about 3 weeks.
> Can somebody in the know confirm whether or not aclmode is gone in
> update 10? I didn''t think they''d backport such a feature
disabling
> change to Solaris 10, seems to not line up with the "long term
stability
> and compatibility" that''s supposed to be the benefit there...
I just tried on a U9 and U10 box. On the U10 system, I did a
simple ''chmod g+s'' on a directory with an ACL, and wham, the
ACL vanished. Same operation on U9, and the ACL is preserved.

(This doesn''t affect me all that much, as ACLs on ZFS have never
really worked right, so anything where the ACL is critical gets stored
on ufs [yuck].)

Also, aclmode is no longer listed in the usage message you see
if you do ''zfs get''.

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

hi
I do not have s10u10 but in s10u9 zfs get does have aclmode
but in  s11x(express) zfs get does not have aclmode any more, there was 
big discussion and seems that  illumos has resurrect aclmode
regards
On 9/13/2011 8:21 AM, Peter Tribble wrote:
> On Tue, Sep 13, 2011 at 1:50 AM, Paul B. Henson<henson at acm.org> 
wrote:
>> I recently saw a message posted to the sunmanagers list complaining
>> about installing a kernel patch and suddenly having his ACL''s
disappear
>> completely whenever a chmod occurred. I replied and asked him to check
>> if the aclmode attribute was gone, as it sounded like the default
>> discard that was (questionably) implemented in OpenSolaris/Solaris 11.
>> He confirmed it was, so it looks like the removal of aclmode was
>> backported to Solaris 10? I don''t know exactly what kernel
patch he
>> installed; it doesn''t look like update 10 is out yet.
> Update 10 has been out for about 3 weeks.
>
>> Can somebody in the know confirm whether or not aclmode is gone in
>> update 10? I didn''t think they''d backport such a
feature disabling
>> change to Solaris 10, seems to not line up with the "long term
stability
>> and compatibility" that''s supposed to be the benefit
there...
> I just tried on a U9 and U10 box. On the U10 system, I did a
> simple ''chmod g+s'' on a directory with an ACL, and wham,
the
> ACL vanished. Same operation on U9, and the ACL is preserved.
>
> (This doesn''t affect me all that much, as ACLs on ZFS have never
> really worked right, so anything where the ACL is critical gets stored
> on ufs [yuck].)
>
> Also, aclmode is no longer listed in the usage message you see
> if you do ''zfs get''.
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 642 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110913/2ec6768b/attachment.vcf>

On 9/13/2011 5:07 AM, Paul Kraus wrote:
> Patch-ID# 144500-19 is the kernel update that is the kernel from
> 10U10
Yep, the guy posting on sunmanagers confirmed that was the patch he
installed which broke aclmode.

Did update 10 sneak out under cover of darkness or what? I didn''t see
any announcements or chatter about it, google doesn''t find anything,
and
the Oracle download site still only shows update 9:

http://www.oracle.com/technetwork/server-storage/solaris/overview/solaris-latest-version-170418.html
> Thank you for flagging this, as Oracle support is telling me I have
> to update to this release to get zpool 26 which fixes a zfs bug we
> are running into, but if it breaks the ACL inheritance we have been
> using, then it is non-starter.
Yup. Fortunately, there are no critical bugs (that Oracle is willing to
fix 8-/) that I''ve been waiting for. I thought I''d be able to
keep an
up-to-date S10 install going while I figure out what to do next; guess
not :(.

He already opened a support ticket, they responded:

-----
ZFS appears to be the only file system supporting NFSv4 ACLs
that attempts to preserve ACLs during chmod(2) operations.
Unfortunately, this requires the ACL to be modified in ways that are
confusing to customers and the time has come to stop the confusion and 
to just "discard" the ACL during chmod(2) operations. This implies
that
the ZFS aclmode property will no longer be needed and will be removed 
from ZFS.

This functionality is targetted to be back in Solaris 11 - as per
CR7002239 want ZFS aclmode property back
-----

Interesting that there''s already a new CR to put it back -- I thought 
that bridge was already burned. It''s already back in Illumos. I wonder 
how long it will take to get put back in S10; or I guess it could be one 
of those CR''s that never gets resolved. I suppose I''ll open my
own
ticket to voice support.
> runs the system out of RAM. The snapshot is a partial zfs recv. I am
> told this is a known bug (destruction of large snapshots can run the
> system out of RAM as the destroy operation commits as one TXG). There
> is a fix, but it in the on-disk format, so just zpool upgrading to
> version 26 will not fix existing snapshots. We only have 32 GB in
> this system and the faulty snapshot *should* be about 2.5 TB.
Hmm, if updating the zpool won''t fix the existing snapshot, how is 
support telling you to recover? Is it going to be one of those wipe and 
rebuild resolutions 8-/? Good luck...

I imagine a pool upgraded to version 26 will no longer be compatible 
with other zfs implementations.

-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On 9/13/2011 5:21 AM, Peter Tribble wrote:
> Update 10 has been out for about 3 weeks.
Where was any announcement posted? I haven''t heard anything about it.
As
far as I can tell, the Oracle site still only has update 9 available for 
download:

http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html
> I just tried on a U9 and U10 box. On the U10 system, I did a
> simple ''chmod g+s'' on a directory with an ACL, and wham,
the
> ACL vanished. Same operation on U9, and the ACL is preserved.
Meh, bogus :(. Thanks for the confirmation.


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On 09/14/11 12:21 AM, Peter Tribble wrote:
> I just tried on a U9 and U10 box. On the U10 system, I did a
> simple ''chmod g+s'' on a directory with an ACL, and wham,
the
> ACL vanished. Same operation on U9, and the ACL is preserved.
>
> (This doesn''t affect me all that much, as ACLs on ZFS have never
> really worked right, so anything where the ACL is critical gets stored
> on ufs [yuck].)
>
Not work on what way?

I have a client who makes extensive (more like excessive!) use of ACLs 
on ZFS and we don''t see any problems.  Other than the ridiculous 
complexity of some of the ACLs that have grown over time.

-- 
Ian.

On Tue, Sep 13, 2011 at 8:34 PM, Paul B. Henson <henson at acm.org>
wrote:
> On 9/13/2011 5:21 AM, Peter Tribble wrote:
>
>> Update 10 has been out for about 3 weeks.
>
> Where was any announcement posted? I haven''t heard anything about
it. As far
> as I can tell, the Oracle site still only has update 9 available for
> download:
>
>
http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html
Hm. They updated that a few weeks ago with a new release but you''re
right,
it''s now back to S10U9. Which leaves me with a whole slew of boxes
running
a release that doesn''t exist.

                    Oracle Solaris 10 8/11 s10x_u10wos_17a X86
  Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
                             Assembled 13 July 2011

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

On 9/13/2011 12:46 PM, Peter Tribble wrote:
> Hm. They updated that a few weeks ago with a new release but
you''re right,
> it''s now back to S10U9. Which leaves me with a whole slew of boxes
running
> a release that doesn''t exist.
>
>                      Oracle Solaris 10 8/11 s10x_u10wos_17a X86
>    Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights
reserved.
>                               Assembled 13 July 2011
Interesting. For some reason it just doesn''t surprise me 8-/.

Maybe it got recalled because it broke aclmode ;).

On a different but related note, I went to try and open a support ticket 
about update 10, and for the life of me I couldn''t figure out how to 
tell it the product in question was "Solaris". I''m not sure
if something
has changed since the last time I opened a bug (many months ago), but 
today the support site went beyond its usual painful all the way to 
unusable <sigh>.


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Tue, Sep 13, 2011 at 3:31 PM, Paul B. Henson <henson at acm.org> wrote:
> Did update 10 sneak out under cover of darkness or what? I didn''t
see
> any announcements or chatter about it, google doesn''t find
anything, and
> the Oracle download site still only shows update 9:
    It was supposed to be released on August 17, but I have not had
support point me at it yet. I suspect (based on some chatter) that
they found a bug that has forced them back to the drawing board for a
few weeks.
>> Thank you for flagging this, as Oracle support is telling me I have
>> to update to this release to get zpool 26 which fixes a zfs bug we
>> are running into, but if it breaks the ACL inheritance we have been
>> using, then it is non-starter.
>
> Yup. Fortunately, there are no critical bugs (that Oracle is willing to
> fix 8-/) that I''ve been waiting for. I thought I''d be
able to keep an
> up-to-date S10 install going while I figure out what to do next; guess
> not :(.
    Uhhh, not being able to destroy snapshots that are "too big" is a
pretty big one for us :-(
> He already opened a support ticket, they responded:
>
> -----
> ZFS appears to be the only file system supporting NFSv4 ACLs
> that attempts to preserve ACLs during chmod(2) operations.
> Unfortunately, this requires the ACL to be modified in ways that are
> confusing to customers and the time has come to stop the confusion and to
> just "discard" the ACL during chmod(2) operations. This implies
that the ZFS
> aclmode property will no longer be needed and will be removed from ZFS.
    So the default behavior will be aclmode = discard, wonderful as
that will probably seriously break what I am working on right now.
> This functionality is targetted to be back in Solaris 11 - as per
> CR7002239 want ZFS aclmode property back
> -----
>> runs the system out of RAM. The snapshot is a partial zfs recv. I am
>> told this is a known bug (destruction of large snapshots can run the
>> system out of RAM as the destroy operation commits as one TXG). There
>> is a fix, but it in the on-disk format, so just zpool upgrading to
>> version 26 will not fix existing snapshots. We only have 32 GB in
>> this system and the faulty snapshot *should* be about 2.5 TB.
>
> Hmm, if updating the zpool won''t fix the existing snapshot, how is
support
> telling you to recover? Is it going to be one of those wipe and rebuild
> resolutions 8-/? Good luck...
    I am having that discussion with Oracle Support right now :-) One
option is to load the system with enough RAM to destroy the legacy
snapshots and then not let them grow too big (in other words, take
more frequent snapshots). But I have been having trouble getting
support to tell me how to estimate the amount of RAM necessary to
delete a snapshot based on *something* I can measure.

P.S. The backup server ran into the bug full steam ahead and is off
line right now. On production, once I realized the root cause I
stopped the snapshot destroy script, so they are just piling up now
:-) Production is 20 TB and 400 million objects, we can''t reload that,
the outage would kill us.

-- 
{--------1---------2---------3---------4---------5---------6---------7---------}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Designer: Frankenstein, A New Musical
(http://www.facebook.com/event.php?eid=123170297765140)
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, RPI Players

On Tue, Sep 13, 2011 at 12:41:36PM -0700, Ian Collins
wrote:
> Not work on what way?
> 
> I have a client who makes extensive (more like excessive!) use of ACLs 
> on ZFS and we don''t see any problems.  Other than the ridiculous 
> complexity of some of the ACLs that have grown over time.
>From my perspective, the fact that any random process can come along and
do a legacy chmod() and destroy your acl falls pretty clearly into the
"not work" camp <sigh>... That and the fact that an NFSv4
exclusive open
destroys an inherited ACL :(. If you can somehow keep things from trying
to poke legacy mode bits and don''t use NFS, maybe you''re ok.

-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

Does anyone see long timeouts during zfs snapshot destroys?
How long does it take to destroy in your situation?
If yes, in what kind of configuration (NexentaStor version if applicable,
snapshot size, type of IO to the pool, zpool
type)?
I''m interested to hear if anyone uses NexentaStor/Opensolaris ZFS
platform to provide storage (datastores)
for VMware vSphere environments via NFS/iscsi.

Thanks
Evgueni

On 09/14/11 01:59 PM, Evgueni Martynov wrote:

It''s not a good idea to hijack an existing thread!
> Does anyone see long timeouts during zfs snapshot destroys?
Define long in this context.
> How long does it take to destroy in your situation?
No more than a second or two.
> If yes, in what kind of configuration (NexentaStor version if applicable,
snapshot size, type of IO to the pool, zpool
> type)?
You should start off describing your configuration, the most common 
cause of long snapshot destroys is dedup and insufficient RAM/cache.

-- 
Ian.

On 13/09/2011 10:12 PM, Ian Collins wrote:
> On 09/14/11 01:59 PM, Evgueni Martynov wrote:
>
> It''s not a good idea to hijack an existing thread!
>> Does anyone see long timeouts during zfs snapshot destroys?
>
> Define long in this context.
~1-2 minutes, may be longer
>
>> How long does it take to destroy in your situation?
>
> No more than a second or two.
That''s what I saw before and the numbers that I would be comfortable
with.
I saw bigger timeouts from the NFS client side (ESX servers).
I saw ESX servers loosing datastores for 50 sec and more interested if anyone
saw that as well an what kind of IO going
to that zpool. We have lots of VMs. And I''m looking for ideas of how to
replicate that in none production environment.
Maybe record the IO on a production server and replay during snapshot destroy in
lab environment.
>
>> If yes, in what kind of configuration (NexentaStor version if
applicable, snapshot size, type of IO to the pool, zpool
>> type)?
>
> You should start off describing your configuration, the most common cause
of long snapshot destroys is dedup and
> insufficient RAM/cache.
no dedupe and 96 GB of RAM
>

> [...]
> 
> He already opened a support ticket, they responded:
> 
> -----
> ZFS appears to be the only file system supporting NFSv4 ACLs
> that attempts to preserve ACLs during chmod(2) operations.
> Unfortunately, this requires the ACL to be modified in ways that are
> confusing to customers and the time has come to stop the confusion
> and to just "discard" the ACL during chmod(2) operations. This
> implies that the ZFS aclmode property will no longer be needed and
> will be removed from ZFS.
> 
> This functionality is targetted to be back in Solaris 11 - as per
> CR7002239 want ZFS aclmode property back
> -----
> 
> [...]
Just so you know and can respond to Oracles wrong statements with
facts: ZFS is not the only filesystem doing something like that.
NetApps WAFL provides an option to make it behave very similar.


Regards
Florian Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110914/f74230c9/attachment.bin>

On Sep 13, 2011, at 3:21 , Peter Tribble wrote:
> On Tue, Sep 13, 2011 at 1:50 AM, Paul B. Henson <henson at acm.org>
wrote:
>> I recently saw a message posted to the sunmanagers list complaining
>> about installing a kernel patch and suddenly having his ACL''s
disappear
>> completely whenever a chmod occurred. I replied and asked him to check
>> if the aclmode attribute was gone, as it sounded like the default
>> discard that was (questionably) implemented in OpenSolaris/Solaris 11.
>> He confirmed it was, so it looks like the removal of aclmode was
>> backported to Solaris 10? I don''t know exactly what kernel
patch he
>> installed; it doesn''t look like update 10 is out yet.
> 
> Update 10 has been out for about 3 weeks.
No it has not. As of now Solaris 10 u10 has not been released.

Sami

On 09/14/11 08:49 PM, Sami Ketola wrote:
> On Sep 13, 2011, at 3:21 , Peter Tribble wrote:
>> On Tue, Sep 13, 2011 at 1:50 AM, Paul B. Henson<henson at
acm.org>  wrote:
>>> I recently saw a message posted to the sunmanagers list complaining
>>> about installing a kernel patch and suddenly having his
ACL''s disappear
>>> completely whenever a chmod occurred. I replied and asked him to
check
>>> if the aclmode attribute was gone, as it sounded like the default
>>> discard that was (questionably) implemented in OpenSolaris/Solaris
11.
>>> He confirmed it was, so it looks like the removal of aclmode was
>>> backported to Solaris 10? I don''t know exactly what kernel
patch he
>>> installed; it doesn''t look like update 10 is out yet.
>> Update 10 has been out for about 3 weeks.
> No it has not. As of now Solaris 10 u10 has not been released.
>
It slipped out the back door for a couple of days a few weeks ago.

-- 
Ian.

On 09/14/11 09:22 PM, Ian Collins wrote:
>    On 09/14/11 08:49 PM, Sami Ketola wrote:
>> On Sep 13, 2011, at 3:21 , Peter Tribble wrote:
>>> Update 10 has been out for about 3 weeks.
>> No it has not. As of now Solaris 10 u10 has not been released.
>>
> It slipped out the back door for a couple of days a few weeks ago.
>
It looks like it''s officially out now:

http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html

-- 
Ian.

On Sep 14, 2011, at 12:22 , Ian Collins wrote:
> On 09/14/11 08:49 PM, Sami Ketola wrote:
>> On Sep 13, 2011, at 3:21 , Peter Tribble wrote:
>>> On Tue, Sep 13, 2011 at 1:50 AM, Paul B. Henson<henson at
acm.org>  wrote:
>>>> I recently saw a message posted to the sunmanagers list
complaining
>>>> about installing a kernel patch and suddenly having his
ACL''s disappear
>>>> completely whenever a chmod occurred. I replied and asked him
to check
>>>> if the aclmode attribute was gone, as it sounded like the
default
>>>> discard that was (questionably) implemented in
OpenSolaris/Solaris 11.
>>>> He confirmed it was, so it looks like the removal of aclmode
was
>>>> backported to Solaris 10? I don''t know exactly what
kernel patch he
>>>> installed; it doesn''t look like update 10 is out yet.
>>> Update 10 has been out for about 3 weeks.
>> No it has not. As of now Solaris 10 u10 has not been released.
>> 
> It slipped out the back door for a couple of days a few weeks ago.
And it is out again now.

                       Oracle Solaris 10 8/11 s10s_u10wos_17b SPARC
  Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
                            Assembled 23 August 2011


Sami
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110915/cda95bd5/attachment.html>