zfs discuss - Oct 2009 - Solaris 10 samba in AD mode broken when user in > 32 AD groups

We''re currently using the Sun bundled Samba to provide CIFS access to
our
ZFS user/group directories.

I found a bug in active directory integration mode, where if a user is in
more than 32 active directory groups, samba calls setgroups with a group
list of greater than 32, which fails, resulting in the user having
absolutely no group privileges beyond their primary group.

I opened a Sun service request, #71547904, to try and get this resolved.
When I initially opened it, I did not know what the underlying problem was.
However, I wasn''t making any progress through Sun tech support, so I
ended
up installing the Sun samba source code package and diagnosing the problem
myself. In addition, I provided Sun technical report with a simple two line
patch that fixes the problem.

Unfortunately, I am getting the complete run around on this issue and after
almost 2 months have been unable to get the problem fixed.

They keep telling me that support for more than 32 groups in Solaris is not
a bug, but rather an RFE. I completely agree -- I''m not asking for
Solaris
to support more than 32 groups (although, as an aside, it sure would be
nice if it did -- 32 is pretty small nowadays; I doubt this will get fixed
in Solaris 10, but anyone have any idea about possible progress on that in
openSolaris?); all I''m asking is that samba be fixed so the user at
least gets the first 32 groups they are in rather than none at all. That is
the behavior of a local login or over NFS, the effective group privileges
are that of the first 32 groups.

Evidently the samba engineering group is in Prague. I don''t know if it
is a
language problem, or where the confusion is coming from, but even after
escalating this through our regional support manager, they are still
refusing to fix this bug and claiming it is an RFE.

I think based on the information I provided it should be blindingly obvious
that this is a bug, with a fairly trivial fix. I''m pretty sure if they
had
just fixed it rather than spent all this time arguing about it would
have taken less time and resources than they''ve already wasted 8-/.

While not directly a ZFS problem, I was hoping one of the many intelligent
and skilled Sun engineers that hang out on this mailing list :) might do me
a big favor, look at SR#71547904, confirm that it is actually a bug, and
use their internal contacts to somehow convince the samba sustaining
engineering group to fix it? Please?

Thanks much...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

>
>We''re currently using the Sun bundled Samba to provide CIFS access
to our
>ZFS user/group directories.
So why not the built-in CIFS support in OpenSolaris?  Probably has a 
similar issue, but still.
>I found a bug in active directory integration mode, where if a user is in
>more than 32 active directory groups, samba calls setgroups with a group
>list of greater than 32, which fails, resulting in the user having
>absolutely no group privileges beyond their primary group.
That''s not nice and that should be fixed even when the OS
doesn''t support
more than 32 bits.  How many groups do you want?
>They keep telling me that support for more than 32 groups in Solaris is not
>a bug, but rather an RFE. I completely agree -- I''m not asking for
Solaris
>to support more than 32 groups (although, as an aside, it sure would be
>nice if it did -- 32 is pretty small nowadays; I doubt this will get fixed
>in Solaris 10, but anyone have any idea about possible progress on that in
>openSolaris?); all I''m asking is that samba be fixed so the user at
>least gets the first 32 groups they are in rather than none at all. That is
>the behavior of a local login or over NFS, the effective group privileges
>are that of the first 32 groups.
I''m actually working on fixing this in OpenSolaris and we may even
backport this to S10.
>Evidently the samba engineering group is in Prague. I don''t know if
it is a
>language problem, or where the confusion is coming from, but even after
>escalating this through our regional support manager, they are still
>refusing to fix this bug and claiming it is an RFE.
What''s the bug number?

Casper

On Tue, 13 Oct 2009 Casper.Dik at Sun.COM wrote:
> So why not the built-in CIFS support in OpenSolaris?  Probably has a
> similar issue, but still.
I wouldn''t think it has this same issue; presumably it won''t
support more
than the kernel limit of 32 groups, but I can''t imagine that in the
case
when a user is in more than 32 active directory groups it would simply
discard all group membership :(. I haven''t tested it, but I would guess
it
would behave like the underlying operating system and simply truncate the
group list at 32, with the user losing any additional privileges granted by
the rest of the groups.

I definitely have my eye on transitioning to OpenSolaris, hopefully
sometime in mid to late next year. Unfortunately, OpenSolaris wasn''t
quite
enterprise ready when we went into production with this system, and while I
think by now it''s pretty close if not there, it''s going to
take some time
to put together a prototype, sell management on it, and migrate production
services.
> That''s not nice and that should be fixed even when the OS
doesn''t support
> more than 32 bits.  How many groups do you want?
All of them :). I think currently the most groups any single user is in is
about 100. 64 would probably cover everyone except a handful of users.
Linux currently supports a maximum of 65536 groups per user, while I
won''t
make the mistake of saying no one would ever need more than that ;), I
don''t think we would exceed that any time soon.
> I''m actually working on fixing this in OpenSolaris and we may even
> backport this to S10.
Really? Cool. Any timeline on getting it into a development build?
What''s
the current maximum number of groups you''re working towards? Better
group
support would be another bullet point for transitioning to openSolaris.

Regarding Solaris 10, my understanding was that the current 32 group limit
could only be changed by modifying internal kernel structures that would
break backwards compatibility, which wouldn''t happen because Solaris
guarantees backwards binary compatibility. I could most definitely be
mistaken though.
> What''s the bug number?
There is no bug number :(, as they refuse to classify it as a bug -- they
keep insisting it is an RFE, and pointing towards the existing RFE #''s
for
increasing the number of groups supported by Solaris.

The service request is #71547904, although now that I think about it they
haven''t been keeping the ticket updated. I''ll send you a copy
of the thread
I''ve had with the support engineers directly.

Here''s the patch I submitted. It adds three lines, one of which is
blank
8-/. I''m just really confused why they''d rather spend months
arguing it
isn''t a bug rather than just spending five minutes applying this simple
patch <sigh>. I''d just run the version I compiled locally, but
it''s fairly
clear that the source code provided is not the same as the source code used
to generate the production binary, so I''d really prefer an official
fix.


root at niblet /usr/sfw/src/samba/source/auth # diff -u auth_util.c.orig
auth_util.c
--- auth_util.c.orig    Fri Sep 11 16:18:46 2009
+++ auth_util.c Fri Sep 11 16:25:56 2009
@@ -1042,6 +1042,7 @@
        TALLOC_CTX *mem_ctx;
        NTSTATUS status;
        size_t i;
+       int ngroups_max = groups_max();


        mem_ctx = talloc_new(NULL);
@@ -1099,6 +1100,8 @@
                }
                add_gid_to_array_unique(server_info, gid,
&server_info->groups,
                                        &server_info->n_groups);
+
+               if (server_info->n_groups == ngroups_max) break;
        }

        debug_nt_user_token(DBGC_AUTH, 10, server_info->ptok);



-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

>Regarding Solaris 10, my understanding was that the current 32 group limit
>could only be changed by modifying internal kernel structures that would
>break backwards compatibility, which wouldn''t happen because
Solaris
>guarantees backwards binary compatibility. I could most definitely be
>mistaken though.
That''s not entirely true; the issue is similar having more than 16
groups
as it breaks AUTH_SYS over-the-wire "authentication" but we already
have
that now.

But see:

http://opensolaris.org/jive/thread.jspa?threadID=114685

For now, we''re aiming for 1024 groups but also make sure that the
userland will work without any dependencies.
>> What''s the bug number?
>
>There is no bug number :(, as they refuse to classify it as a bug -- they
>keep insisting it is an RFE, and pointing towards the existing RFE
#''s for
>increasing the number of groups supported by Solaris.
The "change request", then.  It must have a bug id.
>The service request is #71547904, although now that I think about it they
>haven''t been keeping the ticket updated. I''ll send you a
copy of the thread
>I''ve had with the support engineers directly.
>
>Here''s the patch I submitted. It adds three lines, one of which is
blank
>8-/. I''m just really confused why they''d rather spend
months arguing it
>isn''t a bug rather than just spending five minutes applying this
simple
>patch <sigh>. I''d just run the version I compiled locally,
but it''s fairly
>clear that the source code provided is not the same as the source code used
>to generate the production binary, so I''d really prefer an official
fix.
Well, I can understand the sense of that.  (Not for OpenSolaris, but for 
S10)  A backport cost a bit so perhaps that''s what they want to avoid.

Casper

Paul B. Henson wrote:
>> So why not the built-in CIFS support in OpenSolaris?  Probably has a
>> similar issue, but still.
> 
> I wouldn''t think it has this same issue; presumably it
won''t support more
> than the kernel limit of 32 groups, but I can''t imagine that in
the case
> when a user is in more than 32 active directory groups it would simply
> discard all group membership :(. I haven''t tested it, but I would
guess it
> would behave like the underlying operating system and simply truncate the
> group list at 32, with the user losing any additional privileges granted by
> the rest of the groups.
Ah. No. If you''re using idmap and are mapping to an AD server, the
windows SIDs
(which are both users and groups) are stored in a cred struct (in cr_ksid) which
allows more than 32 groups, up to 64k iirc.

Playing around with idmap to map UID/GIDs to SIDs and vice versa can be done 
locally without an AD or LDAP server too.

-Drew

On Tue, 13 Oct 2009 Casper.Dik at Sun.COM wrote:
> That''s not entirely true; the issue is similar having more than 16
groups
> as it breaks AUTH_SYS over-the-wire "authentication" but we
already have
> that now.
[...]
> For now, we''re aiming for 1024 groups but also make sure that the
> userland will work without any dependencies.
Good to know; I''m definitely looking forward to this. 1024 will
hopefully
suffice for at least a while :).
> The "change request", then.  It must have a bug id.
The only number I have unique to my request is the SR #. There has been no
bug opened, and as I mentioned they are referring to an existing RFE
regarding increasing the maximum number of groups supported by the
operating system (these references are in the thread I forwarded you
directly) which is simply not relevant. In fact, it appears my service
request has been marked as "canceled" without my knowledge, leaving
pretty
much no official trail of my request :(.
> Well, I can understand the sense of that.  (Not for OpenSolaris, but for
> S10)  A backport cost a bit so perhaps that''s what they want to
avoid.
I can''t see the cost of applying a three line patch as being
particularly
high, but I guess there is some inherent cost in quality control, testing,
and packaging a patch. But upstream just released some security fixes for
the 3.0.x branch, which hopefully they''re going to incorporate and
release
in a patch, and the incremental cost of adding in my simple fix must be
negligible.


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Tue, 13 Oct 2009, Drew Balfour wrote:
> Ah. No. If you''re using idmap and are mapping to an AD server, the
> windows SIDs (which are both users and groups) are stored in a cred
> struct (in cr_ksid) which allows more than 32 groups, up to 64k iirc.
Ah, yes, I neglected to consider that given the CIFS server in OpenSolaris
runs in-kernel it''s not subject to the same OS limitations as a user
level
process. Once Casper finishes his work and access via NFS is no longer
limited to 32 groups that will be quite sweet...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Tue, Oct 13, 2009 at 09:20:23AM -0700, Paul B. Henson
wrote:
> 
> We''re currently using the Sun bundled Samba to provide CIFS access
to our
> ZFS user/group directories.
...
> Evidently the samba engineering group is in Prague. I don''t know
if it is a
> language problem, or where the confusion is coming from, but even after
> escalating this through our regional support manager, they are still
> refusing to fix this bug and claiming it is an RFE.
Havn''t tested the bundle samba stuff for a "long" time, since
I don''t trust
it: The bundled stuff didn''t work when tested; packages are IMHO
awefully assembled; Problems are not understood by the involved engineers
(or they are not willingly to understand); The team seems to follow the
dogma, fix the symptoms and not the root cause.

So at least if the bundled stuff is modified according to their RFEs on
bugzilla, don''t be suprised, if your environment gets screwed up -
especially when you have a mixed users group, i.e. Windows and *ix based
user, which are using workgroup directories for sharing their stuff.

So we still use the original samba and it causes no headaches. Once
we had a problem when switching some desktops to Vista, MS Office 2007
due to the new win strategy "save changes to a tmp file, than rename to
the original file" -> wrong ACLs, however this has been fixed within
ONE DAY: Just did some code scanning, talked to Jeremy Allison via smb IRC
channel and viola, he came up with a fix pretty fast. So I didn''t need
to waste my time explaining the problem again and again to SUN support,
creating explorer archives, which usually hang the NFS services which
couldn''t be fixed without a reboot!, and waiting several months to get
it fixed (BTW: IIRC, I opened a case for this via sun support, so if it
hasn''t be silently closed, its probably still open ...).

Since we guess, that CIFS gets screwed up by the same team, we don''t
use
it either (well, and can''t because we''ve no ADS ;-)).

My 10?.

Regards,
jel.
-- 
Otto-von-Guericke University     http://www.cs.uni-magdeburg.de/
Department of Computer Science   Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany         Tel: +49 391 67 12768

On 14/10/2009, at 2:27 AM, Casper.Dik at Sun.COM wrote:
> So why not the built-in CIFS support in OpenSolaris?  Probably has a
> similar issue, but still.
In my case, it?s at least two reasons:

  * Crossing mountpoints requires separate shares - Samba can share an  
entire hierarchy regardless of ZFS filesystems beneath the sharepoint.

  * LDAP integration - the in-kernel CIFS only supports real AD (LDAP 
+krb5) for directory binding otherwise all users must have a  
separately managed local system accounts.

Until these features are available via the in-kernel CIFS  
implementation, I?m forced to stick with Samba for our CIFS needs.

cheers,
James

Jens Elkner wrote:
> On Tue, Oct 13, 2009 at 09:20:23AM -0700, Paul B. Henson wrote:
>> We''re currently using the Sun bundled Samba to provide CIFS
access to our
>> ZFS user/group directories.
> ...
>> Evidently the samba engineering group is in Prague. I don''t
know if it is a
>> language problem, or where the confusion is coming from, but even after
>> escalating this through our regional support manager, they are still
>> refusing to fix this bug and claiming it is an RFE.
> 
> Havn''t tested the bundle samba stuff for a "long" time,
since I don''t trust
> it: The bundled stuff didn''t work when tested; packages are IMHO
> awefully assembled; Problems are not understood by the involved engineers
> (or they are not willingly to understand); The team seems to follow the
> dogma, fix the symptoms and not the root cause.
For Opensolaris, Solaris CIFS != samba. Solaris now has a native in kernel CIFS 
server which has nothing to do with samba. Apart from having it''s
commands start
with "smb", which can be confusing.

http://www.opensolaris.org/os/project/cifs-server/

-Drew

On Tue, Oct 13, 2009 at 10:59:37PM -0600, Drew Balfour wrote:
...
> For Opensolaris, Solaris CIFS != samba. Solaris now has a native in kernel 
> CIFS server which has nothing to do with samba. Apart from having
it''s
> commands start with "smb", which can be confusing.
> 
> http://www.opensolaris.org/os/project/cifs-server/
Ah ok. Thanx for clarification!

Regards,
jel.
-- 
Otto-von-Guericke University     http://www.cs.uni-magdeburg.de/
Department of Computer Science   Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany         Tel: +49 391 67 12768