Mike Hanby
2011-Mar-24 20:21 UTC
[Lustre-discuss] Secondary Groups permissions issue on files / directories
Howdy, Our Lustre file system isn''t properly supporting secondary groups any longer. This used to work back when we had 1.8.3 installed, when we upgraded to 1.8.5 we also installed a new MDS and backed up / restored our MTD. All servers and clients are CentOS 5 x86_64 with kernel 2.6.18-194.17.1.el5 Here''s what''s happening: 1. Create a file as user root and assign group to my users primary group $ sudo touch /lustre/testfile $ sudo chgrp myprimary /lustre/testfile $ sudo chmod 770 /lustre/testfile 2. Writing to the file with 770 and primary group works $ echo test > /lustre/testfile $ cat /lustre/testfile test 3. Change groups to wheel (the first of the secondary groups I am assigned) $ sudo chgrp wheel /lustre/testfile 4. I can still read and write the file $ echo test2 > /lustre/testfile $ cat /lustre/testfile test2 5. Without changing anything do an ls and then a cat on the file results in permission denied $ ls -l /lustre/testfile -rwxrwx--- 1 root wheel 6 Mar 24 15:07 /lustre/testfile $ cat /lustre/testfile cat: /lustre/testfile: Permission denied 6. Change the group back to my primary and I can read and write the file many times without issue 7. Change it back to wheel and I can cat the file many times, but once I do an ''ls -l '' on it, I can no longer cat it 8. If I set it to any of my other secondary groups (after changing back to my primary), I can''t cat it at all. This happens on both Infiniband and GigE clients. The mdt is being mounted with acl support /dev/sdb /mnt/mdt-mtpt lustre acl 0 0 Any suggestions on where look?
Mike Hanby
2011-Mar-24 21:22 UTC
[Lustre-discuss] Secondary Groups permissions issue on files / directories
I forgot to point out, both the clients and servers are using the Lustre official RPMs for EL5. Also, on the clients, the "l_getgroups -d <my-UID>" reports the correct GIDs for my user. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Mike Hanby Sent: Thursday, March 24, 2011 3:22 PM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] Secondary Groups permissions issue on files / directories Howdy, Our Lustre file system isn''t properly supporting secondary groups any longer. This used to work back when we had 1.8.3 installed, when we upgraded to 1.8.5 we also installed a new MDS and backed up / restored our MTD. All servers and clients are CentOS 5 x86_64 with kernel 2.6.18-194.17.1.el5 Here''s what''s happening: 1. Create a file as user root and assign group to my users primary group $ sudo touch /lustre/testfile $ sudo chgrp myprimary /lustre/testfile $ sudo chmod 770 /lustre/testfile 2. Writing to the file with 770 and primary group works $ echo test > /lustre/testfile $ cat /lustre/testfile test 3. Change groups to wheel (the first of the secondary groups I am assigned) $ sudo chgrp wheel /lustre/testfile 4. I can still read and write the file $ echo test2 > /lustre/testfile $ cat /lustre/testfile test2 5. Without changing anything do an ls and then a cat on the file results in permission denied $ ls -l /lustre/testfile -rwxrwx--- 1 root wheel 6 Mar 24 15:07 /lustre/testfile $ cat /lustre/testfile cat: /lustre/testfile: Permission denied 6. Change the group back to my primary and I can read and write the file many times without issue 7. Change it back to wheel and I can cat the file many times, but once I do an ''ls -l '' on it, I can no longer cat it 8. If I set it to any of my other secondary groups (after changing back to my primary), I can''t cat it at all. This happens on both Infiniband and GigE clients. The mdt is being mounted with acl support /dev/sdb /mnt/mdt-mtpt lustre acl 0 0 Any suggestions on where look? _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Johann Lombardi
2011-Mar-24 21:38 UTC
[Lustre-discuss] Secondary Groups permissions issue on files / directories
On Thu, Mar 24, 2011 at 04:22:25PM -0500, Mike Hanby wrote:> I forgot to point out, both the clients and servers are using the Lustre official RPMs for EL5. > > Also, on the clients, the "l_getgroups -d <my-UID>" reports the correct GIDs for my user.Could you please run lctl get_param mds.*.group_upcall on the MDS node? Johann
Mike Hanby
2011-Mar-24 21:46 UTC
[Lustre-discuss] Secondary Groups permissions issue on files / directories
# lctl get_param mds.*.group_upcall mds.lustre-MDT0000.group_upcall=NONE -----Original Message----- From: Johann Lombardi [mailto:johann at whamcloud.com] Sent: Thursday, March 24, 2011 4:38 PM To: Mike Hanby Cc: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] Secondary Groups permissions issue on files / directories On Thu, Mar 24, 2011 at 04:22:25PM -0500, Mike Hanby wrote:> I forgot to point out, both the clients and servers are using the Lustre official RPMs for EL5. > > Also, on the clients, the "l_getgroups -d <my-UID>" reports the correct GIDs for my user.Could you please run lctl get_param mds.*.group_upcall on the MDS node? Johann
Johann Lombardi
2011-Mar-24 22:02 UTC
[Lustre-discuss] Secondary Groups permissions issue on files / directories
On Thu, Mar 24, 2011 at 04:46:00PM -0500, Mike Hanby wrote:> # lctl get_param mds.*.group_upcall > mds.lustre-MDT0000.group_upcall=NONEThat''s the problem then. You should be using the l_getgroups group upcall. You can fix this by running the following command on the MGS node: lctl conf_param $FSNAME-MDT0000.mdt.group_upcall=/usr/sbin/l_getgroups ($FSNAME must be replaced with the name of your filesystem) Johann -- Johann Lombardi Whamcloud, Inc. www.whamcloud.com
Mike Hanby
2011-Mar-24 22:14 UTC
[Lustre-discuss] Secondary Groups permissions issue on files / directories
Thanks Johann, that fixed it. I see where it was set to NONE during the tunefs.lustre when adding a new nid. Now I just have to figure out why. -----Original Message----- From: Johann Lombardi [mailto:johann at whamcloud.com] Sent: Thursday, March 24, 2011 5:02 PM To: Mike Hanby Cc: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] Secondary Groups permissions issue on files / directories On Thu, Mar 24, 2011 at 04:46:00PM -0500, Mike Hanby wrote:> # lctl get_param mds.*.group_upcall > mds.lustre-MDT0000.group_upcall=NONEThat''s the problem then. You should be using the l_getgroups group upcall. You can fix this by running the following command on the MGS node: lctl conf_param $FSNAME-MDT0000.mdt.group_upcall=/usr/sbin/l_getgroups ($FSNAME must be replaced with the name of your filesystem) Johann -- Johann Lombardi Whamcloud, Inc. www.whamcloud.com