Hi, I was wondering if there were any additional test we could perform to help debug the group write-permissions issue? Thanks Pat On 06/27/2017 12:29 PM, Pat Haley wrote:> > Hi Soumya, > > One example, we have a common working directory dri_fleat in the > gluster volume > > drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat > > my user (phaley) does not own that directory but is a member of the > group dri_fleat and should have write permissions. When I go to the > nfs-mounted version and try to use the touch command I get the following > > ibfdr-compute-0-4(dri_fleat)% touch dum > touch: cannot touch `dum': Permission denied > > One of the sub-directories under dri_fleat is "test" which phaley owns > > drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test > > Under this directory (mounted via nfs) user phaley can write > > ibfdr-compute-0-4(test)% touch dum > ibfdr-compute-0-4(test)% > > I have put the packet captures in > > http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ > > capture_nfsfail.pcap has the results from the failed touch experiment > capture_nfssucceed.pcap has the results from the successful touch > experiment > > The command I used for these was > > tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w /root/capture_nfstest.pcap > > The brick log files are also in the above link. If I read them > correctly they both funny times. Specifically I see entries from > around 2017-06-27 14:02:37.404865 even though the system time was > 2017-06-27 12:00:00. > > One final item, another reply to my post had a link for possible > problems that could arise from users belonging to too many group. We > have seen the above problem even with a user belonging to only 4 groups. > > Let me know what additional information I can provide. > > Thanks > > Pat > > > On 06/27/2017 02:45 AM, Soumya Koduri wrote: >> >> >> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote: >>> The only problem with using gluster mounted via NFS is that it does not >>> respect the group write permissions which we need. >>> >>> We have an exercise coming up in the a couple of weeks. It seems to me >>> that in order to improve our write times before then, it would be good >>> to solve the group write permissions for gluster mounted via NFS now. >>> We can then revisit gluster mounted via FUSE afterwards. >>> >>> What information would you need to help us force gluster mounted via >>> NFS >>> to respect the group write permissions? >> >> Is this owning group or one of the auxiliary groups whose write >> permissions are not considered? AFAIK, there are no special >> permission checks done by gNFS server when compared to gluster native >> client. >> >> Could you please provide simple steps to reproduce the issue and >> collect pkt trace and nfs/brick logs as well. >> >> Thanks, >> Soumya >-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
On 06/30/2017 07:56 PM, Pat Haley wrote:> > Hi, > > I was wondering if there were any additional test we could perform to > help debug the group write-permissions issue?Sorry for the delay. Please find response inline --> > Thanks > > Pat > > > On 06/27/2017 12:29 PM, Pat Haley wrote: >> >> Hi Soumya, >> >> One example, we have a common working directory dri_fleat in the >> gluster volume >> >> drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat >> >> my user (phaley) does not own that directory but is a member of the >> group dri_fleat and should have write permissions. When I go to the >> nfs-mounted version and try to use the touch command I get the following >> >> ibfdr-compute-0-4(dri_fleat)% touch dum >> touch: cannot touch `dum': Permission denied >> >> One of the sub-directories under dri_fleat is "test" which phaley owns >> >> drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test >> >> Under this directory (mounted via nfs) user phaley can write >> >> ibfdr-compute-0-4(test)% touch dum >> ibfdr-compute-0-4(test)% >> >> I have put the packet captures in >> >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >> >> capture_nfsfail.pcap has the results from the failed touch experiment >> capture_nfssucceed.pcap has the results from the successful touch >> experiment >> >> The command I used for these was >> >> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w /root/capture_nfstest.pcapI hope these pkts were captured on the node where NFS server is running. Could you please use '-i any' as I do not see glusterfs traffic in the tcpdump. Also looks like NFS v4 is used between client & nfs server. Are you using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster volume)? If that is the case please capture fuse-mnt logs as well. This error may well be coming from kernel-NFS itself before the request is sent to fuse-mnt process. FWIW, we have below option - Option: server.manage-gids Default Value: off Description: Resolve groups on the server-side. I haven't looked into what this option exactly does. But it may worth testing with this option on. Thanks, Soumya>> >> The brick log files are also in the above link. If I read them >> correctly they both funny times. Specifically I see entries from >> around 2017-06-27 14:02:37.404865 even though the system time was >> 2017-06-27 12:00:00. >> >> One final item, another reply to my post had a link for possible >> problems that could arise from users belonging to too many group. We >> have seen the above problem even with a user belonging to only 4 groups. >> >> Let me know what additional information I can provide.>> >> Thanks >> >> Pat >> >> >> On 06/27/2017 02:45 AM, Soumya Koduri wrote: >>> >>> >>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote: >>>> The only problem with using gluster mounted via NFS is that it does not >>>> respect the group write permissions which we need. >>>> >>>> We have an exercise coming up in the a couple of weeks. It seems to me >>>> that in order to improve our write times before then, it would be good >>>> to solve the group write permissions for gluster mounted via NFS now. >>>> We can then revisit gluster mounted via FUSE afterwards. >>>> >>>> What information would you need to help us force gluster mounted via >>>> NFS >>>> to respect the group write permissions? >>> >>> Is this owning group or one of the auxiliary groups whose write >>> permissions are not considered? AFAIK, there are no special >>> permission checks done by gNFS server when compared to gluster native >>> client. >>> >>> Could you please provide simple steps to reproduce the issue and >>> collect pkt trace and nfs/brick logs as well. >>> >>> Thanks, >>> Soumya >> >
Hi Soumya, When I originally did the tests I ran tcpdump on the client. I have rerun the tests, doing tcpdump on the server tcpdump -i any -nnSs 0 host 172.16.1.121 -w /root/capture_nfsfail.pcap The results are in the same place http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ capture_nfsfail.pcap has the results from the failed touch experiment capture_nfssucceed.pcap has the results from the successful touch experiment The brick log files are there too. I believe we are using kernel-NFS exporting a fuse mounted gluster volume. I am having Steve confirm this. I tried to find the fuse-mnt logs but failed. Where should I look for them? Thanks Pat On 07/03/2017 07:58 AM, Soumya Koduri wrote:> > > On 06/30/2017 07:56 PM, Pat Haley wrote: >> >> Hi, >> >> I was wondering if there were any additional test we could perform to >> help debug the group write-permissions issue? > > Sorry for the delay. Please find response inline -- > >> >> Thanks >> >> Pat >> >> >> On 06/27/2017 12:29 PM, Pat Haley wrote: >>> >>> Hi Soumya, >>> >>> One example, we have a common working directory dri_fleat in the >>> gluster volume >>> >>> drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat >>> >>> my user (phaley) does not own that directory but is a member of the >>> group dri_fleat and should have write permissions. When I go to the >>> nfs-mounted version and try to use the touch command I get the >>> following >>> >>> ibfdr-compute-0-4(dri_fleat)% touch dum >>> touch: cannot touch `dum': Permission denied >>> >>> One of the sub-directories under dri_fleat is "test" which phaley owns >>> >>> drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test >>> >>> Under this directory (mounted via nfs) user phaley can write >>> >>> ibfdr-compute-0-4(test)% touch dum >>> ibfdr-compute-0-4(test)% >>> >>> I have put the packet captures in >>> >>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >>> >>> capture_nfsfail.pcap has the results from the failed touch experiment >>> capture_nfssucceed.pcap has the results from the successful touch >>> experiment >>> >>> The command I used for these was >>> >>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w /root/capture_nfstest.pcap > > I hope these pkts were captured on the node where NFS server is > running. Could you please use '-i any' as I do not see glusterfs > traffic in the tcpdump. > > Also looks like NFS v4 is used between client & nfs server. Are you > using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster > volume)? > If that is the case please capture fuse-mnt logs as well. This error > may well be coming from kernel-NFS itself before the request is sent > to fuse-mnt process. > > FWIW, we have below option - > > Option: server.manage-gids > Default Value: off > Description: Resolve groups on the server-side. > > I haven't looked into what this option exactly does. But it may worth > testing with this option on. > > Thanks, > Soumya > > >>> >>> The brick log files are also in the above link. If I read them >>> correctly they both funny times. Specifically I see entries from >>> around 2017-06-27 14:02:37.404865 even though the system time was >>> 2017-06-27 12:00:00. >>> >>> One final item, another reply to my post had a link for possible >>> problems that could arise from users belonging to too many group. We >>> have seen the above problem even with a user belonging to only 4 >>> groups. >>> >>> Let me know what additional information I can provide. > >>> >>> Thanks >>> >>> Pat >>> >>> >>> On 06/27/2017 02:45 AM, Soumya Koduri wrote: >>>> >>>> >>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote: >>>>> The only problem with using gluster mounted via NFS is that it >>>>> does not >>>>> respect the group write permissions which we need. >>>>> >>>>> We have an exercise coming up in the a couple of weeks. It seems >>>>> to me >>>>> that in order to improve our write times before then, it would be >>>>> good >>>>> to solve the group write permissions for gluster mounted via NFS now. >>>>> We can then revisit gluster mounted via FUSE afterwards. >>>>> >>>>> What information would you need to help us force gluster mounted via >>>>> NFS >>>>> to respect the group write permissions? >>>> >>>> Is this owning group or one of the auxiliary groups whose write >>>> permissions are not considered? AFAIK, there are no special >>>> permission checks done by gNFS server when compared to gluster native >>>> client. >>>> >>>> Could you please provide simple steps to reproduce the issue and >>>> collect pkt trace and nfs/brick logs as well. >>>> >>>> Thanks, >>>> Soumya >>> >>-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301