Hi Soumya, (1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ I've placed the following 2 log files etc-glusterfs-glusterd.vol.log gdata.log The first has repeated messages about nfs disconnects. The second had the <fuse_mnt_direcotry>.log name (but not much information). (2) About the gluster-NFS native server: do you know where we can find documentation on how to use/install it? We haven't had success in our searches. Thanks Pat On 07/04/2017 05:01 AM, Soumya Koduri wrote:> > > On 07/03/2017 09:01 PM, Pat Haley wrote: >> >> Hi Soumya, >> >> When I originally did the tests I ran tcpdump on the client. >> >> I have rerun the tests, doing tcpdump on the server >> >> tcpdump -i any -nnSs 0 host 172.16.1.121 -w /root/capture_nfsfail.pcap >> >> The results are in the same place >> >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >> >> capture_nfsfail.pcap has the results from the failed touch experiment >> capture_nfssucceed.pcap has the results from the successful touch >> experiment >> >> The brick log files are there too. > > Thanks for sharing. Looks like the error is not generated > @gluster-server side. The permission denied error was caused by either > kNFS or by fuse-mnt process or probably by the combination. > > To check fuse-mnt logs, please look at > /var/log/glusterfs/<fuse_mnt_direcotry>.log > > For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt > and exported it via kNFS, the log location for that fuse_mnt shall be > at /var/log/glusterfs/mnt-fuse-mnt.log > > > Also why not switch to either gluster-NFS native server or NFS-Ganesha > instead of using kNFS, as they are recommended NFS servers to use with > gluster? > > Thanks, > Soumya > >> >> I believe we are using kernel-NFS exporting a fuse mounted gluster >> volume. I am having Steve confirm this. I tried to find the fuse-mnt >> logs but failed. Where should I look for them? >> >> Thanks >> >> Pat >> >> >> >> On 07/03/2017 07:58 AM, Soumya Koduri wrote: >>> >>> >>> On 06/30/2017 07:56 PM, Pat Haley wrote: >>>> >>>> Hi, >>>> >>>> I was wondering if there were any additional test we could perform to >>>> help debug the group write-permissions issue? >>> >>> Sorry for the delay. Please find response inline -- >>> >>>> >>>> Thanks >>>> >>>> Pat >>>> >>>> >>>> On 06/27/2017 12:29 PM, Pat Haley wrote: >>>>> >>>>> Hi Soumya, >>>>> >>>>> One example, we have a common working directory dri_fleat in the >>>>> gluster volume >>>>> >>>>> drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat >>>>> >>>>> my user (phaley) does not own that directory but is a member of the >>>>> group dri_fleat and should have write permissions. When I go to the >>>>> nfs-mounted version and try to use the touch command I get the >>>>> following >>>>> >>>>> ibfdr-compute-0-4(dri_fleat)% touch dum >>>>> touch: cannot touch `dum': Permission denied >>>>> >>>>> One of the sub-directories under dri_fleat is "test" which phaley >>>>> owns >>>>> >>>>> drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test >>>>> >>>>> Under this directory (mounted via nfs) user phaley can write >>>>> >>>>> ibfdr-compute-0-4(test)% touch dum >>>>> ibfdr-compute-0-4(test)% >>>>> >>>>> I have put the packet captures in >>>>> >>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >>>>> >>>>> capture_nfsfail.pcap has the results from the failed touch >>>>> experiment >>>>> capture_nfssucceed.pcap has the results from the successful touch >>>>> experiment >>>>> >>>>> The command I used for these was >>>>> >>>>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w >>>>> /root/capture_nfstest.pcap >>> >>> I hope these pkts were captured on the node where NFS server is >>> running. Could you please use '-i any' as I do not see glusterfs >>> traffic in the tcpdump. >>> >>> Also looks like NFS v4 is used between client & nfs server. Are you >>> using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster >>> volume)? >>> If that is the case please capture fuse-mnt logs as well. This error >>> may well be coming from kernel-NFS itself before the request is sent >>> to fuse-mnt process. >>> >>> FWIW, we have below option - >>> >>> Option: server.manage-gids >>> Default Value: off >>> Description: Resolve groups on the server-side. >>> >>> I haven't looked into what this option exactly does. But it may worth >>> testing with this option on. >>> >>> Thanks, >>> Soumya >>> >>> >>>>> >>>>> The brick log files are also in the above link. If I read them >>>>> correctly they both funny times. Specifically I see entries from >>>>> around 2017-06-27 14:02:37.404865 even though the system time was >>>>> 2017-06-27 12:00:00. >>>>> >>>>> One final item, another reply to my post had a link for possible >>>>> problems that could arise from users belonging to too many group. We >>>>> have seen the above problem even with a user belonging to only 4 >>>>> groups. >>>>> >>>>> Let me know what additional information I can provide. >>> >>>>> >>>>> Thanks >>>>> >>>>> Pat >>>>> >>>>> >>>>> On 06/27/2017 02:45 AM, Soumya Koduri wrote: >>>>>> >>>>>> >>>>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote: >>>>>>> The only problem with using gluster mounted via NFS is that it >>>>>>> does not >>>>>>> respect the group write permissions which we need. >>>>>>> >>>>>>> We have an exercise coming up in the a couple of weeks. It seems >>>>>>> to me >>>>>>> that in order to improve our write times before then, it would be >>>>>>> good >>>>>>> to solve the group write permissions for gluster mounted via NFS >>>>>>> now. >>>>>>> We can then revisit gluster mounted via FUSE afterwards. >>>>>>> >>>>>>> What information would you need to help us force gluster mounted >>>>>>> via >>>>>>> NFS >>>>>>> to respect the group write permissions? >>>>>> >>>>>> Is this owning group or one of the auxiliary groups whose write >>>>>> permissions are not considered? AFAIK, there are no special >>>>>> permission checks done by gNFS server when compared to gluster >>>>>> native >>>>>> client. >>>>>> >>>>>> Could you please provide simple steps to reproduce the issue and >>>>>> collect pkt trace and nfs/brick logs as well. >>>>>> >>>>>> Thanks, >>>>>> Soumya >>>>> >>>> >>-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Hi All, A follow-up question. I've been looking at various pages on nfs-ganesha & gluster. Is there a version of nfs-ganesha that is recommended for use with glusterfs 3.7.11 built on Apr 27 2016 14:09:22 CentOS release 6.8 (Final) Thanks Pat On 07/05/2017 11:36 AM, Pat Haley wrote:> > Hi Soumya, > > (1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ > I've placed the following 2 log files > > etc-glusterfs-glusterd.vol.log > gdata.log > > The first has repeated messages about nfs disconnects. The second had > the <fuse_mnt_direcotry>.log name (but not much information). > > (2) About the gluster-NFS native server: do you know where we can > find documentation on how to use/install it? We haven't had success > in our searches. > > Thanks > > Pat > > > On 07/04/2017 05:01 AM, Soumya Koduri wrote: >> >> >> On 07/03/2017 09:01 PM, Pat Haley wrote: >>> >>> Hi Soumya, >>> >>> When I originally did the tests I ran tcpdump on the client. >>> >>> I have rerun the tests, doing tcpdump on the server >>> >>> tcpdump -i any -nnSs 0 host 172.16.1.121 -w /root/capture_nfsfail.pcap >>> >>> The results are in the same place >>> >>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >>> >>> capture_nfsfail.pcap has the results from the failed touch experiment >>> capture_nfssucceed.pcap has the results from the successful touch >>> experiment >>> >>> The brick log files are there too. >> >> Thanks for sharing. Looks like the error is not generated >> @gluster-server side. The permission denied error was caused by >> either kNFS or by fuse-mnt process or probably by the combination. >> >> To check fuse-mnt logs, please look at >> /var/log/glusterfs/<fuse_mnt_direcotry>.log >> >> For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt >> and exported it via kNFS, the log location for that fuse_mnt shall be >> at /var/log/glusterfs/mnt-fuse-mnt.log >> >> >> Also why not switch to either gluster-NFS native server or >> NFS-Ganesha instead of using kNFS, as they are recommended NFS >> servers to use with gluster? >> >> Thanks, >> Soumya >> >>> >>> I believe we are using kernel-NFS exporting a fuse mounted gluster >>> volume. I am having Steve confirm this. I tried to find the fuse-mnt >>> logs but failed. Where should I look for them? >>> >>> Thanks >>> >>> Pat >>> >>> >>> >>> On 07/03/2017 07:58 AM, Soumya Koduri wrote: >>>> >>>> >>>> On 06/30/2017 07:56 PM, Pat Haley wrote: >>>>> >>>>> Hi, >>>>> >>>>> I was wondering if there were any additional test we could perform to >>>>> help debug the group write-permissions issue? >>>> >>>> Sorry for the delay. Please find response inline -- >>>> >>>>> >>>>> Thanks >>>>> >>>>> Pat >>>>> >>>>> >>>>> On 06/27/2017 12:29 PM, Pat Haley wrote: >>>>>> >>>>>> Hi Soumya, >>>>>> >>>>>> One example, we have a common working directory dri_fleat in the >>>>>> gluster volume >>>>>> >>>>>> drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat >>>>>> >>>>>> my user (phaley) does not own that directory but is a member of the >>>>>> group dri_fleat and should have write permissions. When I go to the >>>>>> nfs-mounted version and try to use the touch command I get the >>>>>> following >>>>>> >>>>>> ibfdr-compute-0-4(dri_fleat)% touch dum >>>>>> touch: cannot touch `dum': Permission denied >>>>>> >>>>>> One of the sub-directories under dri_fleat is "test" which phaley >>>>>> owns >>>>>> >>>>>> drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test >>>>>> >>>>>> Under this directory (mounted via nfs) user phaley can write >>>>>> >>>>>> ibfdr-compute-0-4(test)% touch dum >>>>>> ibfdr-compute-0-4(test)% >>>>>> >>>>>> I have put the packet captures in >>>>>> >>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >>>>>> >>>>>> capture_nfsfail.pcap has the results from the failed touch >>>>>> experiment >>>>>> capture_nfssucceed.pcap has the results from the successful touch >>>>>> experiment >>>>>> >>>>>> The command I used for these was >>>>>> >>>>>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w >>>>>> /root/capture_nfstest.pcap >>>> >>>> I hope these pkts were captured on the node where NFS server is >>>> running. Could you please use '-i any' as I do not see glusterfs >>>> traffic in the tcpdump. >>>> >>>> Also looks like NFS v4 is used between client & nfs server. Are you >>>> using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster >>>> volume)? >>>> If that is the case please capture fuse-mnt logs as well. This error >>>> may well be coming from kernel-NFS itself before the request is sent >>>> to fuse-mnt process. >>>> >>>> FWIW, we have below option - >>>> >>>> Option: server.manage-gids >>>> Default Value: off >>>> Description: Resolve groups on the server-side. >>>> >>>> I haven't looked into what this option exactly does. But it may worth >>>> testing with this option on. >>>> >>>> Thanks, >>>> Soumya >>>> >>>> >>>>>> >>>>>> The brick log files are also in the above link. If I read them >>>>>> correctly they both funny times. Specifically I see entries from >>>>>> around 2017-06-27 14:02:37.404865 even though the system time was >>>>>> 2017-06-27 12:00:00. >>>>>> >>>>>> One final item, another reply to my post had a link for possible >>>>>> problems that could arise from users belonging to too many group. We >>>>>> have seen the above problem even with a user belonging to only 4 >>>>>> groups. >>>>>> >>>>>> Let me know what additional information I can provide. >>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Pat >>>>>> >>>>>> >>>>>> On 06/27/2017 02:45 AM, Soumya Koduri wrote: >>>>>>> >>>>>>> >>>>>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote: >>>>>>>> The only problem with using gluster mounted via NFS is that it >>>>>>>> does not >>>>>>>> respect the group write permissions which we need. >>>>>>>> >>>>>>>> We have an exercise coming up in the a couple of weeks. It seems >>>>>>>> to me >>>>>>>> that in order to improve our write times before then, it would be >>>>>>>> good >>>>>>>> to solve the group write permissions for gluster mounted via >>>>>>>> NFS now. >>>>>>>> We can then revisit gluster mounted via FUSE afterwards. >>>>>>>> >>>>>>>> What information would you need to help us force gluster >>>>>>>> mounted via >>>>>>>> NFS >>>>>>>> to respect the group write permissions? >>>>>>> >>>>>>> Is this owning group or one of the auxiliary groups whose write >>>>>>> permissions are not considered? AFAIK, there are no special >>>>>>> permission checks done by gNFS server when compared to gluster >>>>>>> native >>>>>>> client. >>>>>>> >>>>>>> Could you please provide simple steps to reproduce the issue and >>>>>>> collect pkt trace and nfs/brick logs as well. >>>>>>> >>>>>>> Thanks, >>>>>>> Soumya >>>>>> >>>>> >>> >-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Hi, On 07/07/2017 06:16 AM, Pat Haley wrote:> > Hi All, > > A follow-up question. I've been looking at various pages on nfs-ganesha > & gluster. Is there a version of nfs-ganesha that is recommended for > use with > > glusterfs 3.7.11 built on Apr 27 2016 14:09:22 > CentOS release 6.8 (Final)For glusterfs 3.7, nfs-ganesha-2.3-* version can be used. I see the packages built in centos7 storage sig [1] but not for centos6. Request Niels to comment.> > Thanks > > Pat > > > On 07/05/2017 11:36 AM, Pat Haley wrote: >> >> Hi Soumya, >> >> (1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >> I've placed the following 2 log files >> >> etc-glusterfs-glusterd.vol.log >> gdata.log >> >> The first has repeated messages about nfs disconnects. The second had >> the <fuse_mnt_direcotry>.log name (but not much information). >>Hmm yeah..weird ..there are not much logs in fuse mnt log file.>> (2) About the gluster-NFS native server: do you know where we can >> find documentation on how to use/install it? We haven't had success >> in our searches. >>Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The only requirement is that kernel-NFS has to be disabled for gluster-NFS to come up. Please disable kernel-NFS server and restart glusterd to start gNFS. In case of any issues with starting gNFS server, please look at /var/log/glusterfs/nfs.log. Thanks, Soumya [1] https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/ [2] https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/>> Thanks >> >> Pat >> >> >> On 07/04/2017 05:01 AM, Soumya Koduri wrote: >>> >>> >>> On 07/03/2017 09:01 PM, Pat Haley wrote: >>>> >>>> Hi Soumya, >>>> >>>> When I originally did the tests I ran tcpdump on the client. >>>> >>>> I have rerun the tests, doing tcpdump on the server >>>> >>>> tcpdump -i any -nnSs 0 host 172.16.1.121 -w /root/capture_nfsfail.pcap >>>> >>>> The results are in the same place >>>> >>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >>>> >>>> capture_nfsfail.pcap has the results from the failed touch experiment >>>> capture_nfssucceed.pcap has the results from the successful touch >>>> experiment >>>> >>>> The brick log files are there too. >>> >>> Thanks for sharing. Looks like the error is not generated >>> @gluster-server side. The permission denied error was caused by >>> either kNFS or by fuse-mnt process or probably by the combination. >>> >>> To check fuse-mnt logs, please look at >>> /var/log/glusterfs/<fuse_mnt_direcotry>.log >>> >>> For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt >>> and exported it via kNFS, the log location for that fuse_mnt shall be >>> at /var/log/glusterfs/mnt-fuse-mnt.log >>> >>> >>> Also why not switch to either gluster-NFS native server or >>> NFS-Ganesha instead of using kNFS, as they are recommended NFS >>> servers to use with gluster? >>> >>> Thanks, >>> Soumya >>> >>>> >>>> I believe we are using kernel-NFS exporting a fuse mounted gluster >>>> volume. I am having Steve confirm this. I tried to find the fuse-mnt >>>> logs but failed. Where should I look for them? >>>> >>>> Thanks >>>> >>>> Pat >>>> >>>> >>>> >>>> On 07/03/2017 07:58 AM, Soumya Koduri wrote: >>>>> >>>>> >>>>> On 06/30/2017 07:56 PM, Pat Haley wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I was wondering if there were any additional test we could perform to >>>>>> help debug the group write-permissions issue? >>>>> >>>>> Sorry for the delay. Please find response inline -- >>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Pat >>>>>> >>>>>> >>>>>> On 06/27/2017 12:29 PM, Pat Haley wrote: >>>>>>> >>>>>>> Hi Soumya, >>>>>>> >>>>>>> One example, we have a common working directory dri_fleat in the >>>>>>> gluster volume >>>>>>> >>>>>>> drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat >>>>>>> >>>>>>> my user (phaley) does not own that directory but is a member of the >>>>>>> group dri_fleat and should have write permissions. When I go to the >>>>>>> nfs-mounted version and try to use the touch command I get the >>>>>>> following >>>>>>> >>>>>>> ibfdr-compute-0-4(dri_fleat)% touch dum >>>>>>> touch: cannot touch `dum': Permission denied >>>>>>> >>>>>>> One of the sub-directories under dri_fleat is "test" which phaley >>>>>>> owns >>>>>>> >>>>>>> drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test >>>>>>> >>>>>>> Under this directory (mounted via nfs) user phaley can write >>>>>>> >>>>>>> ibfdr-compute-0-4(test)% touch dum >>>>>>> ibfdr-compute-0-4(test)% >>>>>>> >>>>>>> I have put the packet captures in >>>>>>> >>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/ >>>>>>> >>>>>>> capture_nfsfail.pcap has the results from the failed touch >>>>>>> experiment >>>>>>> capture_nfssucceed.pcap has the results from the successful touch >>>>>>> experiment >>>>>>> >>>>>>> The command I used for these was >>>>>>> >>>>>>> tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w >>>>>>> /root/capture_nfstest.pcap >>>>> >>>>> I hope these pkts were captured on the node where NFS server is >>>>> running. Could you please use '-i any' as I do not see glusterfs >>>>> traffic in the tcpdump. >>>>> >>>>> Also looks like NFS v4 is used between client & nfs server. Are you >>>>> using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted gluster >>>>> volume)? >>>>> If that is the case please capture fuse-mnt logs as well. This error >>>>> may well be coming from kernel-NFS itself before the request is sent >>>>> to fuse-mnt process. >>>>> >>>>> FWIW, we have below option - >>>>> >>>>> Option: server.manage-gids >>>>> Default Value: off >>>>> Description: Resolve groups on the server-side. >>>>> >>>>> I haven't looked into what this option exactly does. But it may worth >>>>> testing with this option on. >>>>> >>>>> Thanks, >>>>> Soumya >>>>> >>>>> >>>>>>> >>>>>>> The brick log files are also in the above link. If I read them >>>>>>> correctly they both funny times. Specifically I see entries from >>>>>>> around 2017-06-27 14:02:37.404865 even though the system time was >>>>>>> 2017-06-27 12:00:00. >>>>>>> >>>>>>> One final item, another reply to my post had a link for possible >>>>>>> problems that could arise from users belonging to too many group. We >>>>>>> have seen the above problem even with a user belonging to only 4 >>>>>>> groups. >>>>>>> >>>>>>> Let me know what additional information I can provide. >>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Pat >>>>>>> >>>>>>> >>>>>>> On 06/27/2017 02:45 AM, Soumya Koduri wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote: >>>>>>>>> The only problem with using gluster mounted via NFS is that it >>>>>>>>> does not >>>>>>>>> respect the group write permissions which we need. >>>>>>>>> >>>>>>>>> We have an exercise coming up in the a couple of weeks. It seems >>>>>>>>> to me >>>>>>>>> that in order to improve our write times before then, it would be >>>>>>>>> good >>>>>>>>> to solve the group write permissions for gluster mounted via >>>>>>>>> NFS now. >>>>>>>>> We can then revisit gluster mounted via FUSE afterwards. >>>>>>>>> >>>>>>>>> What information would you need to help us force gluster >>>>>>>>> mounted via >>>>>>>>> NFS >>>>>>>>> to respect the group write permissions? >>>>>>>> >>>>>>>> Is this owning group or one of the auxiliary groups whose write >>>>>>>> permissions are not considered? AFAIK, there are no special >>>>>>>> permission checks done by gNFS server when compared to gluster >>>>>>>> native >>>>>>>> client. >>>>>>>> >>>>>>>> Could you please provide simple steps to reproduce the issue and >>>>>>>> collect pkt trace and nfs/brick logs as well. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Soumya >>>>>>> >>>>>> >>>> >> >