Hello All,
Having trouble exporting a lustre mount with NFS..
When mounting the NFS export on client:
[root at nfsclient /]# mount 192.168.100.77:/mnt/lustre_mail_fs
/mnt/lustre (have tried nearly every possible switch that I can think
of, don''t think it''s related to "how" I attempt to
mount)
mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by
server: Permission denied
Server shows log:
Jul 14 10:24:28 lustreclient mountd[9868]: authenticated mount request
from 192.168.100.2:720 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs)
Jul 14 10:24:28 lustreclient mountd[9868]: can''t stat exported dir
/mnt/lustre_mail_fs: Permission denied
Everything from a NFS service/config perspective appears to be
functional, and I can indeed export a local filesystem without errors...
Additionally, I have no trouble using the mounted lustre filesystem, I
can even rsync data from the nfs client to the lustre file system on the
nfs server..
Any clues? Your knowledge and experience would be greatly appreciated!
See below for config specifics..
Thanks!
Billy Olson
Server/Client Specifics:
Lustre Client/NFS Server: 192.168.100.77
CentOS release 5.4 (Final)
lustre-client-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3
lustre-client-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3
Kernel: 2.6.18-164.11.1.el5
/etc/exports:
/mnt/lustre_mail_fs 192.168.100.0/24(ro,insecure)
I''ve tried various other options all the same outcome, these seemed
like
the best to test with though..
[root at lustreclient ]# rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 645 status
100024 1 tcp 648 status
100011 1 udp 702 rquotad
100011 2 udp 702 rquotad
100011 1 tcp 705 rquotad
100011 2 tcp 705 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 37527 nlockmgr
100021 3 udp 37527 nlockmgr
100021 4 udp 37527 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 52555 nlockmgr
100021 3 tcp 52555 nlockmgr
100021 4 tcp 52555 nlockmgr
100005 1 udp 716 mountd
100005 1 tcp 719 mountd
100005 2 udp 716 mountd
100005 2 tcp 719 mountd
100005 3 udp 716 mountd
100005 3 tcp 719 mountd
[root at lustreclient ]# exportfs -v -r
exporting 192.168.100.0/24:/mnt/lustre_mail_fs
NFS Client: 192.168.100.2
Somebody, anybody? I''m sure it''s something fairly simple, but it escapes me, assistance would be greatly appreciated! Thanks Again!> Hello All, > > Having trouble exporting a lustre mount with NFS.. > > When mounting the NFS export on client: > [root at nfsclient /]# mount 192.168.100.77:/mnt/lustre_mail_fs > /mnt/lustre (have tried nearly every possible switch that I can think > of, don''t think it''s related to "how" I attempt to mount) > mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by > server: Permission denied > > Server shows log: > Jul 14 10:24:28 lustreclient mountd[9868]: authenticated mount request > from 192.168.100.2:720 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs) > Jul 14 10:24:28 lustreclient mountd[9868]: can''t stat exported dir > /mnt/lustre_mail_fs: Permission denied > > Everything from a NFS service/config perspective appears to be > functional, and I can indeed export a local filesystem without errors... > > Additionally, I have no trouble using the mounted lustre filesystem, I > can even rsync data from the nfs client to the lustre file system on the > nfs server.. > > Any clues? Your knowledge and experience would be greatly appreciated! > See below for config specifics.. > > Thanks! > Billy Olson > > Server/Client Specifics: > Lustre Client/NFS Server: 192.168.100.77 > CentOS release 5.4 (Final) > lustre-client-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 > lustre-client-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 > Kernel: 2.6.18-164.11.1.el5 > > /etc/exports: > /mnt/lustre_mail_fs 192.168.100.0/24(ro,insecure) > I''ve tried various other options all the same outcome, these seemed like > the best to test with though.. > > [root at lustreclient ]# rpcinfo -p > program vers proto port > 100000 2 tcp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 645 status > 100024 1 tcp 648 status > 100011 1 udp 702 rquotad > 100011 2 udp 702 rquotad > 100011 1 tcp 705 rquotad > 100011 2 tcp 705 rquotad > 100003 2 udp 2049 nfs > 100003 3 udp 2049 nfs > 100003 4 udp 2049 nfs > 100021 1 udp 37527 nlockmgr > 100021 3 udp 37527 nlockmgr > 100021 4 udp 37527 nlockmgr > 100003 2 tcp 2049 nfs > 100003 3 tcp 2049 nfs > 100003 4 tcp 2049 nfs > 100021 1 tcp 52555 nlockmgr > 100021 3 tcp 52555 nlockmgr > 100021 4 tcp 52555 nlockmgr > 100005 1 udp 716 mountd > 100005 1 tcp 719 mountd > 100005 2 udp 716 mountd > 100005 2 tcp 719 mountd > 100005 3 udp 716 mountd > 100005 3 tcp 719 mountd > > [root at lustreclient ]# exportfs -v -r > exporting 192.168.100.0/24:/mnt/lustre_mail_fs > > NFS Client: 192.168.100.2 > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On 2010-07-15, at 08:33, William Olson wrote:> Somebody, anybody? I''m sure it''s something fairly simple, but it > escapes me, assistance would be greatly appreciated!I can''t comment, since my NFS re-exporting (to a MacOS client) is working fine. Hopefully soon I can stop doing that and use the native Mac client, but I don''t think my wife is willing to use alpha software yet. Have you restarted the NFS server after changing the exports? Are you sure the permissions are correct? Have you verified that exporting a non-lustre filesystem from this server works? My /etc/exports is: /myth 192.168.10.160(rw,async,insecure)>> Hello All, >> >> Having trouble exporting a lustre mount with NFS.. >> >> When mounting the NFS export on client: >> [root at nfsclient /]# mount 192.168.100.77:/mnt/lustre_mail_fs >> /mnt/lustre (have tried nearly every possible switch that I can think >> of, don''t think it''s related to "how" I attempt to mount) >> mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by >> server: Permission denied >> >> Server shows log: >> Jul 14 10:24:28 lustreclient mountd[9868]: authenticated mount request >> from 192.168.100.2:720 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs) >> Jul 14 10:24:28 lustreclient mountd[9868]: can''t stat exported dir >> /mnt/lustre_mail_fs: Permission denied >> >> Everything from a NFS service/config perspective appears to be >> functional, and I can indeed export a local filesystem without errors... >> >> Additionally, I have no trouble using the mounted lustre filesystem, I >> can even rsync data from the nfs client to the lustre file system on the >> nfs server.. >> >> Any clues? Your knowledge and experience would be greatly appreciated! >> See below for config specifics.. >> >> Thanks! >> Billy Olson >> >> Server/Client Specifics: >> Lustre Client/NFS Server: 192.168.100.77 >> CentOS release 5.4 (Final) >> lustre-client-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >> lustre-client-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >> Kernel: 2.6.18-164.11.1.el5 >> >> /etc/exports: >> /mnt/lustre_mail_fs 192.168.100.0/24(ro,insecure) >> I''ve tried various other options all the same outcome, these seemed like >> the best to test with though.. >> >> [root at lustreclient ]# rpcinfo -p >> program vers proto port >> 100000 2 tcp 111 portmapper >> 100000 2 udp 111 portmapper >> 100024 1 udp 645 status >> 100024 1 tcp 648 status >> 100011 1 udp 702 rquotad >> 100011 2 udp 702 rquotad >> 100011 1 tcp 705 rquotad >> 100011 2 tcp 705 rquotad >> 100003 2 udp 2049 nfs >> 100003 3 udp 2049 nfs >> 100003 4 udp 2049 nfs >> 100021 1 udp 37527 nlockmgr >> 100021 3 udp 37527 nlockmgr >> 100021 4 udp 37527 nlockmgr >> 100003 2 tcp 2049 nfs >> 100003 3 tcp 2049 nfs >> 100003 4 tcp 2049 nfs >> 100021 1 tcp 52555 nlockmgr >> 100021 3 tcp 52555 nlockmgr >> 100021 4 tcp 52555 nlockmgr >> 100005 1 udp 716 mountd >> 100005 1 tcp 719 mountd >> 100005 2 udp 716 mountd >> 100005 2 tcp 719 mountd >> 100005 3 udp 716 mountd >> 100005 3 tcp 719 mountd >> >> [root at lustreclient ]# exportfs -v -r >> exporting 192.168.100.0/24:/mnt/lustre_mail_fs >> >> NFS Client: 192.168.100.2 >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
On 7/15/2010 5:48 PM, Andreas Dilger wrote:> On 2010-07-15, at 08:33, William Olson wrote: > >> Somebody, anybody? I''m sure it''s something fairly simple, but it >> escapes me, assistance would be greatly appreciated! >> > I can''t comment, since my NFS re-exporting (to a MacOS client) is working fine. Hopefully soon I can stop doing that and use the native Mac client, but I don''t think my wife is willing to use alpha software yet. > >LOL, yeah. Understood.> Have you restarted the NFS server after changing the exports? >Many many times after every change..> Are you sure the permissions are correct? >As stated below all processes that I''ve tested have full access and no permissions issues when interacting with the lustre file system directly.. Accepting for the mountd while attempting to stat the lustre fs during an NFS client connection..> Have you verified that exporting a non-lustre filesystem from this server works? >Indeed, as stated below I have done so.> My /etc/exports is: > /myth 192.168.10.160(rw,async,insecure) >I may not have tried the async option, I''ll give that a shot really quick just for kicks.. ;) Nope, same result still... hrm.. Thanks for Responding at least Andreas! I''m convinced that this is more of a lustre centric issue than an NFS issue, must be some interaction either on the kernel level or otherwise that''s running into a permissions block somewhere..>>> Hello All, >>> >>> Having trouble exporting a lustre mount with NFS.. >>> >>> When mounting the NFS export on client: >>> [root at nfsclient /]# mount 192.168.100.77:/mnt/lustre_mail_fs >>> /mnt/lustre (have tried nearly every possible switch that I can think >>> of, don''t think it''s related to "how" I attempt to mount) >>> mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by >>> server: Permission denied >>> >>> Server shows log: >>> Jul 14 10:24:28 lustreclient mountd[9868]: authenticated mount request >>> from 192.168.100.2:720 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs) >>> Jul 14 10:24:28 lustreclient mountd[9868]: can''t stat exported dir >>> /mnt/lustre_mail_fs: Permission denied >>> >>> Everything from a NFS service/config perspective appears to be >>> functional, and I can indeed export a local filesystem without errors... >>> >>> Additionally, I have no trouble using the mounted lustre filesystem, I >>> can even rsync data from the nfs client to the lustre file system on the >>> nfs server.. >>> >>> Any clues? Your knowledge and experience would be greatly appreciated! >>> See below for config specifics.. >>> >>> Thanks! >>> Billy Olson >>> >>> Server/Client Specifics: >>> Lustre Client/NFS Server: 192.168.100.77 >>> CentOS release 5.4 (Final) >>> lustre-client-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >>> lustre-client-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >>> Kernel: 2.6.18-164.11.1.el5 >>> >>> /etc/exports: >>> /mnt/lustre_mail_fs 192.168.100.0/24(ro,insecure) >>> I''ve tried various other options all the same outcome, these seemed like >>> the best to test with though.. >>> >>> [root at lustreclient ]# rpcinfo -p >>> program vers proto port >>> 100000 2 tcp 111 portmapper >>> 100000 2 udp 111 portmapper >>> 100024 1 udp 645 status >>> 100024 1 tcp 648 status >>> 100011 1 udp 702 rquotad >>> 100011 2 udp 702 rquotad >>> 100011 1 tcp 705 rquotad >>> 100011 2 tcp 705 rquotad >>> 100003 2 udp 2049 nfs >>> 100003 3 udp 2049 nfs >>> 100003 4 udp 2049 nfs >>> 100021 1 udp 37527 nlockmgr >>> 100021 3 udp 37527 nlockmgr >>> 100021 4 udp 37527 nlockmgr >>> 100003 2 tcp 2049 nfs >>> 100003 3 tcp 2049 nfs >>> 100003 4 tcp 2049 nfs >>> 100021 1 tcp 52555 nlockmgr >>> 100021 3 tcp 52555 nlockmgr >>> 100021 4 tcp 52555 nlockmgr >>> 100005 1 udp 716 mountd >>> 100005 1 tcp 719 mountd >>> 100005 2 udp 716 mountd >>> 100005 2 tcp 719 mountd >>> 100005 3 udp 716 mountd >>> 100005 3 tcp 719 mountd >>> >>> [root at lustreclient ]# exportfs -v -r >>> exporting 192.168.100.0/24:/mnt/lustre_mail_fs >>> >>> NFS Client: 192.168.100.2 >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > >
My only other suggestion is to dump the Lustre kernel debug log on the NFS server after a mount failure to see where/why it is getting the permission error. # lctl clear # (mount NFS client) # lctl dk /tmp/debug Then search through the logs for -2 errors (-EPERM). Cheers, Andreas On 2010-07-16, at 10:06, William Olson <lustre_admin at reachone.com> wrote:> On 7/15/2010 5:48 PM, Andreas Dilger wrote: >> On 2010-07-15, at 08:33, William Olson wrote: >> >>> Somebody, anybody? I''m sure it''s something fairly simple, but it >>> escapes me, assistance would be greatly appreciated! >>> >> I can''t comment, since my NFS re-exporting (to a MacOS client) is working fine. Hopefully soon I can stop doing that and use the native Mac client, but I don''t think my wife is willing to use alpha software yet. >> >> > LOL, yeah. Understood. >> Have you restarted the NFS server after changing the exports? >> > Many many times after every change.. >> Are you sure the permissions are correct? >> > As stated below all processes that I''ve tested have full access and no permissions issues when interacting with the lustre file system directly.. Accepting for the mountd while attempting to stat the lustre fs during an NFS client connection.. >> Have you verified that exporting a non-lustre filesystem from this server works? >> > Indeed, as stated below I have done so. >> My /etc/exports is: >> /myth 192.168.10.160(rw,async,insecure) >> > I may not have tried the async option, I''ll give that a shot really quick just for kicks.. ;) > Nope, same result still... hrm.. > > Thanks for Responding at least Andreas! > > I''m convinced that this is more of a lustre centric issue than an NFS issue, must be some interaction either on the kernel level or otherwise that''s running into a permissions block somewhere.. > >>>> Hello All, >>>> >>>> Having trouble exporting a lustre mount with NFS.. >>>> >>>> When mounting the NFS export on client: >>>> [root at nfsclient /]# mount 192.168.100.77:/mnt/lustre_mail_fs >>>> /mnt/lustre (have tried nearly every possible switch that I can think >>>> of, don''t think it''s related to "how" I attempt to mount) >>>> mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by >>>> server: Permission denied >>>> >>>> Server shows log: >>>> Jul 14 10:24:28 lustreclient mountd[9868]: authenticated mount request >>>> from 192.168.100.2:720 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs) >>>> Jul 14 10:24:28 lustreclient mountd[9868]: can''t stat exported dir >>>> /mnt/lustre_mail_fs: Permission denied >>>> >>>> Everything from a NFS service/config perspective appears to be >>>> functional, and I can indeed export a local filesystem without errors... >>>> >>>> Additionally, I have no trouble using the mounted lustre filesystem, I >>>> can even rsync data from the nfs client to the lustre file system on the >>>> nfs server.. >>>> >>>> Any clues? Your knowledge and experience would be greatly appreciated! >>>> See below for config specifics.. >>>> >>>> Thanks! >>>> Billy Olson >>>> >>>> Server/Client Specifics: >>>> Lustre Client/NFS Server: 192.168.100.77 >>>> CentOS release 5.4 (Final) >>>> lustre-client-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >>>> lustre-client-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >>>> Kernel: 2.6.18-164.11.1.el5 >>>> >>>> /etc/exports: >>>> /mnt/lustre_mail_fs 192.168.100.0/24(ro,insecure) >>>> I''ve tried various other options all the same outcome, these seemed like >>>> the best to test with though.. >>>> >>>> [root at lustreclient ]# rpcinfo -p >>>> program vers proto port >>>> 100000 2 tcp 111 portmapper >>>> 100000 2 udp 111 portmapper >>>> 100024 1 udp 645 status >>>> 100024 1 tcp 648 status >>>> 100011 1 udp 702 rquotad >>>> 100011 2 udp 702 rquotad >>>> 100011 1 tcp 705 rquotad >>>> 100011 2 tcp 705 rquotad >>>> 100003 2 udp 2049 nfs >>>> 100003 3 udp 2049 nfs >>>> 100003 4 udp 2049 nfs >>>> 100021 1 udp 37527 nlockmgr >>>> 100021 3 udp 37527 nlockmgr >>>> 100021 4 udp 37527 nlockmgr >>>> 100003 2 tcp 2049 nfs >>>> 100003 3 tcp 2049 nfs >>>> 100003 4 tcp 2049 nfs >>>> 100021 1 tcp 52555 nlockmgr >>>> 100021 3 tcp 52555 nlockmgr >>>> 100021 4 tcp 52555 nlockmgr >>>> 100005 1 udp 716 mountd >>>> 100005 1 tcp 719 mountd >>>> 100005 2 udp 716 mountd >>>> 100005 2 tcp 719 mountd >>>> 100005 3 udp 716 mountd >>>> 100005 3 tcp 719 mountd >>>> >>>> [root at lustreclient ]# exportfs -v -r >>>> exporting 192.168.100.0/24:/mnt/lustre_mail_fs >>>> >>>> NFS Client: 192.168.100.2 >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >>>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >
Without more information about the server error messages and exact nfs configuration, not sure anyone can help more than this. A common problem with Lustre NFS exports, one that isn''t due to normal NFS/configuration issues, is getting error -43 when the mds did not have the client''s IDs in its /etc/passwd and /etc/group files. Dumb question, but have you checked the permissions on the NFS server''s Lustre mount point (before/after Lustre is mounted), and exported a non-Lustre directory successfully? Kevin Andreas Dilger wrote:> My only other suggestion is to dump the Lustre kernel debug log on the NFS server after a mount failure to see where/why it is getting the permission error. > > # lctl clear > # (mount NFS client) > # lctl dk /tmp/debug > > Then search through the logs for -2 errors (-EPERM). > > Cheers, Andreas > > On 2010-07-16, at 10:06, William Olson <lustre_admin at reachone.com> wrote: > > >> On 7/15/2010 5:48 PM, Andreas Dilger wrote: >> >>> On 2010-07-15, at 08:33, William Olson wrote: >>> >>> >>>> Somebody, anybody? I''m sure it''s something fairly simple, but it >>>> escapes me, assistance would be greatly appreciated! >>>>
On 2010-07-16, at 17:35, Kevin Van Maren wrote:> Without more information about the server error messages and exact nfs > configuration, not sure anyone can help more than this. A common > problem with Lustre NFS exports, one that isn''t due to normal > NFS/configuration issues, is getting error -43 when the mds did not have > the client''s IDs in its /etc/passwd and /etc/group files. > > Dumb question, but have you checked the permissions on the NFS server''s > Lustre mount point (before/after Lustre is mounted), and exported a > non-Lustre directory successfully?This is covered earlier in the thread.> Andreas Dilger wrote: >> My only other suggestion is to dump the Lustre kernel debug log on the NFS server after a mount failure to see where/why it is getting the permission error. >> >> # lctl clear >> # (mount NFS client) >> # lctl dk /tmp/debug >> >> Then search through the logs for -2 errors (-EPERM).Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
On 2010-07-16, at 16:55, William Olson wrote:> On 7/16/2010 9:16 AM, Andreas Dilger wrote: >> My only other suggestion is to dump the Lustre kernel debug log on the NFS server after a mount failure to see where/why it is getting the permission error. >> >> # lctl clear >> # (mount NFS client) >> # lctl dk /tmp/debug >> > > > [root at lustreclient ~]# lctl clear > > [root at NFSclient ~]# mount 192.168.100.77:/mnt/lustre_mail_fs /mnt/lustre > mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by server: Permission denied > > Jul 16 15:52:51 lustreclient mountd[14552]: authenticated mount request from 192.168.100.2:954 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs) > Jul 16 15:52:51 lustreclient mountd[14552]: can''t stat exported dir /mnt/lustre_mail_fs: Permission denied > > [root at lustreclient ~]# lctl dk /tmp/debug > Debug log: 0 lines, 0 kept, 0 dropped, 0 bad. >Hmm, you must have a low debug level. Please try enabling full debug during the mount: # lctl clear # DBGSAVE=$(lctl get_param -n debug) # lctl set_param debug=-1 # mount ... # lctl dk /tmp/debug # lctl set_param debug="$DBGSAVE">> Then search through the logs for -2 errors (-EPERM). >>>> Cheers, Andreas >> >> On 2010-07-16, at 10:06, William Olson<lustre_admin at reachone.com> wrote: >> >> >>> On 7/15/2010 5:48 PM, Andreas Dilger wrote: >>> >>>> On 2010-07-15, at 08:33, William Olson wrote: >>>> >>>> >>>>> Somebody, anybody? I''m sure it''s something fairly simple, but it >>>>> escapes me, assistance would be greatly appreciated! >>>>> >>>>> >>>> I can''t comment, since my NFS re-exporting (to a MacOS client) is working fine. Hopefully soon I can stop doing that and use the native Mac client, but I don''t think my wife is willing to use alpha software yet. >>>> >>>> >>>> >>> LOL, yeah. Understood. >>> >>>> Have you restarted the NFS server after changing the exports? >>>> >>>> >>> Many many times after every change.. >>> >>>> Are you sure the permissions are correct? >>>> >>>> >>> As stated below all processes that I''ve tested have full access and no permissions issues when interacting with the lustre file system directly.. Accepting for the mountd while attempting to stat the lustre fs during an NFS client connection.. >>> >>>> Have you verified that exporting a non-lustre filesystem from this server works? >>>> >>>> >>> Indeed, as stated below I have done so. >>> >>>> My /etc/exports is: >>>> /myth 192.168.10.160(rw,async,insecure) >>>> >>>> >>> I may not have tried the async option, I''ll give that a shot really quick just for kicks.. ;) >>> Nope, same result still... hrm.. >>> >>> Thanks for Responding at least Andreas! >>> >>> I''m convinced that this is more of a lustre centric issue than an NFS issue, must be some interaction either on the kernel level or otherwise that''s running into a permissions block somewhere.. >>> >>> >>>>>> Hello All, >>>>>> >>>>>> Having trouble exporting a lustre mount with NFS.. >>>>>> >>>>>> When mounting the NFS export on client: >>>>>> [root at nfsclient /]# mount 192.168.100.77:/mnt/lustre_mail_fs >>>>>> /mnt/lustre (have tried nearly every possible switch that I can think >>>>>> of, don''t think it''s related to "how" I attempt to mount) >>>>>> mount: 192.168.100.77:/mnt/lustre_mail_fs failed, reason given by >>>>>> server: Permission denied >>>>>> >>>>>> Server shows log: >>>>>> Jul 14 10:24:28 lustreclient mountd[9868]: authenticated mount request >>>>>> from 192.168.100.2:720 for /mnt/lustre_mail_fs (/mnt/lustre_mail_fs) >>>>>> Jul 14 10:24:28 lustreclient mountd[9868]: can''t stat exported dir >>>>>> /mnt/lustre_mail_fs: Permission denied >>>>>> >>>>>> Everything from a NFS service/config perspective appears to be >>>>>> functional, and I can indeed export a local filesystem without errors... >>>>>> >>>>>> Additionally, I have no trouble using the mounted lustre filesystem, I >>>>>> can even rsync data from the nfs client to the lustre file system on the >>>>>> nfs server.. >>>>>> >>>>>> Any clues? Your knowledge and experience would be greatly appreciated! >>>>>> See below for config specifics.. >>>>>> >>>>>> Thanks! >>>>>> Billy Olson >>>>>> >>>>>> Server/Client Specifics: >>>>>> Lustre Client/NFS Server: 192.168.100.77 >>>>>> CentOS release 5.4 (Final) >>>>>> lustre-client-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >>>>>> lustre-client-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3 >>>>>> Kernel: 2.6.18-164.11.1.el5 >>>>>> >>>>>> /etc/exports: >>>>>> /mnt/lustre_mail_fs 192.168.100.0/24(ro,insecure) >>>>>> I''ve tried various other options all the same outcome, these seemed like >>>>>> the best to test with though.. >>>>>> >>>>>> [root at lustreclient ]# rpcinfo -p >>>>>> program vers proto port >>>>>> 100000 2 tcp 111 portmapper >>>>>> 100000 2 udp 111 portmapper >>>>>> 100024 1 udp 645 status >>>>>> 100024 1 tcp 648 status >>>>>> 100011 1 udp 702 rquotad >>>>>> 100011 2 udp 702 rquotad >>>>>> 100011 1 tcp 705 rquotad >>>>>> 100011 2 tcp 705 rquotad >>>>>> 100003 2 udp 2049 nfs >>>>>> 100003 3 udp 2049 nfs >>>>>> 100003 4 udp 2049 nfs >>>>>> 100021 1 udp 37527 nlockmgr >>>>>> 100021 3 udp 37527 nlockmgr >>>>>> 100021 4 udp 37527 nlockmgr >>>>>> 100003 2 tcp 2049 nfs >>>>>> 100003 3 tcp 2049 nfs >>>>>> 100003 4 tcp 2049 nfs >>>>>> 100021 1 tcp 52555 nlockmgr >>>>>> 100021 3 tcp 52555 nlockmgr >>>>>> 100021 4 tcp 52555 nlockmgr >>>>>> 100005 1 udp 716 mountd >>>>>> 100005 1 tcp 719 mountd >>>>>> 100005 2 udp 716 mountd >>>>>> 100005 2 tcp 719 mountd >>>>>> 100005 3 udp 716 mountd >>>>>> 100005 3 tcp 719 mountd >>>>>> >>>>>> [root at lustreclient ]# exportfs -v -r >>>>>> exporting 192.168.100.0/24:/mnt/lustre_mail_fs >>>>>> >>>>>> NFS Client: 192.168.100.2 >>>>>> >>>>>> _______________________________________________ >>>>>> Lustre-discuss mailing list >>>>>> Lustre-discuss at lists.lustre.org >>>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>>> >>>>> >>>> Cheers, Andreas >>>> -- >>>> Andreas Dilger >>>> Lustre Technical Lead >>>> Oracle Corporation Canada Inc. >>>> >>>> >>>> >>> >Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
On 7/16/2010 4:35 PM, Kevin Van Maren wrote:> Without more information about the server error messages and exact nfs > configuration, not sure anyone can help more than this. A common > problem with Lustre NFS exports, one that isn''t due to normal > NFS/configuration issues, is getting error -43 when the mds did not > have the client''s IDs in its /etc/passwd and /etc/group files. >No -43 errors, and I have group_upcall turned off anyhow( I think that''s what the -43 corresponds too.. ) I''m not having any issues with permissions when using the lustre mount locally, or when rsyncing data from another client to the server hosting the lustre fs.> Dumb question, but have you checked the permissions on the NFS > server''s Lustre mount point (before/after Lustre is mounted), and > exported a non-Lustre directory successfully? >Lustre mounted: drwxrwxrwx 29 root root 4.0K Jul 12 17:03 lustre_mail_fs Lustre not mounted: drwxrwxrwx 2 root root 4.0K Jun 10 13:26 lustre_mail_fs I have no trouble exporting a local fs..> Kevin > > > Andreas Dilger wrote: >> My only other suggestion is to dump the Lustre kernel debug log on >> the NFS server after a mount failure to see where/why it is getting >> the permission error. >> # lctl clear >> # (mount NFS client) >> # lctl dk /tmp/debug >> >> Then search through the logs for -2 errors (-EPERM). >> >> Cheers, Andreas >> >> On 2010-07-16, at 10:06, William Olson <lustre_admin at reachone.com> >> wrote: >> >>> On 7/15/2010 5:48 PM, Andreas Dilger wrote: >>>> On 2010-07-15, at 08:33, William Olson wrote: >>>>> Somebody, anybody? I''m sure it''s something fairly simple, but it >>>>> escapes me, assistance would be greatly appreciated! >
On 7/16/2010 4:50 PM, Andreas Dilger wrote:> > Hmm, you must have a low debug level. Please try enabling full debug during the mount: > > # lctl clear > # DBGSAVE=$(lctl get_param -n debug) > # lctl set_param debug=-1 > # mount ... > # lctl dk /tmp/debug > # lctl set_param debug="$DBGSAVE" > > >>> Then search through the logs for -2 errors (-EPERM). >>> >>> >Well that improved the debug level, but didn''t reveal any -2 errors.. In fact I can''t seem to find a line with an error in it... Is there a specific verbiage used on error lines that I can grep for? 90% is "Process entered" or "Process leaving"... Thanks Again!
On 2010-07-16, at 18:09, William Olson wrote:> On 7/16/2010 4:50 PM, Andreas Dilger wrote: >>>> Then search through the logs for -2 errors (-EPERM). >>>> >>>> >> > > Well that improved the debug level, but didn''t reveal any -2 errors.. In fact I can''t seem to find a line with an error in it... Is there a specific verbiage used on error lines that I can grep for? 90% is "Process entered" or "Process leaving"...You could try "strace -f" on the mount process, to see which syscall is failing. It may be failing with something before it gets to Lustre. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
On 7/16/2010 5:12 PM, Andreas Dilger wrote:> >> Well that improved the debug level, but didn''t reveal any -2 errors.. In fact I can''t seem to find a line with an error in it... Is there a specific verbiage used on error lines that I can grep for? 90% is "Process entered" or "Process leaving"... >> > You could try "strace -f" on the mount process, to see which syscall is failing. It may be failing with something before it gets to Lustre. >Results of strace below: [root at lustreclient mnt]# strace -f -p 15964 Process 15964 attached - interrupt to quit select(1024, [3 4 5 6 7], NULL, NULL, NULL) = 1 (in [6]) recvmsg(6, {msg_name(16)={sa_family=AF_INET, sin_port=htons(42303), sin_addr=inet_addr("192.168.100.2")}, msg_iov(1)=[{"O^\240\350\0\0\0\0\0\0\0\2\0\1\206\245\0\0\0\3\0\0\0\0\0\0\0\0\0\0\0\0"..., 8800}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 40 stat("/etc/hosts.allow", {st_mode=S_IFREG|0644, st_size=189, ...}) = 0 stat("/etc/hosts.deny", {st_mode=S_IFREG|0644, st_size=347, ...}) = 0 sendmsg(6, {msg_name(16)={sa_family=AF_INET, sin_port=htons(42303), sin_addr=inet_addr("192.168.100.2")}, msg_iov(1)=[{"O^\240\350\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 24}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 24 select(1024, [3 4 5 6 7], NULL, NULL, NULL) = 1 (in [6]) recvmsg(6, {msg_name(16)={sa_family=AF_INET, sin_port=htons(1002), sin_addr=inet_addr("192.168.100.2")}, msg_iov(1)=[{"{\243\34\22\0\0\0\0\0\0\0\2\0\1\206\245\0\0\0\3\0\0\0\1\0\0\0\1\0\0\0D"..., 8800}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 132 stat("/etc/hosts.allow", {st_mode=S_IFREG|0644, st_size=189, ...}) = 0 stat("/etc/hosts.deny", {st_mode=S_IFREG|0644, st_size=347, ...}) = 0 open("/var/lib/nfs/etab", O_RDONLY) = 10 fstat(10, {st_mode=S_IFREG|0644, st_size=184, ...}) = 0 close(10) = 0 lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES (Permission denied) stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 sendto(9, "<29>Jul 16 17:22:06 mountd[15964"..., 132, MSG_NOSIGNAL, NULL, 0) = 132 stat("/mnt/lustre_mail_fs", 0x7fff4bd4b410) = -1 EACCES (Permission denied) stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 sendto(9, "<28>Jul 16 17:22:06 mountd[15964"..., 97, MSG_NOSIGNAL, NULL, 0) = 97 sendmsg(6, {msg_name(16)={sa_family=AF_INET, sin_port=htons(1002), sin_addr=inet_addr("192.168.100.2")}, msg_iov(1)=[{"{\243\34\22\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\r", 28}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 28 select(1024, [3 4 5 6 7], NULL, NULL, NULL
Looks like a problem with your mount point. What are the permissions on the client directory? On Jul 16, 2010, at 6:23 PM, William Olson <lustre_admin at reachone.com> wrote:> On 7/16/2010 5:12 PM, Andreas Dilger wrote: >> >>> Well that improved the debug level, but didn''t reveal any -2 >>> errors.. In fact I can''t seem to find a line with an error in >>> it... Is there a specific verbiage used on error lines that I can >>> grep for? 90% is "Process entered" or "Process leaving"... >>> >> You could try "strace -f" on the mount process, to see which >> syscall is failing. It may be failing with something before it >> gets to Lustre. >> > Results of strace below: > > [root at lustreclient mnt]# strace -f -p 15964 > Process 15964 attached - interrupt to quit> lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 > lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES (Permission > denied) > stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0
On 7/16/2010 5:41 PM, Kevin Van Maren wrote:> Looks like a problem with your mount point. What are the permissions > on the client directory? >NFSServer/Lustre Client Lustre mounted: drwxrwxrwx 29 root root 4.0K Jul 12 17:03 lustre_mail_fs Lustre not mounted: drwxrwxrwx 2 root root 4.0K Jun 10 13:26 lustre_mail_fs NFSClient mount dir: drwxrwxrwx 2 root root 4.0K Jul 12 15:09 lustre> > On Jul 16, 2010, at 6:23 PM, William Olson <lustre_admin at reachone.com> > wrote: > >> On 7/16/2010 5:12 PM, Andreas Dilger wrote: >>> >>>> Well that improved the debug level, but didn''t reveal any -2 >>>> errors.. In fact I can''t seem to find a line with an error in >>>> it... Is there a specific verbiage used on error lines that I can >>>> grep for? 90% is "Process entered" or "Process leaving"... >>>> >>> You could try "strace -f" on the mount process, to see which syscall >>> is failing. It may be failing with something before it gets to Lustre. >>> >> Results of strace below: >> >> [root at lustreclient mnt]# strace -f -p 15964 >> Process 15964 attached - interrupt to quit > > > >> lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >> lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES (Permission >> denied) >> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 > >
But the client is doing a lstat on /mnt/lustre_mail_fs, not /mnt/ lustre -- what is the mount command again? On Jul 16, 2010, at 6:50 PM, William Olson <lustre_admin at reachone.com> wrote:> On 7/16/2010 5:41 PM, Kevin Van Maren wrote: >> Looks like a problem with your mount point. What are the >> permissions on the client directory? >> > NFSServer/Lustre Client > > Lustre mounted: > drwxrwxrwx 29 root root 4.0K Jul 12 17:03 lustre_mail_fs > Lustre not mounted: > drwxrwxrwx 2 root root 4.0K Jun 10 13:26 lustre_mail_fs > > NFSClient mount dir: > drwxrwxrwx 2 root root 4.0K Jul 12 15:09 lustre > >> >> On Jul 16, 2010, at 6:23 PM, William Olson >> <lustre_admin at reachone.com> wrote: >> >>> On 7/16/2010 5:12 PM, Andreas Dilger wrote: >>>> >>>>> Well that improved the debug level, but didn''t reveal any -2 >>>>> errors.. In fact I can''t seem to find a line with an error in >>>>> it... Is there a specific verbiage used on error lines that I >>>>> can grep for? 90% is "Process entered" or "Process leaving"... >>>>> >>>> You could try "strace -f" on the mount process, to see which >>>> syscall is failing. It may be failing with something before it >>>> gets to Lustre. >>>> >>> Results of strace below: >>> >>> [root at lustreclient mnt]# strace -f -p 15964 >>> Process 15964 attached - interrupt to quit >> >> >> >>> lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>> lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES >>> (Permission denied) >>> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) >>> = 0 >> >> >
On 7/16/2010 6:16 PM, Kevin Van Maren wrote:> But the client is doing a lstat on /mnt/lustre_mail_fs, not > /mnt/lustre -- what is the mount command again? >I think my server names are confusing the situation.. Lustre client = NFS server Lustre is mounted in /mnt/lustre_mail_fs /mnt/lustre_mail_fs is exported with NFS NFS client is attempting to mount the /mnt/lustre_mail_fs export to it''s /mnt/lustre directory.. mountd(on the NFSserver/lustre client) fails to stat the correctly mounted and fully operational /mnt/lustre_mail_fs, during an NFS client connection attempt. The NFS client authenticates properly according to the logs, it''s only when mountd attempts to stat the lustre fs that problems arise.. Again, for clarity, I can successfully export and mount any other directory from this same machine, to the same client..> On Jul 16, 2010, at 6:50 PM, William Olson <lustre_admin at reachone.com> > wrote: > >> On 7/16/2010 5:41 PM, Kevin Van Maren wrote: >>> Looks like a problem with your mount point. What are the >>> permissions on the client directory? >>> >> NFSServer/Lustre Client >> >> Lustre mounted: >> drwxrwxrwx 29 root root 4.0K Jul 12 17:03 lustre_mail_fs >> Lustre not mounted: >> drwxrwxrwx 2 root root 4.0K Jun 10 13:26 lustre_mail_fs >> >> NFSClient mount dir: >> drwxrwxrwx 2 root root 4.0K Jul 12 15:09 lustre >> >>> >>> On Jul 16, 2010, at 6:23 PM, William Olson >>> <lustre_admin at reachone.com> wrote: >>> >>>> On 7/16/2010 5:12 PM, Andreas Dilger wrote: >>>>> >>>>>> Well that improved the debug level, but didn''t reveal any -2 >>>>>> errors.. In fact I can''t seem to find a line with an error in >>>>>> it... Is there a specific verbiage used on error lines that I can >>>>>> grep for? 90% is "Process entered" or "Process leaving"... >>>>>> >>>>> You could try "strace -f" on the mount process, to see which >>>>> syscall is failing. It may be failing with something before it >>>>> gets to Lustre. >>>>> >>>> Results of strace below: >>>> >>>> [root at lustreclient mnt]# strace -f -p 15964 >>>> Process 15964 attached - interrupt to quit >>> >>> >>> >>>> lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES >>>> (Permission denied) >>>> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 >>> >>> >>
schroete at iup.physik.uni-bremen.de
2010-Jul-19 16:12 UTC
[Lustre-discuss] NFS Export Issues
Zitat von William Olson <lustre_admin at reachone.com>: We had such a problem when first experimenting with a lustre reexport via nfs. In our case the "permission denied" was a problem that the "user(owner)" of the reexporting machine (directory) wasn''t known on the mds. Regards Heiko> On 7/16/2010 6:16 PM, Kevin Van Maren wrote: >> But the client is doing a lstat on /mnt/lustre_mail_fs, not >> /mnt/lustre -- what is the mount command again? >> > > I think my server names are confusing the situation.. > > Lustre client = NFS server > Lustre is mounted in /mnt/lustre_mail_fs > /mnt/lustre_mail_fs is exported with NFS > > NFS client is attempting to mount the /mnt/lustre_mail_fs export to it''s > /mnt/lustre directory.. > > mountd(on the NFSserver/lustre client) fails to stat the correctly > mounted and fully operational /mnt/lustre_mail_fs, during an NFS client > connection attempt. > The NFS client authenticates properly according to the logs, it''s only > when mountd attempts to stat the lustre fs that problems arise.. > > Again, for clarity, I can successfully export and mount any other > directory from this same machine, to the same client.. > >> On Jul 16, 2010, at 6:50 PM, William Olson <lustre_admin at reachone.com> >> wrote: >> >>> On 7/16/2010 5:41 PM, Kevin Van Maren wrote: >>>> Looks like a problem with your mount point. What are the >>>> permissions on the client directory? >>>> >>> NFSServer/Lustre Client >>> >>> Lustre mounted: >>> drwxrwxrwx 29 root root 4.0K Jul 12 17:03 lustre_mail_fs >>> Lustre not mounted: >>> drwxrwxrwx 2 root root 4.0K Jun 10 13:26 lustre_mail_fs >>> >>> NFSClient mount dir: >>> drwxrwxrwx 2 root root 4.0K Jul 12 15:09 lustre >>> >>>> >>>> On Jul 16, 2010, at 6:23 PM, William Olson >>>> <lustre_admin at reachone.com> wrote: >>>> >>>>> On 7/16/2010 5:12 PM, Andreas Dilger wrote: >>>>>> >>>>>>> Well that improved the debug level, but didn''t reveal any -2 >>>>>>> errors.. In fact I can''t seem to find a line with an error in >>>>>>> it... Is there a specific verbiage used on error lines that I can >>>>>>> grep for? 90% is "Process entered" or "Process leaving"... >>>>>>> >>>>>> You could try "strace -f" on the mount process, to see which >>>>>> syscall is failing. It may be failing with something before it >>>>>> gets to Lustre. >>>>>> >>>>> Results of strace below: >>>>> >>>>> [root at lustreclient mnt]# strace -f -p 15964 >>>>> Process 15964 attached - interrupt to quit >>>> >>>> >>>> >>>>> lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES >>>>> (Permission denied) >>>>> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2875, ...}) = 0 >>>> >>>> >>> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
> As far as i remember we had to explicitly align the "mounting" user uids and gids. So the mounting uid:gid must be known (/etc/passwd and groups i think) on the mds and allowed to mount stuff. > > Could it be a "root squash" problem by accident ? > > >I''ve tried it explicitly with the no_root_squash option and it still behaves the same way.. What I find really frustrating is that if I unmount lustre, I can mount the same nfs export, no problems. As soon as lustre is mounted to that directory I can no longer mount that nfs export. I don''t understand where it''s failing.. Thanks for all your help so far, any more ideas?
On Tuesday, July 20, 2010, William Olson wrote:> > As far as i remember we had to explicitly align the "mounting" user uids > > and gids. So the mounting uid:gid must be known (/etc/passwd and groups > > i think) on the mds and allowed to mount stuff. > > > > Could it be a "root squash" problem by accident ? > > I''ve tried it explicitly with the no_root_squash option and it still > behaves the same way.. > What I find really frustrating is that if I unmount lustre, I can mount > the same nfs export, no problems. As soon as lustre is mounted to that > directory I can no longer mount that nfs export. > I don''t understand where it''s failing.. > > Thanks for all your help so far, any more ideas?Have you tried to add "fsid=xxx" to your exports line? I think with recently Lustre versions (I don''t remember the implementation details) it should not be required any more and so it should not with recent nfs-utils and until-linux (the filesystem uuid is automatically used with those, instead of device major/minor as fsid), but maybe both type of workarounds conflict on your system? You also might consider to simply use unfs3, although performance will be limited to about 120MB/s, as unfs3 is only single threaded. It also does not support NFS locks. If it still does not work out, you should enabled lustre debugging, nfs debugging and you probably should use wireshark to see what it going on. Hope it helps, Bernd -- Bernd Schubert DataDirect Networks
>> I''ve tried it explicitly with the no_root_squash option and it still >> behaves the same way.. >> What I find really frustrating is that if I unmount lustre, I can mount >> the same nfs export, no problems. As soon as lustre is mounted to that >> directory I can no longer mount that nfs export. >> I don''t understand where it''s failing.. >> >> Thanks for all your help so far, any more ideas? >> > Have you tried to add "fsid=xxx" to your exports line? I think with recently > Lustre versions (I don''t remember the implementation details) it should not be > required any more and so it should not with recent nfs-utils and until-linux > (the filesystem uuid is automatically used with those, instead of device > major/minor as fsid), but maybe both type of workarounds conflict on your > system? > >Yeah, I''ve tried it with fsid set and unset, same result either way.> You also might consider to simply use unfs3, although performance will be > limited to about 120MB/s, as unfs3 is only single threaded. It also does not > support NFS locks. > >I would rather not if I don''t have to.. While I only need the nfs to work for a short time I can''t see a single threaded solution working very well, this will be the backend filesystem for a production email cluster, we''re transitioning from another solution to lustre and need the nfs to play it''s part while we move from one to the other.. They both have kernel version requirements so I can only setup one file system directly on the cluster machines at a time unless one of them is mounted with NFS..> If it still does not work out, you should enabled lustre debugging, nfs > debugging and you probably should use wireshark to see what it going on. > >This may indeed be my next step in the troubleshooting process.. I was hoping it was something obvious that somebody had run into before but it seems this is a "special case".. Oh well, time to dig into the dirty details I guess.. As always though, if anybody has any other ideas or can suggest a better way to achieve what I''m trying to do, please let me know.> Hope it helps, > Bernd > >
On Fri, Jul 16, 2010 at 05:23:16PM -0700, William Olson wrote:> [root at lustreclient mnt]# strace -f -p 15964(...)> lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES (Permission denied)When it comes to inexplicable permission problems, have you checked if SELinux is turned off on the NFS server? Regards, Daniel.
Isn''t this simply because you are doing this as root? If you did this as a user known to Lustre you should be able to do this. We have seen this in our NFS re-exported Lustre shares. Thanks. Suvendra. On 7/21/10 4:12 AM, "Daniel Kobras" <kobras at linux.de> wrote:> On Fri, Jul 16, 2010 at 05:23:16PM -0700, William Olson wrote: >> [root at lustreclient mnt]# strace -f -p 15964 > (...) >> lstat("/mnt/lustre_mail_fs", 0x7fff4bd4b2b0) = -1 EACCES (Permission denied) > > When it comes to inexplicable permission problems, have you checked if > SELinux is turned off on the NFS server? > > Regards, > > Daniel. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> When it comes to inexplicable permission problems, have you checked if > SELinux is turned off on the NFS server? > >I knew if I was patient somebody would point out the simple answer and make me look like an idiot!! hahahaha THANK YOU!! So, set selinux into permissive mode, adjusted iptables(wasn''t part of the original problem, but I didn''t save my rules before rebooting) and guess what?.. It works. :) YAY! I think my sysadmin badge needs to be revoked for a day...> Regards, > > Daniel. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
If someone who is familiar with SELinux had the time, I''d be thrilled to find some way to exclude Lustre mountpoints from SELinux automatically, and then submit it upstream. Cheers, Andreas On 2010-07-21, at 8:24, William Olson <lustre_admin at reachone.com> wrote:> >> When it comes to inexplicable permission problems, have you checked if >> SELinux is turned off on the NFS server? >> >> > I knew if I was patient somebody would point out the simple answer and > make me look like an idiot!! hahahaha > > THANK YOU!! > > So, set selinux into permissive mode, adjusted iptables(wasn''t part of > the original problem, but I didn''t save my rules before rebooting) and > guess what?.. It works. :) > YAY! > > I think my sysadmin badge needs to be revoked for a day... >> Regards, >> >> Daniel. >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
After my installation of OS, the first thing is to turn off the SElinux, it seems no use, and often makes a lot of trouble... On Wed, Jul 21, 2010 at 10:24 PM, William Olson <lustre_admin at reachone.com> wrote:> >> When it comes to inexplicable permission problems, have you checked if >> SELinux is turned off on the NFS server? >> >> > I knew if I was patient somebody would point out the simple answer and > make me look like an idiot!! hahahaha > > THANK YOU!! > > So, set selinux into permissive mode, adjusted iptables(wasn''t part of > the original problem, but I didn''t save my rules before rebooting) and > guess what?.. It works. :) > YAY! > > I think my sysadmin badge needs to be revoked for a day... >> Regards, >> >> Daniel. >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On 7/22/2010 1:10 AM, Larry wrote:> After my installation of OS, the first thing is to turn off the > SElinux, it seems no use, and often makes a lot of trouble... >Larry, Yeah, I usually end up turning it off too but make it a point to only do so when it is actually in the way.. This time it totally slipped my mind..