Hello List, we re-export the file system via NFS for a couple of things. All the re-exporters are Red Hat 5.5 servers running kernel 2.6.18-194.17.1.el5 (patchless clients). We upgraded our Lustre system from 1.6.7 to 1.8.3.ddn3.3 last week. That seems to have introduced a problem. Since this upgrade, some of our NFS clients can''t mount the file system any more. They get a permission denied error. Others can mount the same file system just fine. The difference appears to be that the clients that can''t mount it try to use mountd version 1. Some investigation exposed the following behaviour: [root at pc030 ~]# mount -o udp -o mountvers=1 cs04r-sc-com99-01:/mnt/play01/tmp/tina/mounttest/ /tmp/mnttest/ [root at pc030 ~]# ls -l /tmp/mnttest/ total 0 ?--------- ? ? ? ? ? realdir I.e. when I use mountvers=1 I get this weird ''broken link'' type entry (I get the ''permission denied'' error if I try to mount anything below ''realdir'' in this example). Whereas when using mountvers=3: [root at pc030 ~]# mount -o udp -o mountvers=3 cs04r-sc-com99-01:/mnt/play01/tmp/tina/mounttest/ /tmp/mnttest/ [root at pc030 ~]# ls -l /tmp/mnttest/ total 4 drwxrwxr-x 3 root root 4096 Nov 12 10:04 realdir all is as it should be. This is only when NFS exporting our Lustre file system; when exporting a local file system instead all versions of mountd work fine. Anyone seen this before? Any ideas? Unfortunately the systems using the mountd version 1 are an embedded box so I''m not sure the way they mount can be changed. -- Tina Friedrich, Computer Systems Administrator, Diamond Light Source Ltd Diamond House, Harwell Science and Innovation Campus - 01235 77 8442 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
Hello Tina, On Friday, November 12, 2010, Tina Friedrich wrote:> Hello List, > > we re-export the file system via NFS for a couple of things. All the > re-exporters are Red Hat 5.5 servers running kernel 2.6.18-194.17.1.el5 > (patchless clients).that is your problem. You MUST use a patched version, or least a kernel with 8kB stack size. RHEL5 has 4kB by default, which is not sufficient and therefore in early 1.8 versions a patch landed that disallowed NFS exports. Cheers, Bernd -- Bernd Schubert DataDirect Networks
It does seem to allow NFS exports using RPC version 3 just fine though. It''s just the version 1 & 2 were it doesn''t work. But thanks, I''ll try the patched kernel. Tina On 12/11/10 12:46, Bernd Schubert wrote:> Hello Tina, > > On Friday, November 12, 2010, Tina Friedrich wrote: >> Hello List, >> >> we re-export the file system via NFS for a couple of things. All the >> re-exporters are Red Hat 5.5 servers running kernel 2.6.18-194.17.1.el5 >> (patchless clients). > > that is your problem. You MUST use a patched version, or least a kernel with > 8kB stack size. RHEL5 has 4kB by default, which is not sufficient and > therefore in early 1.8 versions a patch landed that disallowed NFS exports. > > > Cheers, > Bernd > > >-- Tina Friedrich, Computer Systems Administrator, Diamond Light Source Ltd Diamond House, Harwell Science and Innovation Campus - 01235 77 8442 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
Hello again, nope, running with / exporting from a server with the patched kernel running does not change this behaviour at all. mountvers=3 works, 1 and 2 don''t. Tina On 12/11/10 13:28, Tina Friedrich wrote:> It does seem to allow NFS exports using RPC version 3 just fine though. > It''s just the version 1& 2 were it doesn''t work. But thanks, I''ll try > the patched kernel. > > Tina > > On 12/11/10 12:46, Bernd Schubert wrote: >> Hello Tina, >> >> On Friday, November 12, 2010, Tina Friedrich wrote: >>> Hello List, >>> >>> we re-export the file system via NFS for a couple of things. All the >>> re-exporters are Red Hat 5.5 servers running kernel 2.6.18-194.17.1.el5 >>> (patchless clients). >> >> that is your problem. You MUST use a patched version, or least a kernel with >> 8kB stack size. RHEL5 has 4kB by default, which is not sufficient and >> therefore in early 1.8 versions a patch landed that disallowed NFS exports. >> >> >> Cheers, >> Bernd >> >> >> > >-- Tina Friedrich, Computer Systems Administrator, Diamond Light Source Ltd Diamond House, Harwell Science and Innovation Campus - 01235 77 8442 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
Hello Tina, On 11/12/2010 03:44 PM, Tina Friedrich wrote:> Hello again, > > nope, running with / exporting from a server with the patched kernel > running does not change this behaviour at all. mountvers=3 works, 1 and > 2 don''t.I can reproduce it, so NFSv2 support got broken. Which issue has higher priority, tar or NFSv2? Cheers, Bernd -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20101112/78580dec/attachment.bin
You''re asking questions! No chance we can upgrade the Lustre version again for the next couple of months anyway and it appears to be no problem to make the stupid embedded things use NFSv3. So tar, I would say. Tina On 12/11/10 16:22, Bernd Schubert wrote:> Hello Tina, > > On 11/12/2010 03:44 PM, Tina Friedrich wrote: >> Hello again, >> >> nope, running with / exporting from a server with the patched kernel >> running does not change this behaviour at all. mountvers=3 works, 1 and >> 2 don''t. > > I can reproduce it, so NFSv2 support got broken. Which issue has higher > priority, tar or NFSv2? > > > Cheers, > Bernd >-- Tina Friedrich, Computer Systems Administrator, Diamond Light Source Ltd Diamond House, Harwell Science and Innovation Campus - 01235 77 8442 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
Hi Tina, if i correctly remember, lustre inode identifier can''t correctly encoded in NFS fid v1, due NFS limits. NFS have too short structure to store FS id, generated from lustre client superblock. that is reason to have incorrect conversion from NFS fid to lustre inode identifier and nfsd can''t find corresponded lustre inode to get attributes. you can try to specify fsid option in mount command line, but not sure that is helps. On Nov 12, 2010, at 14:26, Tina Friedrich wrote:> Hello List, > > we re-export the file system via NFS for a couple of things. All the > re-exporters are Red Hat 5.5 servers running kernel 2.6.18-194.17.1.el5 > (patchless clients). > > We upgraded our Lustre system from 1.6.7 to 1.8.3.ddn3.3 last week. That > seems to have introduced a problem. Since this upgrade, some of our NFS > clients can''t mount the file system any more. They get a permission > denied error. Others can mount the same file system just fine. The > difference appears to be that the clients that can''t mount it try to use > mountd version 1. > ''-------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com
Bernd,
that is problem not related to stack size.
mountd should encode {sb->sb_dev, inode->i_no} in own structure, but for
mount v1 sb_dev too short to store full sb_dev (it is 16bit, but nfs fid v1 has
8 for it) bit but lustre client can generate that.
in that case nfsd got invalid NFS fid to client and broken to find correct inode
to get attributes and return -ENOENT. in that case ls -l will show "?"
instead of real inode attributes.
On Nov 12, 2010, at 15:46, Bernd Schubert wrote:
> Hello Tina,
>
> On Friday, November 12, 2010, Tina Friedrich wrote:
>> Hello List,
>>
>> we re-export the file system via NFS for a couple of things. All the
>> re-exporters are Red Hat 5.5 servers running kernel 2.6.18-194.17.1.el5
>> (patchless clients).
>
> that is your problem. You MUST use a patched version, or least a kernel
with
> 8kB stack size. RHEL5 has 4kB by default, which is not sufficient and
> therefore in early 1.8 versions a patch landed that disallowed NFS exports.
>
>
> Cheers,
> Bernd
>
>
>
> --
> Bernd Schubert
> DataDirect Networks
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss