dovecot - Mar 2010 - I stream read - stale NFS file handle (reboot of server)

In the old days NFS Shared Path had a static handle (ie a number), 
normal based on some number pulled out of the file system/inode.
To fix (well work around) a security issue, for about 10+ years now, 
when a NFS server reboots, it generates a new random handle for the NFS 
Share. (sever may generate a new random handle per mount request)

The NFS Stale Handle happens when the client is still using the old NFS 
handle, the only fix is a remount

Some NFSv3 server let you fix the handle id for NFS share so it does not 
change over reboots. (this is how NFS clustering works, sometimes you 
need to install cluster software if the exportfs/share command does not 
let you hard code the fsid by hand.

The other option is to use NFSv4 which is design to handle reboots by 
servers.

On Wed, 2010-03-17 at 01:59 +1100, Damon Atkins wrote:
> In the old days NFS Shared Path had a static handle (ie a number), 
> normal based on some number pulled out of the file system/inode.
> To fix (well work around) a security issue, for about 10+ years now, 
> when a NFS server reboots, it generates a new random handle for the NFS 
> Share. (sever may generate a new random handle per mount request)
> 
> The NFS Stale Handle happens when the client is still using the old NFS 
> handle, the only fix is a remount
But that applies only to file handles that were opened before reboot,
not to new fds opened after reboot, right? At least I can't see it
working any other way.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100316/fbee4719/attachment-0002.bin>

> To fix (well work around) a security issue, for about 10+ years now,
> when a NFS server reboots, it generates a new random handle for the
> NFS Share. (sever may generate a new random handle per mount
> request)
I don't concur.

NFS is stateless and designed to survive server reboots (why would you have stad
otherwise?). What you do is inode randomization on the file system backing the
NFS export.

You get those stale handles when someone on the client has a file on the mount
open and some other way the file gets deleted (by another client or right on the
server). Since NFS is stateless, the server knows nothing about the file being
open. If the file was open on a local file system, the kernel wouldn't
actually free the inode because there's still a reference (albeit with no
directory entry) on it. But NFS lacks this reference. So clients can work around
this by converting unlinks to renames to .nfsXXXX names and sending an unlink to
the server only on the last local unlink. Of course, this works only with one
client.

Just to make sure I rememberer all this correctly, I just confirmed with The
BooK: See the fourth paragraph of "The NFS Protocol" in Section 9.2 of
McKusick/Bostic/Karels/Quarterman (in my edition it's on page 317)

NFS Security 101 for NFSv2 and v3 (NOT NFSv4 a long time ago I was part 
of the discussion group for NFSv4 spec the short comings of v2 and v3 
have been fixed)

SRV: Server Exports File System /abc/123 access only to host=xyz.domain.com
XYZ: Client Mount mount's SRV:/abc/123
SRV: "mountd" gets a request from SRV check access list, and if the 
client is allowed access, returns File Handle for the top of the mount 
point (I will call this the FileSystemHandle, this changes after every 
reboot)
XYZ: Client talks to the SRV NFS Server "nfsd" using FileSystemHandle 
(which represents /abc/123), To find a FileHandle for a file the client 
sends the NFS server  the  FileSystemHandle, then finds the next File 
Handle (/mydir) and the next FileHandle (myfile) and the client has 
found the FileHandle for /abc/123/mydir/myfile

NFSD (v2/v3) is stateless other than the information provided by mountd 
(mount requests) and lockd (file locking). When you share/export a file 
system, as part of this a FileSystemHandle is generated and stored  
somewhere for /abc/123 most likely store in the kernel.

SRV: Reboots, a new FileSystemHandle is allocated
XYZ: Reports Stale File Handle for everything (if not most things) which 
was mounted from SRV or the mount point on the client does not respond.

A long time ago FileSystemHandle would stay the same between reboots and 
you wouldn't get this problem other than on an individual file.

To handle deleted files which are in use by NFS clients some servers 
rename them to .nfs* because if one client deleted and other clients 
where accessing the file then they would get Stale NFS handle.  Once in 
a while a NFS server will do find $dir -type f -name .nfs\* -mtime +7 
-mount -exec rm -f {} \; to clean up.


NFSv4 does not use mountd or lockd.
My advice to anyone on old versions of NFS upgrade to NFSv4 as soon as 
possible and implement KerbV5 for improved authentication and disallow 
NFSv2/3 access.

If you do not get the Stale File Handle error when the server reboots, 
it most likely means the FileSystemHandle is not changing between 
reboots, but then you may have more security issues. Software which 
Clusters NFS Servers ensures that the node which takes over, uses the 
same FileSystemHandle (which may only change if the whole Cluster is 
shutdown)

Cheers
Damon.