NFS server (b39): bash-3.00# zfs get quota nfs-s5-s8/d5201 nfs-s5-p0/d5110 NAME PROPERTY VALUE SOURCE nfs-s5-p0/d5110 quota 600G local nfs-s5-s8/d5201 quota 600G local bash-3.00# bash-3.00# df -h | egrep "d5201|d5110" nfs-s5-p0/d5110 600G 527G 73G 88% /nfs-s5-p0/d5110 nfs-s5-s8/d5201 600G 314G 269G 54% /nfs-s5-s8/d5201 bash-3.00# NFS client (S10U1 + patches, NFSv3 mount over TCP): bash-3.00# df -h | egrep "d5201|d5110" NFS-srv:/nfs-s5-p0/d5110 600G 527G 73G 88% /opt/d5110 NFS-srv:/nfs-s5-s8/d5201 583G 314G 269G 54% /opt/d5201 bash-3.00# Well why I get 583GB size for d5201 on NFS client? ps. maybe I''m tired and missiong something really obvious...? This message posted from opensolaris.org
Robert Milkowski
2006-Jun-08 23:43 UTC
[zfs-discuss] Re: Wrong reported free space over NFS
bash-3.00# zfs list|grep 5201 nfs-s5-s8/d5201 331G 269G 314G /nfs-s5-s8/d5201 nfs-s5-s8/d5201 at r1 17.2G - 331G - nfs-s5-s8/d5201 at r2 9.29M - 314G - 600g-17g = 583gb Hmmm... why after setting quota to a filesystem reported size over nfs is lowered by the filesystem''s snapshot size (referenced size)? The filesystem (5201) was created by recv (first full send @r1 then incramental @r2) then quota was set. This message posted from opensolaris.org
The problem is that statvfs() only returns two values (total blocks and free blocks) from which we have to calculate three values: size, free, and available space. Prior to pooled storage, available = size - free. This isn''t true with ZFS. On your local filesystem, df(1) recognizes it as a ZFS filesystem, and uses libzfs to get the real amount of available space. Over NFS, we have no choice but to stick with POSIX semantics, which means that we can never provide you with the right answer. For implementation details, check out adjust_total_blocks() in usr/src/cmd/fs.d/df.c. - Eric On Thu, Jun 08, 2006 at 04:38:57PM -0700, Robert Milkowski wrote:> NFS server (b39): > > bash-3.00# zfs get quota nfs-s5-s8/d5201 nfs-s5-p0/d5110 > NAME PROPERTY VALUE SOURCE > nfs-s5-p0/d5110 quota 600G local > nfs-s5-s8/d5201 quota 600G local > bash-3.00# > bash-3.00# df -h | egrep "d5201|d5110" > nfs-s5-p0/d5110 600G 527G 73G 88% /nfs-s5-p0/d5110 > nfs-s5-s8/d5201 600G 314G 269G 54% /nfs-s5-s8/d5201 > bash-3.00# > > > NFS client (S10U1 + patches, NFSv3 mount over TCP): > > bash-3.00# df -h | egrep "d5201|d5110" > NFS-srv:/nfs-s5-p0/d5110 600G 527G 73G 88% /opt/d5110 > NFS-srv:/nfs-s5-s8/d5201 583G 314G 269G 54% /opt/d5201 > bash-3.00# > > > Well why I get 583GB size for d5201 on NFS client? > > ps. maybe I''m tired and missiong something really obvious...? > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
On Thu, Eric Schrock wrote:> The problem is that statvfs() only returns two values (total blocks and > free blocks) from which we have to calculate three values: size, free,?>From statvfs(2) the following are returned in struct statvfs:fsblkcnt_t f_blocks; /* total # of blocks on file system in units of f_frsize */ fsblkcnt_t f_bfree; /* total # of free blocks */ fsblkcnt_t f_bavail; /* # of free blocks avail to So, the data is being passed back. Is there something I am missing?> and available space. Prior to pooled storage, available = size - free. > This isn''t true with ZFS. On your local filesystem, df(1) recognizes it > as a ZFS filesystem, and uses libzfs to get the real amount of available > space. Over NFS, we have no choice but to stick with POSIX semantics, > which means that we can never provide you with the right answer. For > implementation details, check out adjust_total_blocks() in > usr/src/cmd/fs.d/df.c.So, from the comments, that bit of df code seems to be adjusting for quotas if they exist? I am not sure I understand why zfs'' VFS_STATVFS() function can''t do what the df command is doing and then return the appropriate value to both df and the NFS server? So, in Robert''s case, is that 17GB really available and if so that would seem to be an important thing to report to the NFS clients. Spencer> On Thu, Jun 08, 2006 at 04:38:57PM -0700, Robert Milkowski wrote: > > NFS server (b39): > > > > bash-3.00# zfs get quota nfs-s5-s8/d5201 nfs-s5-p0/d5110 > > NAME PROPERTY VALUE SOURCE > > nfs-s5-p0/d5110 quota 600G local > > nfs-s5-s8/d5201 quota 600G local > > bash-3.00# > > bash-3.00# df -h | egrep "d5201|d5110" > > nfs-s5-p0/d5110 600G 527G 73G 88% /nfs-s5-p0/d5110 > > nfs-s5-s8/d5201 600G 314G 269G 54% /nfs-s5-s8/d5201 > > bash-3.00# > > > > > > NFS client (S10U1 + patches, NFSv3 mount over TCP): > > > > bash-3.00# df -h | egrep "d5201|d5110" > > NFS-srv:/nfs-s5-p0/d5110 600G 527G 73G 88% /opt/d5110 > > NFS-srv:/nfs-s5-s8/d5201 583G 314G 269G 54% /opt/d5201 > > bash-3.00# > > > > > > Well why I get 583GB size for d5201 on NFS client? > > > > ps. maybe I''m tired and missiong something really obvious...? > > > > > > This message posted from opensolaris.org > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > -- > Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Thu, Jun 08, 2006 at 10:53:06PM -0500, Spencer Shepler wrote:> > On Thu, Eric Schrock wrote: > > The problem is that statvfs() only returns two values (total blocks and > > free blocks) from which we have to calculate three values: size, free, > > ? > > >From statvfs(2) the following are returned in struct statvfs: > > fsblkcnt_t f_blocks; /* total # of blocks on file system > in units of f_frsize */ > fsblkcnt_t f_bfree; /* total # of free blocks */ > fsblkcnt_t f_bavail; /* # of free blocks avail to > > So, the data is being passed back. Is there something I am missing?Yes, because these values aren''t as straightforward as they seem. For example, consider the return values from UFS: $ truss -t statvfs -v statvfs df -h / statvfs64("/", 0x080479BC) = 0 bsize=8192 frsize=1024 blocks=8068757 bfree=2258725 bavail=2178038 files=972608 ffree=809612 favail=809612 fsid=0x1980000 basetype=ufs namemax=255 flag=ST_NOTRUNC fstr="" Filesystem size used avail capacity Mounted on /dev/dsk/c1d0s0 7.7G 5.5G 2.1G 73% / $ Notice that the values don''t correspond to your assumption. In particular, ''bfree + bavail != blocks''. The two values for ''bfree'' and ''bavail'' are used for filesystems that have a notion of ''reserved'' blocks, i.e. metadata blocks which are used by the filesystem but not available to the user in the form of free space. That''s why you have two values, and if you look at the source code for df(1), you''ll see that it never uses ''bfree'' (except in rare internal calculations) because it''s basically useless.> So, from the comments, that bit of df code seems to be adjusting > for quotas if they exist? I am not sure I understand why > zfs'' VFS_STATVFS() function can''t do what the df command is doing > and then return the appropriate value to both df and the NFS server? > > So, in Robert''s case, is that 17GB really available and if so that > would seem to be an important thing to report to the NFS clients.Because as mentioned above, ''free'' and ''avail'' are really variations on the same theme. And ZFS _must_ report them as the same value (because metadata is accounted in the same way as data), so we''re constrained by the POSIX interface. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
On Fri, Eric Schrock wrote:> On Thu, Jun 08, 2006 at 10:53:06PM -0500, Spencer Shepler wrote: > > > > On Thu, Eric Schrock wrote: > > > The problem is that statvfs() only returns two values (total blocks and > > > free blocks) from which we have to calculate three values: size, free, > > > > ? > > > > >From statvfs(2) the following are returned in struct statvfs: > > > > fsblkcnt_t f_blocks; /* total # of blocks on file system > > in units of f_frsize */ > > fsblkcnt_t f_bfree; /* total # of free blocks */ > > fsblkcnt_t f_bavail; /* # of free blocks avail to > > > > So, the data is being passed back. Is there something I am missing? > > Yes, because these values aren''t as straightforward as they seem. For > example, consider the return values from UFS: > > $ truss -t statvfs -v statvfs df -h / > statvfs64("/", 0x080479BC) = 0 > bsize=8192 frsize=1024 blocks=8068757 bfree=2258725 > bavail=2178038 files=972608 ffree=809612 favail=809612 > fsid=0x1980000 basetype=ufs namemax=255 > flag=ST_NOTRUNC > fstr="" > Filesystem size used avail capacity Mounted on > /dev/dsk/c1d0s0 7.7G 5.5G 2.1G 73% / > $ > > Notice that the values don''t correspond to your assumption. In > particular, ''bfree + bavail != blocks''. The two values for ''bfree'' and > ''bavail'' are used for filesystems that have a notion of ''reserved'' > blocks, i.e. metadata blocks which are used by the filesystem but not > available to the user in the form of free space. That''s why you have > two values, and if you look at the source code for df(1), you''ll see > that it never uses ''bfree'' (except in rare internal calculations) > because it''s basically useless.I must have been half asleep when looking at this; thanks for the clue bat. Spencer
But you''re not really constrained to the POSIX interface when talking to the NFS server. At least in theory, one could add a new VFS operation (for instance), if what you''re returning right now isn?t sufficient. NFS wants just "total space in bytes", "free space in bytes", and "free space available to this user", which it seems might map into pools fairly well. This message posted from opensolaris.org
Hello Eric, Friday, June 9, 2006, 5:06:49 PM, you wrote: ES> On Thu, Jun 08, 2006 at 10:53:06PM -0500, Spencer Shepler wrote:>> >> On Thu, Eric Schrock wrote: >> > The problem is that statvfs() only returns two values (total blocks and >> > free blocks) from which we have to calculate three values: size, free, >> >> ? >> >> >From statvfs(2) the following are returned in struct statvfs: >> >> fsblkcnt_t f_blocks; /* total # of blocks on file system >> in units of f_frsize */ >> fsblkcnt_t f_bfree; /* total # of free blocks */ >> fsblkcnt_t f_bavail; /* # of free blocks avail to >> >> So, the data is being passed back. Is there something I am missing?ES> Yes, because these values aren''t as straightforward as they seem. For ES> example, consider the return values from UFS: ES> $ truss -t statvfs -v statvfs df -h / ES> statvfs64("/", 0x080479BC) = 0 ES> bsize=8192 frsize=1024 blocks=8068757 bfree=2258725 ES> bavail=2178038 files=972608 ffree=809612 favail=809612 ES> fsid=0x1980000 basetype=ufs namemax=255 ES> flag=ST_NOTRUNC ES> fstr="" ES> Filesystem size used avail capacity Mounted on ES> /dev/dsk/c1d0s0 7.7G 5.5G 2.1G 73% / ES> $ ES> Notice that the values don''t correspond to your assumption. In ES> particular, ''bfree + bavail != blocks''. The two values for ''bfree'' and ES> ''bavail'' are used for filesystems that have a notion of ''reserved'' ES> blocks, i.e. metadata blocks which are used by the filesystem but not ES> available to the user in the form of free space. That''s why you have ES> two values, and if you look at the source code for df(1), you''ll see ES> that it never uses ''bfree'' (except in rare internal calculations) ES> because it''s basically useless.>> So, from the comments, that bit of df code seems to be adjusting >> for quotas if they exist? I am not sure I understand why >> zfs'' VFS_STATVFS() function can''t do what the df command is doing >> and then return the appropriate value to both df and the NFS server? >> >> So, in Robert''s case, is that 17GB really available and if so that >> would seem to be an important thing to report to the NFS clients.ES> Because as mentioned above, ''free'' and ''avail'' are really variations on ES> the same theme. And ZFS _must_ report them as the same value (because ES> metadata is accounted in the same way as data), so we''re constrained by ES> the POSIX interface. I haven''t been looking into a code (no time lately) but what I''m most concerned is reported SIZE of a filesystem as it''s beeing reported smaller than it actually is. If snapshots reffered space would be accounted to free/available space of a filesystem it would be ok. But changing disk size just because we use snapshots is strange at least. But probably I''m still missing something. Or maybe at least on Solaris<->Solaris we could make it right as someone else suggested? -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Possibly Parallel Threads
- virtdf outputs on host differs from df in guest
- Re: virtdf outputs on host differs from df in guest
- [PATCH v3 0/2] builder: Choose better weights in the planner.
- [PATCH 0/2] builder: Choose better weights in the planner.
- [PATCH v2 0/2] builder: Choose better weights in the planner.