zfs discuss - Jun 2006 - Wrong reported free space over NFS

NFS server (b39):

bash-3.00# zfs get quota nfs-s5-s8/d5201 nfs-s5-p0/d5110
NAME             PROPERTY       VALUE                      SOURCE
nfs-s5-p0/d5110  quota          600G                       local
nfs-s5-s8/d5201  quota          600G                       local
bash-3.00#
bash-3.00# df -h | egrep "d5201|d5110"
nfs-s5-p0/d5110        600G   527G    73G    88%    /nfs-s5-p0/d5110
nfs-s5-s8/d5201        600G   314G   269G    54%    /nfs-s5-s8/d5201
bash-3.00#


NFS client (S10U1 + patches, NFSv3 mount over TCP):

bash-3.00# df -h | egrep "d5201|d5110"
NFS-srv:/nfs-s5-p0/d5110   600G   527G    73G    88%    /opt/d5110
NFS-srv:/nfs-s5-s8/d5201   583G   314G   269G    54%    /opt/d5201
bash-3.00#


Well why I get 583GB size for d5201 on NFS client?

ps. maybe I''m tired and missiong something really obvious...?
 
 
This message posted from opensolaris.org

bash-3.00# zfs list|grep 5201
nfs-s5-s8/d5201        331G   269G   314G  /nfs-s5-s8/d5201
nfs-s5-s8/d5201 at r1    17.2G      -   331G  -
nfs-s5-s8/d5201 at r2    9.29M      -   314G  -


600g-17g = 583gb

Hmmm... why after setting quota to a filesystem reported size over nfs is
lowered by the filesystem''s snapshot size (referenced size)?

The filesystem (5201) was created by recv (first full send @r1 then incramental
@r2) then quota was set.
 
 
This message posted from opensolaris.org

The problem is that statvfs() only returns two values (total blocks and
free blocks) from which we have to calculate three values: size, free,
and available space. Prior to pooled storage, available = size - free.
This isn''t true with ZFS.  On your local filesystem, df(1) recognizes
it
as a ZFS filesystem, and uses libzfs to get the real amount of available
space.  Over NFS, we have no choice but to stick with POSIX semantics,
which means that we can never provide you with the right answer.  For
implementation details, check out adjust_total_blocks() in
usr/src/cmd/fs.d/df.c.

- Eric

On Thu, Jun 08, 2006 at 04:38:57PM -0700, Robert Milkowski
wrote:
> NFS server (b39):
> 
> bash-3.00# zfs get quota nfs-s5-s8/d5201 nfs-s5-p0/d5110
> NAME             PROPERTY       VALUE                      SOURCE
> nfs-s5-p0/d5110  quota          600G                       local
> nfs-s5-s8/d5201  quota          600G                       local
> bash-3.00#
> bash-3.00# df -h | egrep "d5201|d5110"
> nfs-s5-p0/d5110        600G   527G    73G    88%    /nfs-s5-p0/d5110
> nfs-s5-s8/d5201        600G   314G   269G    54%    /nfs-s5-s8/d5201
> bash-3.00#
> 
> 
> NFS client (S10U1 + patches, NFSv3 mount over TCP):
> 
> bash-3.00# df -h | egrep "d5201|d5110"
> NFS-srv:/nfs-s5-p0/d5110   600G   527G    73G    88%    /opt/d5110
> NFS-srv:/nfs-s5-s8/d5201   583G   314G   269G    54%    /opt/d5201
> bash-3.00#
> 
> 
> Well why I get 583GB size for d5201 on NFS client?
> 
> ps. maybe I''m tired and missiong something really obvious...?
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

On Thu, Eric Schrock wrote:
> The problem is that statvfs() only returns two values (total blocks and
> free blocks) from which we have to calculate three values: size, free,
?
>From statvfs(2) the following are returned in struct statvfs:
     fsblkcnt_t  f_blocks;            /* total # of blocks on file system
                                         in units of f_frsize */
     fsblkcnt_t  f_bfree;             /* total # of free blocks */
     fsblkcnt_t  f_bavail;            /* # of free blocks avail to

So, the data is being passed back.  Is there something I am missing?
> and available space. Prior to pooled storage, available = size - free.
> This isn''t true with ZFS.  On your local filesystem, df(1)
recognizes it
> as a ZFS filesystem, and uses libzfs to get the real amount of available
> space.  Over NFS, we have no choice but to stick with POSIX semantics,
> which means that we can never provide you with the right answer.  For
> implementation details, check out adjust_total_blocks() in
> usr/src/cmd/fs.d/df.c.
So, from the comments, that bit of df code seems to be adjusting
for quotas if they exist?  I am not sure I understand why 
zfs'' VFS_STATVFS() function can''t do what the df command is
doing
and then return the appropriate value to both df and the NFS server?

So, in Robert''s case, is that 17GB really available and if so that
would seem to be an important thing to report to the NFS clients.

Spencer

> On Thu, Jun 08, 2006 at 04:38:57PM -0700, Robert Milkowski wrote:
> > NFS server (b39):
> > 
> > bash-3.00# zfs get quota nfs-s5-s8/d5201 nfs-s5-p0/d5110
> > NAME             PROPERTY       VALUE                      SOURCE
> > nfs-s5-p0/d5110  quota          600G                       local
> > nfs-s5-s8/d5201  quota          600G                       local
> > bash-3.00#
> > bash-3.00# df -h | egrep "d5201|d5110"
> > nfs-s5-p0/d5110        600G   527G    73G    88%    /nfs-s5-p0/d5110
> > nfs-s5-s8/d5201        600G   314G   269G    54%    /nfs-s5-s8/d5201
> > bash-3.00#
> > 
> > 
> > NFS client (S10U1 + patches, NFSv3 mount over TCP):
> > 
> > bash-3.00# df -h | egrep "d5201|d5110"
> > NFS-srv:/nfs-s5-p0/d5110   600G   527G    73G    88%    /opt/d5110
> > NFS-srv:/nfs-s5-s8/d5201   583G   314G   269G    54%    /opt/d5201
> > bash-3.00#
> > 
> > 
> > Well why I get 583GB size for d5201 on NFS client?
> > 
> > ps. maybe I''m tired and missiong something really obvious...?
> >  
> >  
> > This message posted from opensolaris.org
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> 
> --
> Eric Schrock, Solaris Kernel Development      
http://blogs.sun.com/eschrock
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Thu, Jun 08, 2006 at 10:53:06PM -0500, Spencer Shepler
wrote:
> 
> On Thu, Eric Schrock wrote:
> > The problem is that statvfs() only returns two values (total blocks
and
> > free blocks) from which we have to calculate three values: size, free,
> 
> ?
> 
> >From statvfs(2) the following are returned in struct statvfs:
> 
>      fsblkcnt_t  f_blocks;            /* total # of blocks on file system
>                                          in units of f_frsize */
>      fsblkcnt_t  f_bfree;             /* total # of free blocks */
>      fsblkcnt_t  f_bavail;            /* # of free blocks avail to
> 
> So, the data is being passed back.  Is there something I am missing?
Yes, because these values aren''t as straightforward as they seem.  For
example, consider the return values from UFS:

$ truss -t statvfs -v statvfs df -h /               
statvfs64("/", 0x080479BC)                      = 0
        bsize=8192       frsize=1024      blocks=8068757  bfree=2258725  
        bavail=2178038   files=972608     ffree=809612    favail=809612   
        fsid=0x1980000   basetype=ufs     namemax=255
        flag=ST_NOTRUNC
        fstr=""
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1d0s0        7.7G   5.5G   2.1G    73%    /
$

Notice that the values don''t correspond to your assumption.  In
particular, ''bfree + bavail != blocks''.  The two values for
''bfree'' and
''bavail'' are used for filesystems that have a notion of
''reserved''
blocks, i.e. metadata blocks which are used by the filesystem but not
available to the user in the form of free space.  That''s why you have
two values, and if you look at the source code for df(1), you''ll see
that it never uses ''bfree'' (except in rare internal
calculations)
because it''s basically useless.
> So, from the comments, that bit of df code seems to be adjusting
> for quotas if they exist?  I am not sure I understand why 
> zfs'' VFS_STATVFS() function can''t do what the df command
is doing
> and then return the appropriate value to both df and the NFS server?
> 
> So, in Robert''s case, is that 17GB really available and if so that
> would seem to be an important thing to report to the NFS clients.
Because as mentioned above, ''free'' and
''avail'' are really variations on
the same theme.  And ZFS _must_ report them as the same value (because
metadata is accounted in the same way as data), so we''re constrained by
the POSIX interface. 

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

On Fri, Eric Schrock wrote:
> On Thu, Jun 08, 2006 at 10:53:06PM -0500, Spencer Shepler wrote:
> > 
> > On Thu, Eric Schrock wrote:
> > > The problem is that statvfs() only returns two values (total
blocks and
> > > free blocks) from which we have to calculate three values: size,
free,
> > 
> > ?
> > 
> > >From statvfs(2) the following are returned in struct statvfs:
> > 
> >      fsblkcnt_t  f_blocks;            /* total # of blocks on file
system
> >                                          in units of f_frsize */
> >      fsblkcnt_t  f_bfree;             /* total # of free blocks */
> >      fsblkcnt_t  f_bavail;            /* # of free blocks avail to
> > 
> > So, the data is being passed back.  Is there something I am missing?
> 
> Yes, because these values aren''t as straightforward as they seem. 
For
> example, consider the return values from UFS:
> 
> $ truss -t statvfs -v statvfs df -h /               
> statvfs64("/", 0x080479BC)                      = 0
>         bsize=8192       frsize=1024      blocks=8068757  bfree=2258725  
>         bavail=2178038   files=972608     ffree=809612    favail=809612   
>         fsid=0x1980000   basetype=ufs     namemax=255
>         flag=ST_NOTRUNC
>         fstr=""
> Filesystem             size   used  avail capacity  Mounted on
> /dev/dsk/c1d0s0        7.7G   5.5G   2.1G    73%    /
> $
> 
> Notice that the values don''t correspond to your assumption.  In
> particular, ''bfree + bavail != blocks''.  The two values
for ''bfree'' and
> ''bavail'' are used for filesystems that have a notion of
''reserved''
> blocks, i.e. metadata blocks which are used by the filesystem but not
> available to the user in the form of free space.  That''s why you
have
> two values, and if you look at the source code for df(1), you''ll
see
> that it never uses ''bfree'' (except in rare internal
calculations)
> because it''s basically useless.
I must have been half asleep when looking at this; thanks for the clue bat.

Spencer

But you''re not really constrained to the POSIX interface when talking
to the NFS server. At least in theory, one could add a new VFS operation (for
instance), if what you''re returning right now isn?t sufficient.

NFS wants just "total space in bytes", "free space in
bytes", and "free space available to this user", which it seems
might map into pools fairly well.
 
 
This message posted from opensolaris.org

Hello Eric,

Friday, June 9, 2006, 5:06:49 PM, you wrote:

ES> On Thu, Jun 08, 2006 at 10:53:06PM -0500, Spencer Shepler
wrote:
>> 
>> On Thu, Eric Schrock wrote:
>> > The problem is that statvfs() only returns two values (total
blocks and
>> > free blocks) from which we have to calculate three values: size,
free,
>> 
>> ?
>> 
>> >From statvfs(2) the following are returned in struct statvfs:
>> 
>>      fsblkcnt_t  f_blocks;            /* total # of blocks on file
system
>>                                          in units of f_frsize */
>>      fsblkcnt_t  f_bfree;             /* total # of free blocks */
>>      fsblkcnt_t  f_bavail;            /* # of free blocks avail to
>> 
>> So, the data is being passed back.  Is there something I am missing?
ES> Yes, because these values aren''t as straightforward as they
seem.  For
ES> example, consider the return values from UFS:

ES> $ truss -t statvfs -v statvfs df -h /               
ES> statvfs64("/", 0x080479BC)                      = 0
ES>         bsize=8192       frsize=1024      blocks=8068757  bfree=2258725
ES>         bavail=2178038   files=972608     ffree=809612    favail=809612
ES>         fsid=0x1980000   basetype=ufs     namemax=255
ES>         flag=ST_NOTRUNC
ES>         fstr=""
ES> Filesystem             size   used  avail capacity  Mounted on
ES> /dev/dsk/c1d0s0        7.7G   5.5G   2.1G    73%    /
ES> $

ES> Notice that the values don''t correspond to your assumption.  In
ES> particular, ''bfree + bavail != blocks''.  The two values
for ''bfree'' and
ES> ''bavail'' are used for filesystems that have a notion of
''reserved''
ES> blocks, i.e. metadata blocks which are used by the filesystem but not
ES> available to the user in the form of free space.  That''s why you
have
ES> two values, and if you look at the source code for df(1), you''ll
see
ES> that it never uses ''bfree'' (except in rare internal
calculations)
ES> because it''s basically useless.
>> So, from the comments, that bit of df code seems to be adjusting
>> for quotas if they exist?  I am not sure I understand why 
>> zfs'' VFS_STATVFS() function can''t do what the df
command is doing
>> and then return the appropriate value to both df and the NFS server?
>> 
>> So, in Robert''s case, is that 17GB really available and if so
that
>> would seem to be an important thing to report to the NFS clients.
ES> Because as mentioned above, ''free'' and
''avail'' are really variations on
ES> the same theme.  And ZFS _must_ report them as the same value (because
ES> metadata is accounted in the same way as data), so we''re
constrained by
ES> the POSIX interface. 

I haven''t been looking into a code (no time lately) but what
I''m most
concerned is reported SIZE of a filesystem as it''s beeing reported
smaller than it actually is. If snapshots reffered space would be
accounted to free/available space of a filesystem it would be ok.
But changing disk size just because we use snapshots is strange at
least.

But probably I''m still missing something.
Or maybe at least on Solaris<->Solaris we could make it right as
someone else suggested?

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com