zfs discuss - Mar 2006 - Lots of filesystems and df

Are there any plans to do something about the "lots of filesystems"
and
df output problem or is it not perceived as being a problem?

And, are there any other utilities, like df, that are going to suffer from
there being a plethora of filesystems with ZFS when people create one
for each local user...all n-000 of them?  If you''re outside of Sun and
wondering what I''m thinking about, imagie you have 2000 users and
then write a script to create 2000 ZFS filesystems as the home directory
for each user and then do "df" to see how much disk space is
free/used.

When I think about what df now shows for ZFS, especially in the case of
home directories, it strikes me as being "too much information".

One idea that came to my mind was to have df, et al, look for some special
flag in the zpool that causes them, by default to only report on the zpool
itself.  Maybe something like what is done for sharing ZFS filesystems?

And then for those who really want to know, add an option to df and friends
to tell them to display filesystems instead of pools.

Thoughts?

Darren

Darren Reed wrote:
> Are there any plans to do something about the "lots of
filesystems" and
> df output problem or is it not perceived as being a problem?
	.
	.
> One idea that came to my mind was to have df, et al, look for some special
> flag in the zpool that causes them, by default to only report on the zpool
> itself.  Maybe something like what is done for sharing ZFS filesystems?
> 
> And then for those who really want to know, add an option to df and friends
> to tell them to display filesystems instead of pools.
Seconded!

Df traditionally reports available vs used space.  So by default,
you''d only report on available pools, which would be sufficient
at first glance.

Further behaviour can be extended to let df
* interpret zfs mounts as traditional file systems
* report on mounts with reservations
* report up to /n/ levels deep

Henk

Darren sez:
>Are there any plans to do something about the "lots of
filesystems" and
>df output problem or is it not perceived as being a problem?
I would agree that in the ZFS world df is not very useful. If in your 
example you have 2000 filesystems in one pool, seeing the same stats for 
each tells the user nothing useful.

What is more intetesting is how much of the filesystems quota has been used 
and is available.

So while it might be hard to change the default behavior, it might be 
useful to add a new "-q" option that people could be retrained to use.

Additionally it might be useful to have an option to modify the output 
(like "-h") that only displays pools.

	-David

On Sat, Mar 04, 2006 at 11:23:11PM -0800, Darren Reed
wrote:
> Are there any plans to do something about the "lots of
filesystems" and
> df output problem or is it not perceived as being a problem?
What exactly is the problem?  The problem of utilities like share(1M)
not scaling is well understood and in progress of being fixed.  For
df(1), what piece of information are you trying to get out of the
utility where lots of filesystems presents a problem?
> And, are there any other utilities, like df, that are going to suffer from
> there being a plethora of filesystems with ZFS when people create one
> for each local user...all n-000 of them?  If you''re outside of Sun
and
> wondering what I''m thinking about, imagie you have 2000 users and
> then write a script to create 2000 ZFS filesystems as the home directory
> for each user and then do "df" to see how much disk space is
free/used.
Why wouldn''t you use "zfs list tank/home" in the above
situation?  And
then you can use "zfs list -r tank/home" if you really wanted to see
it
on a per-filesystem level.
> When I think about what df now shows for ZFS, especially in the case of
> home directories, it strikes me as being "too much information".
> 
> One idea that came to my mind was to have df, et al, look for some special
> flag in the zpool that causes them, by default to only report on the zpool
> itself.  Maybe something like what is done for sharing ZFS filesystems?
> 
> And then for those who really want to know, add an option to df and friends
> to tell them to display filesystems instead of pools.
Certainly we can''t change the default to only show pools.  But what
benefit would showing pools have if what you really wanted to know was
capacity information?  A single entry for ''tank''
couldn''t show that
''tank/home/eschrock'' is nearing its quota, for example.

I guess I''m just unclear on what the user is actually trying to get
from
df (with no arguments).  Certainly in the world of ZFS root, I would
imagine df(1) becoming obsolete pretty quickly.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Interesting...

I totally saw where Darren was coming from ("the problem") but after
reading Eric''s POV maybe there is no problem?

I guess the issue is that sys-admins currently type: df -kh to keep an
eye on whats happening in /var, /, /opt, etc. Typically that''s a pretty
manageable list and its an easy way to glance at whats going on.

When you have boatloads of NFS mounted volumes, the "-l" option of df
will show you only the LOCAL mounts... (which comes in handy, again
making the df output "useful" / human-eye-parse able.)

If with the advent of zfs home-directory-volumes (per user) we end up
with a df -kh that goes on for 12-pages, yeah that''s not so useful... I
don''t think the "-l" option will help here, since these
volumes would
still be considered LOCAL. I guess you can specify that df ONLY report
on ufs volumes, tmpfs, and perhaps some Vx filesystems but that gets old
quick...

--

As someone else suggested some sort of "-z" option to not show
individual ZFS volumes, or somehow show only the zpool stats would
likely eliminate the problem. (as you really shouldn''t have 2000
separate zpools)

--

As you said, if zfs goes "everywhere" root, zfs, (tmpfs?), etc. then
things get better in that there won''t be such a need for the
traditional
"df" anymore.... Although if we continue to co-exists with NFS,
VxVM/FS,
UFS, etc. etc. "df" will likely continue to be used.

I don''t have a situation where I''m dealing with 2000 zfs
volumes yet,
we''ll see how it shakes out when the admins actually start maintaining
such systems.

Thanks,

 -- MikeE

-----Original Message-----
From: zfs-discuss-bounces at opensolaris.org
[mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Eric Schrock
Sent: Sunday, March 05, 2006 1:45 PM
To: Darren Reed
Cc: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] Lots of filesystems and df


On Sat, Mar 04, 2006 at 11:23:11PM -0800, Darren Reed
wrote:
> Are there any plans to do something about the "lots of
filesystems"
and
> df output problem or is it not perceived as being a problem?
What exactly is the problem?  The problem of utilities like share(1M)
not scaling is well understood and in progress of being fixed.  For
df(1), what piece of information are you trying to get out of the
utility where lots of filesystems presents a problem?
> And, are there any other utilities, like df, that are going to suffer
from
> there being a plethora of filesystems with ZFS when people create one
> for each local user...all n-000 of them?  If you''re outside of Sun
and
> wondering what I''m thinking about, imagie you have 2000 users and
> then write a script to create 2000 ZFS filesystems as the home
directory
> for each user and then do "df" to see how much disk space is
free/used.

Why wouldn''t you use "zfs list tank/home" in the above
situation?  And
then you can use "zfs list -r tank/home" if you really wanted to see
it
on a per-filesystem level.
> When I think about what df now shows for ZFS, especially in the case
of
> home directories, it strikes me as being "too much information".
> 
> One idea that came to my mind was to have df, et al, look for some
special
> flag in the zpool that causes them, by default to only report on the
zpool
> itself.  Maybe something like what is done for sharing ZFS
filesystems?
> 
> And then for those who really want to know, add an option to df and
friends
> to tell them to display filesystems instead of pools.
Certainly we can''t change the default to only show pools.  But what
benefit would showing pools have if what you really wanted to know was
capacity information?  A single entry for ''tank''
couldn''t show that
''tank/home/eschrock'' is nearing its quota, for example.

I guess I''m just unclear on what the user is actually trying to get
from
df (with no arguments).  Certainly in the world of ZFS root, I would
imagine df(1) becoming obsolete pretty quickly.

- Eric

--
Eric Schrock, Solaris Kernel Development
http://blogs.sun.com/eschrock
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Sun, Mar 05, 2006 at 09:40:17AM +0100, Henk Langeveld
wrote:
> Darren Reed wrote:
> >Are there any plans to do something about the "lots of
filesystems" and
> >df output problem or is it not perceived as being a problem?
> 	.
> 	.
> 
> >One idea that came to my mind was to have df, et al, look for some
special
> >flag in the zpool that causes them, by default to only report on the
zpool
> >itself.  Maybe something like what is done for sharing ZFS filesystems?
> >
> >And then for those who really want to know, add an option to df and
friends
> >to tell them to display filesystems instead of pools.
> 
> Seconded!
> 
> Df traditionally reports available vs used space.  So by default,
> you''d only report on available pools, which would be sufficient
> at first glance.
I use mount(1M) then df(1) to find out what NFS mount points are
hanging...  :^|

And I use df -k to get an idea of how much free space I might have, at
least where I have small disks.

I think df(1) -k should show quota usage for ZFS filesystems with
quotas, else pool space usage.

Perhaps we could have property to indicate whether a ZFS filesystem
should or should not be listed in df(1) output normally.  Similarly, a
pool property to indicate whether the pool should be listed in df(1)
output, even though pools aren''t filesystems.

Nico
--

As an admin, I look at ''df'' output to see what might be full. 
I
think that an option to df that
would report on the first filesystem in a pool (excluding the rest)  
would report what is desired.

I don''t really like the idea of having thousands of filesystems to  
grep through.  Perhaps a ''-u'' option
that would report on one filesystem per pool?

On an unrelated note, another feature that I''d like to see would be  
the ability to assign the mountpoint
on a filesystem create.  Something like:

zfs create -o mountpoint=/export/home local/home

That would save a step in the filesystem create where the filesystem  
isn''t using the default hierarchy.

On Mar 5, 2006, at 3:39 PM, Nicolas Williams wrote:
> On Sun, Mar 05, 2006 at 09:40:17AM +0100, Henk Langeveld wrote:
>> Darren Reed wrote:
>>> Are there any plans to do something about the "lots of  
>>> filesystems" and
>>> df output problem or is it not perceived as being a problem?
>> 	.
>> 	.
>>
>>> One idea that came to my mind was to have df, et al, look for  
>>> some special
>>> flag in the zpool that causes them, by default to only report on  
>>> the zpool
>>> itself.  Maybe something like what is done for sharing ZFS  
>>> filesystems?
>>>
>>> And then for those who really want to know, add an option to df  
>>> and friends
>>> to tell them to display filesystems instead of pools.
>>
>> Seconded!
>>
>> Df traditionally reports available vs used space.  So by default,
>> you''d only report on available pools, which would be
sufficient
>> at first glance.
>
> I use mount(1M) then df(1) to find out what NFS mount points are
> hanging...  :^|
>
> And I use df -k to get an idea of how much free space I might have, at
> least where I have small disks.
>
> I think df(1) -k should show quota usage for ZFS filesystems with
> quotas, else pool space usage.
>
> Perhaps we could have property to indicate whether a ZFS filesystem
> should or should not be listed in df(1) output normally.  Similarly, a
> pool property to indicate whether the pool should be listed in df(1)
> output, even though pools aren''t filesystems.
>
> Nico
> -- 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-----
Gregory Shaw, IT Architect
Phone: (303) 673-8273        Fax: (303) 673-8273
SunIT, Sun Microsystems Corporation
1 StorageTek Drive MS 4382              greg.shaw at sun.com (work)
Louisville, CO 80028-4382                 shaw at fmsoft.com (home)
"When Microsoft writes an application for Linux, I''ve Won." -
Linus
Torvalds

On Sun, 2006-03-05 at 17:39, Nicolas Williams wrote:
> Perhaps we could have property to indicate whether a ZFS filesystem
> should or should not be listed in df(1) output normally.  Similarly, a
> pool property to indicate whether the pool should be listed in df(1)
> output, even though pools aren''t filesystems.
And there''s already a mount property -- "ignore" -- which
causes df to
skip the filesystem unless -a is given; a zfs property which sets the
ignore flag would do the trick quite nicely (and, on that note,
/devices, /proc, /etc/mnttab, and /system/* would be good candidates for
adding "ignore".).

					- Bill

Eric -

One thing I could think of that makes the df problem a problem is that 
many sites have their own monitoring scripts that check out filesystem 
usage. The check against a rule of (for example) 90% full generates a 
warning, etc.

Getting 2000 warnings due to a zpool nearing capacity would be a 
significant burden on the monitoring tools and potentially the network 
(depending on how those warnings are generated / sent). It might also 
drown out useful information due to block deleteds of warnings.

Worse yet, if they customers decided to subscribe to the ''zfs''
way using
''zfs list'' etc, then they would multiple methods in their
monitoring,
just to account for zfs, when every other filesystem type they monitor 
works just fine using plain old df.

Makes things more complicated for the customer, and more work for the 
admin...

I''m not saying either way is right or wrong, but just trying to help
you
understand why some folks may not be enamoured with having 2000 odd 
filesystems to contend with.

Nathan. :)



Eric Schrock wrote:
> On Sat, Mar 04, 2006 at 11:23:11PM -0800, Darren Reed wrote:
> 
>>Are there any plans to do something about the "lots of
filesystems" and
>>df output problem or is it not perceived as being a problem?
> 
> 
> What exactly is the problem?  The problem of utilities like share(1M)
> not scaling is well understood and in progress of being fixed.  For
> df(1), what piece of information are you trying to get out of the
> utility where lots of filesystems presents a problem?
> 
> 
>>And, are there any other utilities, like df, that are going to suffer
from
>>there being a plethora of filesystems with ZFS when people create one
>>for each local user...all n-000 of them?  If you''re outside of
Sun and
>>wondering what I''m thinking about, imagie you have 2000 users
and
>>then write a script to create 2000 ZFS filesystems as the home directory
>>for each user and then do "df" to see how much disk space is
free/used.
> 
> 
> Why wouldn''t you use "zfs list tank/home" in the above
situation?  And
> then you can use "zfs list -r tank/home" if you really wanted to
see it
> on a per-filesystem level.
> 
> 
>>When I think about what df now shows for ZFS, especially in the case of
>>home directories, it strikes me as being "too much
information".
>>
>>One idea that came to my mind was to have df, et al, look for some
special
>>flag in the zpool that causes them, by default to only report on the
zpool
>>itself.  Maybe something like what is done for sharing ZFS filesystems?
>>
>>And then for those who really want to know, add an option to df and
friends
>>to tell them to display filesystems instead of pools.
> 
> 
> Certainly we can''t change the default to only show pools.  But
what
> benefit would showing pools have if what you really wanted to know was
> capacity information?  A single entry for ''tank''
couldn''t show that
> ''tank/home/eschrock'' is nearing its quota, for example.
> 
> I guess I''m just unclear on what the user is actually trying to
get from
> df (with no arguments).  Certainly in the world of ZFS root, I would
> imagine df(1) becoming obsolete pretty quickly.
> 
> - Eric
> 
> --
> Eric Schrock, Solaris Kernel Development      
http://blogs.sun.com/eschrock
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

In reading replies from other people, it''s clear that while I may have
failed to convey what my problem is here, a number of others understand
the issue - it''s a usability issue...

Eric Schrock wrote:
>On Sat, Mar 04, 2006 at 11:23:11PM -0800, Darren Reed wrote:
>
>>Are there any plans to do something about the "lots of
filesystems" and
>>df output problem or is it not perceived as being a problem?
>>
>
>What exactly is the problem?  The problem of utilities like share(1M)
>not scaling is well understood and in progress of being fixed.  For
>df(1), what piece of information are you trying to get out of the
>utility where lots of filesystems presents a problem?
>
I want a concise report of all the local filesystems disk usage and
an indication.  "df" gives me that today with UFS (and limited use
of ZFS.)

When I have 1000s of ZFS filesystems in pools, "df" output gets
spammed by ZFS (and takes a long time to run.)

I think Hank Langeveld summarised the problem with:

"df traditionally reports available vs used space. So by default,
you''d only report on available pools, which would be sufficient
at first glance. "
>>And, are there any other utilities, like df, that are going to suffer
from
>>there being a plethora of filesystems with ZFS when people create one
>>for each local user...all n-000 of them?  If you''re outside of
Sun and
>>wondering what I''m thinking about, imagie you have 2000 users
and
>>then write a script to create 2000 ZFS filesystems as the home directory
>>for each user and then do "df" to see how much disk space is
free/used.
>>
>
>Why wouldn''t you use "zfs list tank/home" in the above
situation?  And
>then you can use "zfs list -r tank/home" if you really wanted to
see it
>on a per-filesystem level.
>
Because "zfs list" is not going to cover all my local filesystems,
only ZFS filesystems.  Maybe after ZFS has conquered the world of
filesystems this will cease to be an issue :)

But meantime there are a lot of scripts that have been built up over
the years (see Nathan Kroenert''s comment) that now need some sort
of hackery to use a combination of "zfs list" and "df".  I
can''t see that
going down too well...
>>When I think about what df now shows for ZFS, especially in the case of
>>home directories, it strikes me as being "too much
information".
>>
>>One idea that came to my mind was to have df, et al, look for some
special
>>flag in the zpool that causes them, by default to only report on the
zpool
>>itself.  Maybe something like what is done for sharing ZFS filesystems?
>>
>>And then for those who really want to know, add an option to df and
friends
>>to tell them to display filesystems instead of pools.
>>
>
>Certainly we can''t change the default to only show pools.  But what
>benefit would showing pools have if what you really wanted to know was
>capacity information?  A single entry for ''tank''
couldn''t show that
>''tank/home/eschrock'' is nearing its quota, for example.
>
I''m not concerned about quota and I''m not sure that most
administrators are,
either, when it comes to per-user filesystems in a ZFS pool.

By default, my reflex is to use "df -kl" on systems I interact with
because
I want to know how much disk space is used locally and if I need to think
about adding more disk space (or if /var/tmp or /tmp is going to have enough
free space, etc.)

When I do this with ZFS, I see all the local ZFS filesystems and the pools
are quickly lost.  Doing a "df" on jurassic, today, is a mistake, 
because all
of the useful information scrolls off the top of the screen before I have a
chance to blink - there is too much information and much more than I need or
care for when using "df".

Maybe a "-p" option, so I can do "df -lkp" to only show ZFS
pools when it
comes to displaying ZFS information?

Or maybe a pool option to set the "ignore" mount property (see Bill
Sommerfeld''s comment) for all filesystems in that pool?

Darren

On 3/5/06, Darren Reed <Darren.Reed at sun.com>
wrote:
> In reading replies from other people, it''s clear that while I may
have
> failed to convey what my problem is here, a number of others understand
> the issue - it''s a usability issue...
>
> Eric Schrock wrote:
>
> >On Sat, Mar 04, 2006 at 11:23:11PM -0800, Darren Reed wrote:
> >
> >>Are there any plans to do something about the "lots of
filesystems" and
> >>df output problem or is it not perceived as being a problem?
> >>
> >
> >What exactly is the problem?  The problem of utilities like share(1M)
> >not scaling is well understood and in progress of being fixed.  For
> >df(1), what piece of information are you trying to get out of the
> >utility where lots of filesystems presents a problem?
> >
>
> I want a concise report of all the local filesystems disk usage and
> an indication.  "df" gives me that today with UFS (and limited
use
> of ZFS.)
>
> When I have 1000s of ZFS filesystems in pools, "df" output gets
> spammed by ZFS (and takes a long time to run.)
>
> I think Hank Langeveld summarised the problem with:
>
> "df traditionally reports available vs used space. So by default,
> you''d only report on available pools, which would be sufficient
> at first glance. "
>
> >>And, are there any other utilities, like df, that are going to
suffer from
> >>there being a plethora of filesystems with ZFS when people create
one
> >>for each local user...all n-000 of them?  If you''re
outside of Sun and
> >>wondering what I''m thinking about, imagie you have 2000
users and
> >>then write a script to create 2000 ZFS filesystems as the home
directory
> >>for each user and then do "df" to see how much disk space
is free/used.
> >>
> >
my idea, for an extension would be add a -s argument to df it would
function like  -s in du (sumarize). It would print one line for each 
filesystem or pool. If everything is on a single pool, after zfs
boot/root is intergrated it would print a single line telling space
used and remaining in that pool. In the current world it would print
out a line for each ufs or cdrom based filesystem.

James Dickens
uadmin.blogspot.com
> >Why wouldn''t you use "zfs list tank/home" in the
above situation?  And
> >then you can use "zfs list -r tank/home" if you really wanted
to see it
> >on a per-filesystem level.
> >
>
> Because "zfs list" is not going to cover all my local
filesystems,
> only ZFS filesystems.  Maybe after ZFS has conquered the world of
> filesystems this will cease to be an issue :)
>
> But meantime there are a lot of scripts that have been built up over
> the years (see Nathan Kroenert''s comment) that now need some sort
> of hackery to use a combination of "zfs list" and "df".
I can''t see that
> going down too well...
>
> >>When I think about what df now shows for ZFS, especially in the
case of
> >>home directories, it strikes me as being "too much
information".
> >>
> >>One idea that came to my mind was to have df, et al, look for some
special
> >>flag in the zpool that causes them, by default to only report on
the zpool
> >>itself.  Maybe something like what is done for sharing ZFS
filesystems?
> >>
> >>And then for those who really want to know, add an option to df and
friends
> >>to tell them to display filesystems instead of pools.
> >>
> >
> >Certainly we can''t change the default to only show pools.  But
what
> >benefit would showing pools have if what you really wanted to know was
> >capacity information?  A single entry for ''tank''
couldn''t show that
> >''tank/home/eschrock'' is nearing its quota, for
example.
> >
>
> I''m not concerned about quota and I''m not sure that most
administrators are,
> either, when it comes to per-user filesystems in a ZFS pool.
>
> By default, my reflex is to use "df -kl" on systems I interact
with because
> I want to know how much disk space is used locally and if I need to think
> about adding more disk space (or if /var/tmp or /tmp is going to have
enough
> free space, etc.)
>
> When I do this with ZFS, I see all the local ZFS filesystems and the pools
> are quickly lost.  Doing a "df" on jurassic, today, is a mistake,
> because all
> of the useful information scrolls off the top of the screen before I have a
> chance to blink - there is too much information and much more than I need
or
> care for when using "df".
>
> Maybe a "-p" option, so I can do "df -lkp" to only show
ZFS pools when it
> comes to displaying ZFS information?
>
> Or maybe a pool option to set the "ignore" mount property (see
Bill
> Sommerfeld''s comment) for all filesystems in that pool?
>
> Darren
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Sun, Mar 05, 2006 at 04:21:03PM -0700, Gregory Shaw
wrote:
> As an admin, I look at ''df'' output to see what might be
full.   I
> think that an option to df that
> would report on the first filesystem in a pool (excluding the rest)  
> would report what is desired.
> 
> I don''t really like the idea of having thousands of filesystems to
> grep through.  Perhaps a ''-u'' option
> that would report on one filesystem per pool?
Well, once we have the ability to sort ''zfs list'' output, why
not just
do:

# zfs list -o name,avail -S avail

You can trim off as many as you''d like with head(1).
> On an unrelated note, another feature that I''d like to see would
be
> the ability to assign the mountpoint
> on a filesystem create.  Something like:
> 
> zfs create -o mountpoint=/export/home local/home
> 
> That would save a step in the filesystem create where the filesystem  
> isn''t using the default hierarchy.
Yes, there is an RFE open for this - I don''t have the bug ID handy...

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

On Sun, Mar 05, 2006 at 06:22:05PM -0500, Bill Sommerfeld
wrote:
> 
> And there''s already a mount property -- "ignore" --
which causes df to
> skip the filesystem unless -a is given; a zfs property which sets the
> ignore flag would do the trick quite nicely (and, on that note,
> /devices, /proc, /etc/mnttab, and /system/* would be good candidates for
> adding "ignore".).
> 
This would be a simple RFE.  The joys of inheritance would allow you to
set it pool wide in a snap.  This is way better than trying to make df
report on pools instead of filesystems.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

> When I have 1000s of ZFS filesystems in pools, "df" output gets
> spammed by ZFS (and takes a long time to run.)
I believe this is the first time we''ve been called spam.  ;-)
But I agree that there''s a problem worth solving here.

By decoupling naming from storage, ZFS has exposed an intrinsic
ambiguity in df(1M): it assumes that filesystems are the things
that have free disk space.  It also assumes that filesystems have
a fixed size, so that used + free = constant.  No longer true.

[Incidentally, these assumptions have been wrong for tmpfs for years.
Oddly enough, tmpfs was actually Sun''s first pooled-storage filesystem.
All tmpfs mounts consume shared swap space, automatically grow, etc.]

So there are now two distinct reasons to use df(1M): to see how much
space is free, and to see how much space is used.  They''re no longer
the same question.

On a desktop, you''ll typically have few enough filesystems that it
doesn''t matter if we spam you.  But on a server with thousands of
filesystems, the output you want depends which question you''re asking.
If you''re asking about free space, you probably only care about pools.
If you''re asking about space usage, you probably want to see
everything,
or perhaps the top 50 (the space hogs).

My inclination is to provide something like this:

	df -p		only report on things that provide physical space,
			such as UFS filesystems, ZFS storage pools, and
			swap devices; omit logical consumers of this space,
			such as ZFS filesystems (which use ZFS storage pools)
			and tmpfs filesystems (which use swap space).

	df -t N		only report the top N space consumers

I don''t think we should change the default, because (a) it''s
what you
want on most systems, and (b) it''s what we''ve always done with
tmpfs.
I also don''t think we should make df reporting a settable ZFS property.
Control over what you observe belongs in the tool doing the observing,
and in the hands of the admin using the tool.  Different admins will
want to see different things at different times -- especially if you
have more than one admin on site.

For example, here''s my desktop today (boring lines omitted):

$ df -hl
Filesystem             size   used  avail capacity  Mounted on
swap                   5.2G   1.1M   5.2G     1%    /etc/svc/volatile
swap                   5.9G   718M   5.2G    12%    /tmp
swap                   5.2G    72K   5.2G     1%    /var/run
builds                 7.9G     9K   4.3G     1%    /builds
builds/fixes           7.9G   3.6G   4.3G    46%    /builds/fixes

Here''s what you''d get with df -p:

$ df -hlp
Filesystem             size   used  avail capacity  Mounted on
swap                   5.9G   719M   5.2G    12%    swap
builds                 7.9G   3.6G   4.3G    46%    builds

I''m open to suggestions -- this is just the first thing that came to
mind.
If you have an idea for how to make this better, please speak up.

Thanks,

Jeff

Jeff Bonwick wrote:
> $ df -hlp
> Filesystem             size   used  avail capacity  Mounted on
> swap                   5.9G   719M   5.2G    12%    swap
> builds                 7.9G   3.6G   4.3G    46%    builds
> 
> I''m open to suggestions -- this is just the first thing that came
to mind.
> If you have an idea for how to make this better, please speak up.
Love it thats exactly what I want to see, just the physical
storage first.  If I then have more questions later I can
narrow it down to the appropriate file systems within the
physical storage.

One minor nit IMO the first column shouldn''t be called
"Filesystem"
if "-p" is given instead it should probably be something like
"Storage Pool" which may be 1:1 with a Filesystem if it is UFS
but as you said with tmps and ZFS it isn''t.

I''d also really really like support for "ignore", not just
for
ZFS but also for lofs (to hide those stupid libc lofs mounts!)

For reference on snv_32 running on jurassic (for non Sun people
this is 1.6T of ZFS home dir storage as a big NFS server) it took
5:20.09 (thats 5 minutes 20 seconds) to do a "df -hl | wc -l" there
were
2375 ZFS file systems in that df output.  Doing "zfs list -t filesystems 
| wc -l" took 6:44.30 (6 minutes 44 seconds).

For me the biggest issue is the time it takes to do all the ioctl
calls for the ZFS filesystems rather than the output but fixing
the output would be great too.


-- 
Darren J Moffat

On Mon, 2006-03-06 at 02:32, Jeff Bonwick wrote:
> > When I have 1000s of ZFS filesystems in pools, "df" output
gets
> > spammed by ZFS (and takes a long time to run.)
Typing df (without -l) on a busy multiuser machine where
everyone''s home directory is separately automounted has
done the same for years.
> I believe this is the first time we''ve been called spam.  ;-)
> But I agree that there''s a problem worth solving here.
But where? By changing df, or changing user behaviour?
> My inclination is to provide something like this:
> 
> 	df -p		only report on things that provide physical space,
> 			such as UFS filesystems, ZFS storage pools, and
> 			swap devices; omit logical consumers of this space,
> 			such as ZFS filesystems (which use ZFS storage pools)
> 			and tmpfs filesystems (which use swap space).
This changes the behaviour of df somewhat. What I don''t see is the
justifications for modifying df at all, rather than getting users
to use different tools that show this information.
> 	df -t N		only report the top N space consumers
Top in which sense? Most used? Least free? Largest percentage
used? Each is interesting, depending on the question at the
time.
> I don''t think we should change the default, because (a)
it''s what you
> want on most systems, and (b) it''s what we''ve always done
with tmpfs.
> I also don''t think we should make df reporting a settable ZFS
property.
> Control over what you observe belongs in the tool doing the observing,
> and in the hands of the admin using the tool.  Different admins will
> want to see different things at different times -- especially if you
> have more than one admin on site.
Indeed. I don''t think we should modify df much, if at all.

If it were to be modified, then I would prefer to see better
filtering: the ability to supply multiple -F options, and
the ability to filter out certain filesystem types (maybe
a -X option).
> For example, here''s my desktop today (boring lines omitted):
> 
> $ df -hl
> Filesystem             size   used  avail capacity  Mounted on
> swap                   5.2G   1.1M   5.2G     1%    /etc/svc/volatile
> swap                   5.9G   718M   5.2G    12%    /tmp
> swap                   5.2G    72K   5.2G     1%    /var/run
> builds                 7.9G     9K   4.3G     1%    /builds
> builds/fixes           7.9G   3.6G   4.3G    46%    /builds/fixes
> 
> Here''s what you''d get with df -p:
> 
> $ df -hlp
> Filesystem             size   used  avail capacity  Mounted on
> swap                   5.9G   719M   5.2G    12%    swap
> builds                 7.9G   3.6G   4.3G    46%    builds
This confuses me. Sorry, but builds is a filesystem that has
9K used, and nothing''s mounted on swap at all. I would much rather
use the zfs and swap commands to find out what''s really going on
in those areas than try to subvert df.

-- 
-Peter Tribble
L.I.S., University of Hertfordshire - http://www.herts.ac.uk/
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

Eric Schrock <eric.schrock at sun.com> writes:
> > On an unrelated note, another feature that I''d like to see
would be
> > the ability to assign the mountpoint
> > on a filesystem create.  Something like:
> > 
> > zfs create -o mountpoint=/export/home local/home
> > 
> > That would save a step in the filesystem create where the filesystem  
> > isn''t using the default hierarchy.
> 
> Yes, there is an RFE open for this - I don''t have the bug ID
handy...
Perhaps this could be

	zfs create -m <mountpoint> <zfs>

for consistency with

	zpool create -m <mountpoint> ...

And just for good measure, add

	zfs create -p <zfs>

to behave like mkdir -p :-)

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

On Mon, Mar 06, 2006 at 08:27:03PM +0100, Rainer Orth
wrote:
> And just for good measure, add
> 
> 	zfs create -p <zfs>
> 
> to behave like mkdir -p :-)
Yes!  OR, not having to have every directory in the path be a ZFS
filesystem.  This was a rude awakening the first time I tried to create
a ZFS filesystem not in the root directory of another.

Rainer Orth wrote:
> And just for good measure, add
> 
> 	zfs create -p <zfs>
> 
> to behave like mkdir -p :-)
I think this is probably a good idea, but what would the exact semantics 
be?  Would the intermediate nodes that are implicitly created be 
directories or file systems?  For example, if /foo is an existing, empty 
ZFS file system, and I ussue the command

	zfs create -p /foo/bar/baz

It''s clear that "baz" should be a file system, but what is
"bar"?

	--Ed
-- 
Ed Gould                Sun Microsystems
File System Architect   Sun Cluster
ed.gould at sun.com        17 Network Circle
+1.650.786.4937         M/S UMPK17-201
x84937                  Menlo Park, CA  94025

On Mon, Mar 06, 2006 at 11:48:41AM -0800, Ed Gould
wrote:
> Rainer Orth wrote:
> >And just for good measure, add
> >
> >	zfs create -p <zfs>
> >
> >to behave like mkdir -p :-)
> 
> I think this is probably a good idea, but what would the exact semantics 
> be?  Would the intermediate nodes that are implicitly created be 
> directories or file systems?  For example, if /foo is an existing, empty 
> ZFS file system, and I ussue the command
> 
> 	zfs create -p /foo/bar/baz
                      ^
		      no slash there :)
> It''s clear that "baz" should be a file system, but what
is "bar"?
That''s the problem, ZFS currently requires foo and foo/bar, in this
example, both be the root directories of ZFS filesystems; irksome.

Nico
--

Ed Gould writes:
> I think this is probably a good idea, but what would the exact semantics 
> be?  Would the intermediate nodes that are implicitly created be 
> directories or file systems?  For example, if /foo is an existing, empty 
> ZFS file system, and I ussue the command
> 
> 	zfs create -p /foo/bar/baz
> 
> It''s clear that "baz" should be a file system, but what
is "bar"?
I think it would have to be a filesystem.  I would expect something like

	zpool create export <vdefs>
	zfs create -p export/opt/SUNWjass
	zfs create export/opt/SUNWjet

to work, just as it would with mkdir -p.  If export/opt were not an ZFS
filesystem, the second zfs create would fail.

Although I agree with e.g. Peter Tribble that the current situation of
having ZFS filesystems like export/opt when you don''t intend to store
any
data in them, but just need them as containers for actual filesystems, is
a bit confusing, especially since you e.g. have to NFS-export them if you
want to make use of attribute inheritance.

	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

On Mon, Mar 06, 2006 at 08:55:50PM +0100, Rainer Orth
wrote:
> Ed Gould writes:
> 
> > I think this is probably a good idea, but what would the exact
semantics
> > be?  Would the intermediate nodes that are implicitly created be 
> > directories or file systems?  For example, if /foo is an existing,
empty
> > ZFS file system, and I ussue the command
> > 
> > 	zfs create -p /foo/bar/baz
> > 
> > It''s clear that "baz" should be a file system, but
what is "bar"?
There is an RFE open for this:

6290249 zfs {create,clone,rename} -p
> I think it would have to be a filesystem.  I would expect something like
> 
> 	zpool create export <vdefs>
> 	zfs create -p export/opt/SUNWjass
> 	zfs create export/opt/SUNWjet
> 
> to work, just as it would with mkdir -p.  If export/opt were not an ZFS
> filesystem, the second zfs create would fail.
> 
> Although I agree with e.g. Peter Tribble that the current situation of
> having ZFS filesystems like export/opt when you don''t intend to
store any
> data in them, but just need them as containers for actual filesystems, is
> a bit confusing, especially since you e.g. have to NFS-export them if you
> want to make use of attribute inheritance.
There is an RFE open for this:

6366244 ''nomount'' option for container-like functionality

Hope that helps.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Hmmm, while I think about responding some more, let me present a
use-case scenario that will be moderately common in many organisations.

I''m a systems administrator and I look after a collection of systems,
maybe I''m lucky and only have Solaris on SPARC but more than likely
not.

Every night (or maybe more often), I have something that runs through
all of my systems and prepares a disk usage report for me the next
morning.  For Solaris sytems it uses "df -kl" (or some variation
thereof)
to report on all the local filesystems.

The primary piece of information I''m concerned about is how much disk
space is used in a partition and when a partition crosses the (say) 80%
line.  At that point I know I need to start planning to add more disk space.

Enter Solaris10 with ZFS.

Preferentially I don''t want to have to change my Solaris scripts or
change
my setup such that it treats Solaris 10 systems in any way differently to
what it does Solaris 8 and 9.  What I want to do is add my new Solaris 10
system to the list of hosts I care about and that''s about it.  I might
also
have some sort of timeout built into the data generation so that if it takes
more than 30 seconds, it is aborted and an alarm generated.

If I do this today, the first report I get for my Solaris 10 system is going
to look something like this:

Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1d0s0      14120763 7967233 6012323    57%    /
/devices                   0       0       0     0%    /devices
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                 2737664     644 2737020     1%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
/usr/lib/libc/libc_hwcap2.so.1
                     14120763 7967233 6012323    57%    /lib/libc.so.1
fd                         0       0       0     0%    /dev/fd
/dev/dsk/c1d0s5      6051294  255543 5735239     5%    /var
swap                 2742328    5308 2737020     1%    /tmp
swap                 2737052      32 2737020     1%    /var/run
/dev/dsk/c1d0s6      218237965 4003947 212051639     2%    /export
/dev/dsk/c1d0p1        56068       0   56068     0%    /diagpart
data/home         210983182 2124514 208858668     2%    /zfs/data/home

and I''m happy with this.  However, if I start adding snapshots, the
picture starts to get murky but that''s another issue...

Now if I change the model of how I use ZFS to something like jurassic,
I get the above UFS plus the below in df output:

Filesystem            kbytes    used   avail capacity  Mounted on
...
tank                 kkkkkkkkkk    tttt yyyyyyyyy     1%    /tank
tank/home            kkkkkkkkkk hhhhhhhhh yyyyyyyyy     2%    /tank/home
tank/dump            kkkkkkkkkk ddddddddd yyyyyyyyy    10%    /tank/dump
tank/misc            kkkkkkkkkk      mm yyyyyyyyy     1%    /tank/misc
tank/home/A          kkkkkkkkkk     AAA yyyyyyyyy     1%    /tank/home/A
tank/home/A/e        kkkkkkkkkk   eeeee zzzzzzzzz     1%    /tank/home/A/e
tank/home/A/b        kkkkkkkkkk    bbbb zzzzzzzzz     1%    /tank/home/A/b
tank/home/A/c        kkkkkkkkkk    cccc zzzzzzzzz     1%    /tank/home/A/c

Now in terms of reporting disk space being used, etc, the numbers that
I''m
concerned about are kkkkkkkkkk and yyyyyyyyy/zzzzzzzzz.  For the purpose of
reporting and planning, I don''t need to see ''n'' lines
of output (one for
each
filesystem in the pool), just one that tells me if the pool is short on 
space
or not.  The only catch here is it might look a bit strange to see:

Filesystem            kbytes    used   avail capacity  Mounted on
...
tank                 kkkkkkkkkk    1000 1000    99%    /tank

as we''re used to seeing "kbytes = used + avail".

While I agree that different admins may want to observe filesystems in a
different manner, that doesn''t preclude a "site" having a
default manner
in which to present data and different "sites" choosing different
defaults
and maybe not all admins agree on it but as a professional you work with it.

To draw this discussion back to where it started (I''ve rambled a bit 
here :),
when we''re creating lots of ZFS filesystems in a pool (or pools), the
output
from df becomes increasingly less useful and there is no way to control 
this.

The idea of "df -p" only reporting physical data usage is a reasonable
idea to suggest and the example you presented was quite worthwhile.

However, I would like to ask that you consider adding a mount option or 
zpool
option or something that causes df to not report each individual filesystem
in a zpool because with some extra administration steps it means that 
Solaris
systems with ZFS can more easily be integrated into a mixed Solaris 
environment.

On a side not, a problem that may be related to performance, is that in
stepping through the filesystems above, "home/A" goes from yyyyyyyyy
down
to zzzzzzzzz for "home/A/e".  To my eyes this makes it a bit confusing
because it suggests that somehow A/e is different to A but it isn''t,
really.

Darren

On Mon, Mar 06, 2006 at 01:47:43PM -0800, Darren Reed
wrote:
> 
> and I''m happy with this.  However, if I start adding snapshots,
the
> picture starts to get murky but that''s another issue...
I assume you mean ZFS snapshots.  Snapshots should not show up in
''df'',
as they don''t even show up in /etc/mnttab.  If you see something
otherwise, please let us know.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Eric Schrock writes:
> There is an RFE open for this:
> 
> 6290249 zfs {create,clone,rename} -p
[...]
> There is an RFE open for this:
> 
> 6366244 ''nomount'' option for container-like functionality
> 
> Hope that helps.
Indeed: both RFEs seem very useful.

Thanks.
	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

On Mon, Mar 06, 2006 at 02:00:42PM -0800, Eric Schrock
wrote:
> On Mon, Mar 06, 2006 at 01:47:43PM -0800, Darren Reed wrote:
> > 
> > and I''m happy with this.  However, if I start adding
snapshots, the
> > picture starts to get murky but that''s another issue...
> 
> I assume you mean ZFS snapshots.  Snapshots should not show up in
''df'',
> as they don''t even show up in /etc/mnttab.  If you see something
> otherwise, please let us know.
I figured he meant clones -- df certainly isn''t smart enough to show
how
much actual space several ZFS filesystems related by cloning actually
consume.

Do I want to know how space a *user* is consuming?  No, ZFS doesn''t
really track that.

Or how much space a *filesystem* (*and* its snapshots and clones and
snapshots and clones thereof, recursively) takes up?  Yes, I think
that''s what I''d want.  And, of course, how much space is left
given
whatever quota is set (hey, percentages make a nice UI, particularly for
the *user*, as opposed to the sysadmin).

And separately I''d want to know how full a volume is.

Nico
--

On 03/06/06 02:32, Jeff Bonwick wrote:
> My inclination is to provide something like this:
> 
> 	df -p		only report on things that provide physical space,
> 			such as UFS filesystems, ZFS storage pools, and
> 			swap devices; omit logical consumers of this space,
> 			such as ZFS filesystems (which use ZFS storage pools)
> 			and tmpfs filesystems (which use swap space).
> 
> 	df -t N		only report the top N space consumers
I like those.  I''d also like a ''df -x'' (along the
lines of ''svcs -x''_ to list
only those items that may be a problem, such as being nearly full or over quota.
The thresholds could have sensible defaults and be influenced by env variables.

Gavin

Eric Schrock <eric.schrock at sun.com> wrote:
> I guess I''m just unclear on what the user is actually trying to
get from
> df (with no arguments).  Certainly in the world of ZFS root, I would
> imagine df(1) becoming obsolete pretty quickly.
Why do you believe a commonly used program would become obsolete?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

Jeff Bonwick <bonwick at zion.eng.sun.com> wrote:
> > When I have 1000s of ZFS filesystems in pools, "df" output
gets
> > spammed by ZFS (and takes a long time to run.)
>
> I believe this is the first time we''ve been called spam.  ;-)
> But I agree that there''s a problem worth solving here.
BTW: will there be some kind of fix for the repeated

Feb  7 16:48:14 opt pseudo: [ID 129642 kern.notice] pseudo-device: zfs0
Feb  7 16:48:14 opt genunix: [ID 936769 kern.notice] zfs0 is /pseudo/zfs at 0

kernel messages on the console?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

Yes.  These messages only appear if you are running df(1) without any
ZFS filesystems.  Regardless, this has been fixed in build 36.

- Eric

On Tue, Mar 07, 2006 at 06:19:31PM +0100, Joerg Schilling
wrote:
> Jeff Bonwick <bonwick at zion.eng.sun.com> wrote:
> 
> > > When I have 1000s of ZFS filesystems in pools, "df"
output gets
> > > spammed by ZFS (and takes a long time to run.)
> >
> > I believe this is the first time we''ve been called spam.  ;-)
> > But I agree that there''s a problem worth solving here.
> 
> BTW: will there be some kind of fix for the repeated
> 
> Feb  7 16:48:14 opt pseudo: [ID 129642 kern.notice] pseudo-device: zfs0
> Feb  7 16:48:14 opt genunix: [ID 936769 kern.notice] zfs0 is /pseudo/zfs at
0
> 
> kernel messages on the console?
> 
> J?rg
> 
> -- 
>  EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353
Berlin
>        js at cs.tu-berlin.de                (uni)  
>        schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
>  URL:  http://cdrecord.berlios.de/old/private/
ftp://ftp.berlios.de/pub/schily
--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock