zfs discuss - May 2011 - Disk space size, used, available mismatch

Thank you for your insight.  This is a system that was handed down to me when
another sysadmin went to greener pastures.  There were no quotas set on the
system.  I used zfs destroy to free up some space and did put a quota on it.  I
still have 0 freespace available.  I think this is due to the quota limit. 
Before I rebooted I had about a 68GB bootpool.  After the zfs destroy I had
about 1.7GB free.  I put a 66.5 GB quota on it which I am hitting so services
will not start up.

I don''t want to saw off the tree branch I am sitting on so I am
reluctant to increase the quota too much.  Here are some questions I have:

1) zfs destroy did free up a snapshot but it is still showing up in lustatus. 
How to I correct this?

2) This system is installed with everything under / so the ETL team can fill up
root with out bounds.  What are the best practices for separating filesystems in
ZFS so I can bound the ETL team with out affecting the OS?

3) I have captured all the critical data on to SAN disk and am thinking about
jumpstarting the host cleanly.  That way I will have a known baseline to start
with.  Does anyone have any suggestions here?

4) We deal with very large data sets.  These usually exist just in Oracle but
this host is for ETL and Informatica processing.  What would be a good quota to
set so I have a back door onto the system to take care of problems.

Thanks for your feedback.
-- 
This message posted from opensolaris.org

Well, the Sun-supported way is separating /var from the common root.

In our systems we do a more fine-tuned hierarchy of separated /usr /opt /var in
each BE and sub-separated /var/adm /var/log /var/cores /var/crash and /var/mail
shared between boot environments. This requires quite many tricks to set up, and
often is apain to maintain with i.e. LiveUpgrade (often the system
doesn''t come up on first reboot after update, because something was
mixed up - luckily these boxes have remote consoles). However this also allows
quota''ing specific datasets, i.e. so that core-dumps don''t eat
up the whole root FS.

In your case it might make sense to separate the application software''s
paths (i.e. /opt/programname/ and /var/opt/programname/) to individual datasets
with quotas and migrate the data via cpio...

LU does not fetch system information straight from the system itself, it
consults its own configuration (with copies in each boot environment). See
/etc/lutab and /etc/lu/ICF.* files (and other /etc/lu/*) but beware that manual
mangling is not supported by Sun. Not that it does not work or help in some
cases ;)

A common approach is to have a separate root pool (slice or better a mirror of
two slices), and depending on your base OS installation footprint, anywhere from
4Gb (no graphics) to 20Gb (enough for several BE revisions) would do. Remainder
of the disk is given to a separate data pool, where our local zone roots live,
as well as distros, backup data, etc. Thus there is very little third-party
software installed in the global zones which act more like hypervisors to many
local zones with actual applications and server software.

You might want or not want to keep the swap and dump volumes in the root pool -
this extends the limit according to your RAM size. For example, on boxes with 4
disks we can use one 2*20Gb mirror as a root pool, and another 2*20Gb mirror as
a pool for swap and dump, leaving equal amounts of disk for the separate data
pool.

In case of 4 slices for the data pool you can select to make it a RAID10, RAIDZ1
or RAIDZ2 - with different processing overheads/performance and different
redundancy levels and different available space. For demo boxes with no critical
performance requirements we use RAIDZ2 as it protects from failure of any two
disks while RAID10 protects from failure of two specific disks (from different
mirrors), or RAIDZ1 when we need more space available.

HTH,
//Jim Klimov
-- 
This message posted from opensolaris.org

Ah, yes, regarding the "backdoor to root fs": if you select to have
some not-quoata''ed space hogs in the same pool as your root FS, you can
look into setting a reservation (and/or refreservation) for the root FS
datasets. For example, if your OS installation uses 4Gb and you don''t
think it would exceed 6Gb by itself (and your ETL software uses a separate
dataset), you can reserve 6Gb for only the root FS (and hopefully its
descendants - but better see manpages):

# zfs set reservation=6G rpool/ROOT/myBeName

HTH,
//Jim
-- 
This message posted from opensolaris.org