Wolfgang Pungartnik
2008-Nov-11 08:48 UTC
[zfs-discuss] Q. ZFS requires freespace in file system ?
Hi, i recently encountered a database crash on ZFS with MaxDB and Solaris 10. The reason was that the ZFS file system was 100% full (quota, reservation set and exceeded). Unfortunately using "df -k" and counting the bytes only 860GB out of 870GB were used for files. No hidden files, no ZFS snapshots or anything else i could see. The big question is who''s using the missing 10GB. Does ZFS requires a reserved amount of space per file system ? I know that due to COW ZFS requires a little bit more spaces until the update is done. But we are still missing 10GB. Any ideas ? Thx. ++++++++++++++++++++++++++++++ sapph10:ph1adm 10% ls -laR .: total 1824522213 drwxr-xr-x 2 sdb sdba 45 Sep 5 17:10 . drwxrwxr-x 6 sdb sdba 11 Jul 2 09:05 .. -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0001 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0002 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0003 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0004 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0005 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0006 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0007 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0008 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0009 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0010 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0011 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0012 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0013 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0014 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0015 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0016 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0017 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0018 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0019 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0020 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0021 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0022 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0023 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0024 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0025 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0026 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0027 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0028 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0029 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0030 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0031 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0032 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0033 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0034 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0035 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0036 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0037 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0038 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0039 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0040 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0041 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0042 -rw-rw---- 1 sdb sdba 21474844672 Sep 30 09:47 DISKD0043 sapph10:ph1adm 12% df -k Filesystem kbytes used avail capacity Mounted on / 976367 278453 688151 29% / /dev 976367 278453 688151 29% /dev /lib 15493250 4193500 11144818 28% /lib /platform 15493250 4193500 11144818 28% /platform /sbin 15493250 4193500 11144818 28% /sbin /usr 15493250 4193500 11144818 28% /usr proc 0 0 0 0% /proc ctfs 0 0 0 0% /system/contract mnttab 0 0 0 0% /etc/mnttab objfs 0 0 0 0% /system/object swap 120834608 328 120834280 1% /etc/svc/volatile fd 0 0 0 0% /dev/fd swap 1048576 48 1048528 1% /tmp swap 120834320 40 120834280 1% /var/run ph1_sap 1023934464 24 106873770 1% /ph1_sap ph1_saplog 51351552 24 25859951 1% /ph1_saplog ph1_sap/sapdb 2097152 2097236 0 100% /sapdb ph1_sap/ph1 4194304 151045 4043259 4% /sapdb/PH1 ph1_sap/sapdata1 912261120 912261120 0 100% /sapdb/PH1/sapdata1 ph1_saplog/saplog 51351552 25488279 25859951 50% /sapdb/PH1/saplog ph1_sap/sapmnt 8388608 1479457 6909151 18% /sapmnt ph1_sap/usrlocal 102400 575 101825 1% /usr/local ph1_sap/usrsap 3145728 900486 2245242 29% /usr/sap root at sunsrv2 # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT ph1_sap 992G 875G 117G 88% ONLINE - ph1_saplog 49.8G 24.3G 25.4G 48% ONLINE - root at sunsrv2 # zfs list NAME USED AVAIL REFER MOUNTPOINT ph1_sap 875G 102G 24.5K /ph1_sap ph1_sap/ph1 148M 3.86G 148M /sapdb/PH1 ph1_sap/sapdata1 870G 0 870G /sapdb/PH1/sapdata1 ph1_sap/sapdb 2.00G 0 2.00G /sapdb ph1_sap/sapmnt 1.41G 6.59G 1.41G /sapmnt ph1_sap/usrlocal 575K 99.4M 575K /usr/local ph1_sap/usrsap 879M 2.14G 879M /usr/sap ph1_saplog 24.3G 24.7G 24.5K /ph1_saplog ph1_saplog/saplog 24.3G 24.7G 24.3G /sapdb/PH1/saplog root at sunsrv2 # zfs get all ph1_sap/sapdata1 NAME PROPERTY VALUE SOURCE ph1_sap/sapdata1 type filesystem - ph1_sap/sapdata1 creation Tue May 20 14:15 2008 - ph1_sap/sapdata1 used 870G - ph1_sap/sapdata1 available 0 - ph1_sap/sapdata1 referenced 870G - ph1_sap/sapdata1 compressratio 1.00x - ph1_sap/sapdata1 mounted yes - ph1_sap/sapdata1 quota 870G local ph1_sap/sapdata1 reservation 870G local ph1_sap/sapdata1 recordsize 8K local ph1_sap/sapdata1 mountpoint /sapdb/PH1/sapdata1 local ph1_sap/sapdata1 sharenfs off default ph1_sap/sapdata1 checksum on default ph1_sap/sapdata1 compression off default ph1_sap/sapdata1 atime on default ph1_sap/sapdata1 devices off temporary ph1_sap/sapdata1 exec on default ph1_sap/sapdata1 setuid on default ph1_sap/sapdata1 readonly off default ph1_sap/sapdata1 zoned on inherited from ph1_sap ph1_sap/sapdata1 snapdir hidden default ph1_sap/sapdata1 aclmode groupmask default ph1_sap/sapdata1 aclinherit secure default ph1_sap/sapdata1 canmount on default ph1_sap/sapdata1 shareiscsi off default ph1_sap/sapdata1 xattr on default root at sunsrv2 # zfs set quota=880G ph1_sap/sapdata1 root at sunsrv2 # zfs set reservation=880G ph1_sap/sapdata1 root at sunsrv2 # zfs get all ph1_sap/sapdb NAME PROPERTY VALUE SOURCE ph1_sap/sapdb type filesystem - ph1_sap/sapdb creation Tue May 20 14:13 2008 - ph1_sap/sapdb used 2.00G - ph1_sap/sapdb available 0 - ph1_sap/sapdb referenced 2.00G - ph1_sap/sapdb compressratio 1.00x - ph1_sap/sapdb mounted yes - ph1_sap/sapdb quota 2G local ph1_sap/sapdb reservation none default ph1_sap/sapdb recordsize 128K default ph1_sap/sapdb mountpoint /sapdb local ph1_sap/sapdb sharenfs off default ph1_sap/sapdb checksum on default ph1_sap/sapdb compression off default ph1_sap/sapdb atime on default ph1_sap/sapdb devices off temporary ph1_sap/sapdb exec on default ph1_sap/sapdb setuid on default ph1_sap/sapdb readonly off default ph1_sap/sapdb zoned on inherited from ph1_sap ph1_sap/sapdb snapdir hidden default ph1_sap/sapdb aclmode groupmask default ph1_sap/sapdb aclinherit secure default ph1_sap/sapdb canmount on default ph1_sap/sapdb shareiscsi off default ph1_sap/sapdb xattr on default root at sunsrv2 # zfs set quota=4G ph1_sap/sapdb root at sunsrv2 # zfs list NAME USED AVAIL REFER MOUNTPOINT ph1_sap 885G 91.9G 24.5K /ph1_sap ph1_sap/ph1 148M 3.86G 148M /sapdb/PH1 ph1_sap/sapdata1 870G 10.0G 870G /sapdb/PH1/sapdata1 ph1_sap/sapdb 2.00G 2.00G 2.00G /sapdb ph1_sap/sapmnt 1.41G 6.59G 1.41G /sapmnt ph1_sap/usrlocal 575K 99.4M 575K /usr/local ph1_sap/usrsap 879M 2.14G 879M /usr/sap ph1_saplog 24.3G 24.7G 24.5K /ph1_saplog ph1_saplog/saplog 24.3G 24.7G 24.3G /sapdb/PH1/saplog -- Mit freundlichen Gruessen / Kind Regards ***************************************************************************** Wolfgang Pungartnik Technical Specialist Tel: (++49 6103) 752-391 Professional Services Delivery Fax: (++49 6103) 752-167 Sun Microsystems GmbH Mobile: (++49 172) 8336949 Amperestrasse 6 mailto:wolfgang.pungartnik at sun.com 63225 Langen http://www.sun.de/ ***************************************************************************** Sun Microsystems GmbH Sitz der Gesellschaft: Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht M?nchen: HRB 161028 Gesch?ftsf?hrer: Thomas Schr?der, Wolfgang Engels, Dr. Roland B?mer Vorsitzender des Aufsichtsrates: Martin H?ring ***************************************************************************** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ BITTE BEACHTEN: Diese Email ist nur zur Verwendung durch den/die beabsichtigten Empfaenger bestimmt und kann geheime und/oder vertrauliche Informationen enthalten. Unbefugtes Lesen sowie unbefugte Nutzung, Weitergabe oder Verbreitung des Inhalts dieser Email sind nicht gestattet. Wenn Sie nicht der beabsichtigte Empfaenger dieser Email sind, informieren Sie hierueber bitte den Absender und vernichten Sie bitte alle Kopien der urspruenglichen Email. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~