James C. McPherson
2006-May-01 00:26 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
Hi all,
I got a new 300Gb SATA disk last friday for my u20. After figuring
out that the current u20 bios barfs on EFI labels and creating a
whole-disk-sized slice0 I used TimF''s script to implement zfsroot.
All seemed well and good until this morning when I did a df:
pieces: $ df -k /
Filesystem kbytes used avail capacity Mounted on
root_pool/root_filesystem
286949376 183288496 103652571 64% /
Now try as I might, I''m unable to account for more than about 26Gb
on this disk.
Where has all my space gone?
pieces: $ uname -a ; modinfo |grep -i zfs
SunOS pieces 5.11 snv_38 i86pc i386 i86pc
30 fffffffff3a55000 6b2d0 8 1 zfs (ZFS filesystem version 1)
30 fffffffff3a55000 6b2d0 182 1 zfs (ZFS storage pool version 1)
pieces: $ zdb -b root_pool
Traversing all blocks to verify nothing leaked ...
prstat
No leaks (block sum matches space maps exactly)
bp count: 2464943
bp logical: 192108967936 avg: 77936
bp physical: 186591301632 avg: 75698 compression: 1.03
bp allocated: 187156245504 avg: 75927 compression: 1.03
SPA allocated: 187156245504 used: 62.70%
pieces: $ zpool status
pool: root_pool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
root_pool ONLINE 0 0 0
c2d0s0 ONLINE 0 0 0
errors: No known data errors
pieces: $ zfs list -o
used,available,referenced,compressratio,volsize,volblocksize,recordsize,compression,devices,
USED AVAIL REFER RATIO VOLSIZE VOLBLOCK RECSIZE COMPRESS DEVICES
175G 98.3G 24.5K 1.00x - - 128K off on
175G 98.3G 175G 1.00x - - 128K off on
What should I be looking into to figure this out? Should I upgrade
from zfs on-disk format version 1 to the newer version?
thanks in advance,
James C. McPherson
--
Solaris Datapath Engineering
Data Management Group
Sun Microsystems
Matthew Ahrens
2006-May-01 06:59 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
On Mon, May 01, 2006 at 10:26:33AM +1000, James C. McPherson wrote:> I got a new 300Gb SATA disk last friday for my u20. After figuring > out that the current u20 bios barfs on EFI labels and creating a > whole-disk-sized slice0 I used TimF''s script to implement zfsroot.To start with, please send us the output of: $ zfs list $ zfs get -r all <pool> $ zdb -bbb <pool> If it seems that a particular filesystem is using more than you expect, try using ''du'' to determine if the space is used according to that (try ''du -hds <mountpoint>''). If the ''du'' size is close to the ''zfs list'' size, then you can continue using ''du'' to figure out which directory is using more space than you expect. If the ''du'' size is much less than the ''zfs list'' size, then hopefully the ''zdb -bbb'' output will give us an idea of where the space has gone. Also check the console or system log to see if there is any unusual output. Also try ''zfs unmount -a; zfs mount -a'' (but in your case with zfs root, you will need to reboot instead), then see if the space used goes down after a minute. thanks, --matt
James C. McPherson
2006-May-01 08:03 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
Matthew Ahrens wrote:> On Mon, May 01, 2006 at 10:26:33AM +1000, James C. McPherson wrote: >> I got a new 300Gb SATA disk last friday for my u20. After figuring >> out that the current u20 bios barfs on EFI labels and creating a >> whole-disk-sized slice0 I used TimF''s script to implement zfsroot. > To start with, please send us the output of: > $ zfs list > $ zfs get -r all <pool> > $ zdb -bbb <pool>pieces:jmcp $ zfs list NAME USED AVAIL REFER MOUNTPOINT root_pool 179G 94.6G 24.5K /root_pool root_pool/root_filesystem 179G 94.6G 179G legacy pieces:jmcp $ zfs get -r all root_pool NAME PROPERTY VALUE SOURCE root_pool type filesystem - root_pool creation Fri Apr 28 13:57 2006 - root_pool used 179G - root_pool available 94.6G - root_pool referenced 24.5K - root_pool compressratio 1.06x - root_pool mounted yes - root_pool quota none default root_pool reservation none default root_pool recordsize 128K default root_pool mountpoint /root_pool default root_pool sharenfs off default root_pool checksum on default root_pool compression lzjb local root_pool atime on default root_pool devices on default root_pool exec on default root_pool setuid on default root_pool readonly off default root_pool zoned off default root_pool snapdir hidden default root_pool aclmode groupmask default root_pool aclinherit secure default root_pool/root_filesystem type filesystem - root_pool/root_filesystem creation Fri Apr 28 13:57 2006 - root_pool/root_filesystem used 179G - root_pool/root_filesystem available 94.6G - root_pool/root_filesystem referenced 179G - root_pool/root_filesystem compressratio 1.06x - root_pool/root_filesystem mounted yes - root_pool/root_filesystem quota none default root_pool/root_filesystem reservation none default root_pool/root_filesystem recordsize 128K default root_pool/root_filesystem mountpoint legacy local root_pool/root_filesystem sharenfs off default root_pool/root_filesystem checksum on default root_pool/root_filesystem compression lzjb inherited from root_pool root_pool/root_filesystem atime off temporary root_pool/root_filesystem devices on default root_pool/root_filesystem exec on default root_pool/root_filesystem setuid on default root_pool/root_filesystem readonly off default root_pool/root_filesystem zoned off default root_pool/root_filesystem snapdir hidden default root_pool/root_filesystem aclmode groupmask default root_pool/root_filesystem aclinherit secure default pieces:jmcp $ zdb -bbb root_pool Traversing all blocks to verify nothing leaked ... No leaks (block sum matches space maps exactly) bp count: 2652179 bp logical: 211064878080 avg: 79581 bp physical: 191661369344 avg: 72265 compression: 1.10 bp allocated: 192270130176 avg: 72495 compression: 1.10 SPA allocated: 192270130176 used: 64.41% Blocks LSIZE PSIZE ASIZE avg comp %Total Type 3 48.0K 7.00K 21.0K 7.00K 6.86 0.00 L1 deferred free 4 28.0K 9.00K 20.5K 5.12K 3.11 0.00 L0 deferred free 7 76.0K 16K 41.5K 5.93K 4.75 0.00 deferred free 1 512 512 1K 1K 1.00 0.00 object directory 3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array 1 16K 1K 2K 2K 16.00 0.00 packed nvlist - - - - - - - packed nvlist size 1 16K 16K 32K 32K 1.00 0.00 bplist - - - - - - - bplist header - - - - - - - SPA space map header 97 1.52M 158K 473K 4.87K 9.85 0.00 L1 SPA space map 1.50K 6.02M 4.48M 8.95M 5.95K 1.34 0.00 L0 SPA space map 1.60K 7.54M 4.63M 9.42M 5.89K 1.63 0.01 SPA space map 1 20.0K 20.0K 20.0K 20.0K 1.00 0.00 ZIL intent log 2 32K 2K 6.00K 3.00K 16.00 0.00 L6 DMU dnode 2 32K 2K 6.00K 3.00K 16.00 0.00 L5 DMU dnode 2 32K 2K 6.00K 3.00K 16.00 0.00 L4 DMU dnode 2 32K 2K 6.00K 3.00K 16.00 0.00 L3 DMU dnode 7 112K 22.5K 67.5K 9.6K 4.98 0.00 L2 DMU dnode 290 4.53M 1.90M 5.70M 20.1K 2.39 0.00 L1 DMU dnode 35.2K 564M 144M 288M 8.18K 3.91 0.16 L0 DMU dnode 35.5K 569M 146M 294M 8.28K 3.89 0.16 DMU dnode 3 3.00K 1.50K 4.50K 1.50K 2.00 0.00 DMU objset - - - - - - - DSL directory 3 1.50K 1.50K 3.00K 1K 1.00 0.00 DSL directory child map 2 1K 1K 2K 1K 1.00 0.00 DSL dataset snap map 4 33.0K 4.50K 9.00K 2.25K 7.33 0.00 DSL props - - - - - - - DSL dataset - - - - - - - ZFS znode - - - - - - - ZFS ACL 293 4.58M 308K 616K 2.10K 15.22 0.00 L2 ZFS plain file 353K 5.51G 358M 716M 2.03K 15.75 0.39 L1 ZFS plain file 2.05M 190G 178G 178G 86.8K 1.07 99.37 L0 ZFS plain file 2.39M 196G 178G 179G 74.6K 1.10 99.76 ZFS plain file 242 3.78M 246K 738K 3.05K 15.74 0.00 L1 ZFS directory 99.4K 93.4M 54.7M 109M 1.10K 1.71 0.06 L0 ZFS directory 100K 97.2M 55.0M 110M 1.11K 1.77 0.06 ZFS directory 2 1K 1K 2K 1K 1.00 0.00 ZFS master node 1 16K 1.50K 4.50K 4.50K 10.67 0.00 L2 ZFS delete queue 17 272K 121K 362K 21.3K 2.26 0.00 L1 ZFS delete queue 2.01K 32.2M 13.7M 27.4M 13.6K 2.35 0.01 L0 ZFS delete queue 2.03K 32.5M 13.8M 27.7M 13.7K 2.35 0.02 ZFS delete queue - - - - - - - zvol object - - - - - - - zvol prop - - - - - - - other uint8[] - - - - - - - other uint64[] - - - - - - - other ZAP - - - - - - - persistent error log 2 32K 2K 6.00K 3.00K 16.00 0.00 L6 Total 2 32K 2K 6.00K 3.00K 16.00 0.00 L5 Total 2 32K 2K 6.00K 3.00K 16.00 0.00 L4 Total 2 32K 2K 6.00K 3.00K 16.00 0.00 L3 Total 301 4.70M 332K 688K 2.29K 14.51 0.00 L2 Total 353K 5.52G 361M 724M 2.05K 15.67 0.39 L1 Total 2.18M 191G 178G 178G 81.7K 1.07 99.60 L0 Total 2.53M 197G 178G 179G 70.8K 1.10 100.00 Total> If it seems that a particular filesystem is using more than you expect, > try using ''du'' to determine if the space is used according to that (try > ''du -hds <mountpoint>''). If the ''du'' size is close to the ''zfs list'' > size, then you can continue using ''du'' to figure out which directory is > using more space than you expect. If the ''du'' size is much less than > the ''zfs list'' size, then hopefully the ''zdb -bbb'' output will give us > an idea of where the space has gone.Aye, there''s the rub - I ran du from / over all my directories, and (excluding /net, /home, /vol and /ws) I didn''t get anywhere near the 179Gb in use. This is basically a fresh install of build 38, with source and local utils copied across from my laptop -- where on ufs they don''t take up anything like that amount of space (on pieces, the u20): pieces:src $ egrep "zfs|ufs" $_ root_pool/root_filesystem / zfs noatime,dev=2d90002 1146459292 /dev/dsk/c1d0s0 /ufsroot ufs rw,intr,largefiles,logging,xattr,onerror=panic,dev=1980000 1146459300 root_pool /root_pool zfs rw,devices,setuid,exec,atime,dev=2d90003 1146459301 pieces:src $ df -k / /ufsroot /root_pool Filesystem kbytes used avail capacity Mounted on root_pool/root_filesystem 286949376 188019183 98920534 66% / /dev/dsk/c1d0s0 8266719 6938082 1245970 85% /ufsroot root_pool 286949376 24 98920534 1% /root_pool 1.6G /export 10G /opt 7.2G /scratch on broken, my laptop: 3.3G /export (has another 1.6gb of photos) 11G /opt 8.6G /scratch> Also check the console or system log to see if there is any unusual > output.There hasn''t been anything on the console> Also try ''zfs unmount -a; zfs mount -a'' (but in your case with zfs root, > you will need to reboot instead), then see if the space used goes down > after a minute.did that too, no joy :( thanks for your help, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Nicolas Williams
2006-May-01 14:22 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
On Mon, May 01, 2006 at 10:26:33AM +1000, James C. McPherson wrote:> All seemed well and good until this morning when I did a df:Is there a large unlinked file in there, perhaps? Aside: What is the best way to find those nowadays? pfiles(1) probably isn''t it since it doesn''t include a link count in its output (an RFE?[*]). lsof works pretty well for finding processes with open unlinked files, but probably does not work on Nevada (haven''t tried it). I''ve filed: 6420048 pfiles should print link count [0] Relevant RFEs: 6183481 pfiles could easily print file offset for files 4286979 Solaris should include a utility like "lsof" Nico --
Matthew Ahrens
2006-May-01 15:08 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
On Mon, May 01, 2006 at 06:03:29PM +1000, James C. McPherson wrote:> Blocks LSIZE PSIZE ASIZE avg comp %Total Type > 4 28.0K 9.00K 20.5K 5.12K 3.11 0.00 L0 deferred free > 2.05M 190G 178G 178G 86.8K 1.07 99.37 L0 ZFS plain file > 2.01K 32.2M 13.7M 27.4M 13.6K 2.35 0.01 L0 ZFS delete queThis tells us that we didn''t waste your space on metadata (since it''s >99% plain file contents). However, there is a suspiciously large amount of space used by the ZFS delete queue (13.7M). I''m guessing that there are a bunch of files that were deleted and put on the delete queue. If all else fails, the delete queue should be processed when the filesystem is remounted, so I''m not sure why that isn''t happening here... I (or a ZPL expert) will have to look into this. There are also a few deferred free blocks, which may be playing into this but probably not to any large extent. Does this machine happen to be on the SWAN? If so, can you send me the machine name and root password so that we can poke around? --matt
Daniel Rock
2006-May-01 17:49 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
Nicolas Williams schrieb:> On Mon, May 01, 2006 at 10:26:33AM +1000, James C. McPherson wrote: >> All seemed well and good until this morning when I did a df: > > Is there a large unlinked file in there, perhaps? > > Aside: What is the best way to find those nowadays? pfiles(1) probably > isn''t it since it doesn''t include a link count in its output (an > RFE?[*]). > > lsof works pretty well for finding processes with open unlinked files, > but probably does not work on Nevada (haven''t tried it).find /proc -type f -links 0 -ls If you just want to search for large unlinked files you can add "-size +xxx" option. Daniel
Matthew Ahrens
2006-May-01 20:45 UTC
[zfs-discuss] where has all my space gone? (with zfs mountroot + b38)
On Mon, May 01, 2006 at 08:08:51AM -0700, Matthew Ahrens wrote:> On Mon, May 01, 2006 at 06:03:29PM +1000, James C. McPherson wrote: > > Blocks LSIZE PSIZE ASIZE avg comp %Total Type > > 4 28.0K 9.00K 20.5K 5.12K 3.11 0.00 L0 deferred free > > 2.05M 190G 178G 178G 86.8K 1.07 99.37 L0 ZFS plain file > > 2.01K 32.2M 13.7M 27.4M 13.6K 2.35 0.01 L0 ZFS delete que > > This tells us that we didn''t waste your space on metadata (since it''s >99% > plain file contents). However, there is a suspiciously large amount of > space used by the ZFS delete queue (13.7M). I''m guessing that there are > a bunch of files that were deleted and put on the delete queue. If all > else fails, the delete queue should be processed when the filesystem is > remounted, so I''m not sure why that isn''t happening here... I (or a ZPL > expert) will have to look into this.It turns out that my suspicion was correct: the delete queue is not being run on the root filesystem. I''ve filed the following bug to track this issue: 6420204 root filesystem''s delete queue is not running --matt
James C. McPherson
2006-May-21 10:14 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
James C. McPherson wrote:> Hi all, > I got a new 300Gb SATA disk last friday for my u20. After figuring > out that the current u20 bios barfs on EFI labels and creating a > whole-disk-sized slice0 I used TimF''s script to implement zfsroot. > All seemed well and good until this morning when I did a df: > pieces: $ df -k / > Filesystem kbytes used avail capacity Mounted on > root_pool/root_filesystem > 286949376 183288496 103652571 64% / > Now try as I might, I''m unable to account for more than about 26Gb > on this disk. > Where has all my space gone?.... Ok, I thought things were going ok until today when I realised that not only has the space in use not gone down, but even after acquiring more disks and moving 22gb of data *off* root_pool, my in use count on root_pool has increased. I''ve bfu''d to 20/May archives and done update_nonON to build 40 as well (though I noticed this yesterday before I updated). Using du -sk on the local directories under /: root_pool/root_filesystem shows (kb) 919 bfu.child 99 bfu.conflicts 796 bfu.parent 0 bin 53330 boot 5 Desktop 445 dev 217 devices 85831 etc 1697108 export 56977 kernel 578545 opt/netbeans 4633 opt/onbld 336935 opt/openoffice.org2.0 1788 opt/OSOLopengrok 4983 opt/PM 6018 opt/schily 7 opt/SUNWits 49 opt/SUNWmlib 441595 opt/SUNWspro 37954 platform 1489 sbin 3149821 usr 3202887 var for a grand total of ~ 9.2gb whereas my new pool, inout, has these filesystems: 18G /scratch 739M /opt/local 945M /opt/csw 2.4G /opt/hometools pieces: # df -k Filesystem kbytes used avail capacity Mounted on root_pool/root_filesystem 286949376 234046924 52865843 82% / /devices 0 0 0 0% /devices ctfs 0 0 0 0% /system/contract proc 0 0 0 0% /proc mnttab 0 0 0 0% /etc/mnttab swap 8097836 768 8097068 1% /etc/svc/volatile objfs 0 0 0 0% /system/object /usr/lib/libc/libc_hwcap2.so.1 286912767 234046924 52865843 82% /lib/libc.so.1 fd 0 0 0 0% /dev/fd swap 8097148 80 8097068 1% /tmp swap 8097104 40 8097064 1% /var/run inout/scratch 132120576 18536683 109397087 15% /scratch /dev/dsk/c1d0s0 8266719 6941900 1242152 85% /ufsroot inout/optlocal 132120576 752264 109397087 1% /opt/local inout/csw 132120576 959117 109397087 1% /opt/csw inout/hometools 132120576 2470511 109397087 3% /opt/hometools inout 132120576 24 109397087 1% /inout root_pool 286949376 24 52865843 1% /root_pool /export/home/jmcp 286912767 234046924 52865843 82% /home/jmcp I''ve had a "zdb -bv root_pool" running for about 30 minutes now.. it just finished and of course told me that everything adds up: Traversing all blocks to verify nothing leaked ... No leaks (block sum matches space maps exactly) bp count: 4575397 bp logical: 384796715008 avg: 84101 bp physical: 238705154048 avg: 52171 compression: 1.61 bp allocated: 239698446336 avg: 52388 compression: 1.61 SPA allocated: 239698446336 used: 80.30% Blocks LSIZE PSIZE ASIZE avg comp %Total Type 12 132K 32.5K 89.0K 7.42K 4.06 0.00 deferred free 1 512 512 1K 1K 1.00 0.00 object directory 3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array 1 16K 1.50K 3.00K 3.00K 10.67 0.00 packed nvlist - - - - - - - packed nvlist size 2 32K 17.0K 35.0K 17.5K 1.88 0.00 bplist - - - - - - - bplist header - - - - - - - SPA space map header 5.78K 24.8M 17.6M 35.6M 6.16K 1.41 0.02 SPA space map 1 16K 16K 16K 16K 1.00 0.00 ZIL intent log 59.5K 952M 237M 477M 8.01K 4.02 0.21 DMU dnode 3 3.00K 1.50K 4.50K 1.50K 2.00 0.00 DMU objset - - - - - - - DSL directory 3 1.50K 1.50K 3.00K 1K 1.00 0.00 DSL directory child map 2 1K 1K 2K 1K 1.00 0.00 DSL dataset snap map 5 64.5K 7.50K 15.0K 3.00K 8.60 0.00 DSL props - - - - - - - DSL dataset - - - - - - - ZFS znode - - - - - - - ZFS ACL 4.18M 357G 222G 223G 53.2K 1.61 99.68 ZFS plain file 110K 110M 61.5M 123M 1.12K 1.78 0.05 ZFS directory 2 1K 1K 2K 1K 1.00 0.00 ZFS master node 8.07K 129M 47.2M 94.9M 11.8K 2.74 0.04 ZFS delete queue - - - - - - - zvol object - - - - - - - zvol prop - - - - - - - other uint8[] - - - - - - - other uint64[] - - - - - - - other ZAP - - - - - - - persistent error log 4.36M 358G 222G 223G 51.2K 1.61 100.00 Total So if >99% of my blocks are allocated to zfs plain files, where oh where is the space going? I dunno that it''s the delete queue not running problem... I''ve whacked the readonly flag on and off lots of times, and kicked off pool scrubs too all to no avail. This is *really* annoying me, and I''m just about at the point of ditching zfs root :( For those of you running zfs root on single-disk pools (eg laptops) do you see this sort of space usage funkiness? thanks in advance, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Mike Gerdts
2006-May-21 11:55 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
On 5/21/06, James C. McPherson <James.McPherson at sun.com> wrote:> Using du -sk on the local directories under /:I assume that you have also looked through /proc/*/fd/* to be sure that there aren''t any big files with a link count of zero (file has been rm''d but process still has it open). The following commands may be helpful: Find the largest files that are open: # du -h /proc/*/fd/* | sort -n or Find all open files that have a link count of zero (have been removed) # ls -l /proc/*/fd/* \ | nawk ''BEGIN { print "Size_MB File" } $2 == 0 && $1 ~ /^-/ { printf "%7d %s\n", $5/1024/1024, $NF }'' Mike -- Mike Gerdts http://mgerdts.blogspot.com/
James C. McPherson
2006-May-21 12:18 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
Mike Gerdts wrote:> On 5/21/06, James C. McPherson <James.McPherson at sun.com> wrote: >> Using du -sk on the local directories under /: > > I assume that you have also looked through /proc/*/fd/* to be sure > that there aren''t any big files with a link count of zero (file has > been rm''d but process still has it open). The following commands may > be helpful: > > Find the largest files that are open: > > # du -h /proc/*/fd/* | sort -n > > or > > Find all open files that have a link count of zero (have been removed) > > # ls -l /proc/*/fd/* \ > | nawk ''BEGIN { print "Size_MB File" } > $2 == 0 && $1 ~ /^-/ { > printf "%7d %s\n", $5/1024/1024, $NF > }''Hi Mike, Thanks for the script, but no, it produced only this: Size_MB File /proc/102286/fd/3: No such file or directory /proc/102286/fd/63: No such file or directory The disk usage has persisted across several reboots. -- James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Casper.Dik at Sun.COM
2006-May-21 12:51 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
>Find the largest files that are open: > ># du -h /proc/*/fd/* | sort -n > >or > >Find all open files that have a link count of zero (have been removed) > ># ls -l /proc/*/fd/* \ > | nawk ''BEGIN { print "Size_MB File" } > $2 == 0 && $1 ~ /^-/ { > printf "%7d %s\n", $5/1024/1024, $NF > }''Nah: find /proc/*/fd -type f -links 0 (Add a -size +1000000c for good measure, to find unlinked files over one SI MB) Casper
Jeff Bonwick
2006-May-21 20:06 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
> I''ve had a "zdb -bv root_pool" running for about 30 minutes now.. it > just finished and of course told me that everything adds up:This is definitely the delete queue problem:> Blocks LSIZE PSIZE ASIZE avg comp %Total Type > 4.18M 357G 222G 223G 53.2K 1.61 99.68 ZFS plain file > 8.07K 129M 47.2M 94.9M 11.8K 2.74 0.04 ZFS delete queueUnder normal circumstances, the delete queue should be empty. Here, the delete queue *itself* is 129M, which means that it''s probably describing many GB of data to be deleted. The problem is described here: 6420204 root filesystem''s delete queue is not running Tabriz, any update on this? Jeff
James C. McPherson
2006-May-21 20:14 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
Hi Jeff, Jeff Bonwick wrote:>> I''ve had a "zdb -bv root_pool" running for about 30 minutes now.. it >> just finished and of course told me that everything adds up: > This is definitely the delete queue problem: >> Blocks LSIZE PSIZE ASIZE avg comp %Total Type >> 4.18M 357G 222G 223G 53.2K 1.61 99.68 ZFS plain file >> 8.07K 129M 47.2M 94.9M 11.8K 2.74 0.04 ZFS delete queue > Under normal circumstances, the delete queue should be empty. > Here, the delete queue *itself* is 129M, which means that it''s > probably describing many GB of data to be deleted.Aha... I''ll go and study the source a bit more carefully. I know that zdb is private and totally unstable, but could we get the manpage for it to at least say what the LSIZE, PSIZE and ASIZE columns mean please?> The problem is described here: > 6420204 root filesystem''s delete queue is not running > Tabriz, any update on this?I guess I discounted this bug (of course I knew of it) because if you look at the %Total column it only shows up as 0.04%. Time for me to utsl. Thanks heaps, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Darren J Moffat
2006-May-22 09:25 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
James C. McPherson wrote:> Hi Jeff, > > Jeff Bonwick wrote: >>> I''ve had a "zdb -bv root_pool" running for about 30 minutes now.. it >>> just finished and of course told me that everything adds up: >> This is definitely the delete queue problem: >>> Blocks LSIZE PSIZE ASIZE avg comp %Total Type >>> 4.18M 357G 222G 223G 53.2K 1.61 99.68 ZFS plain file >>> 8.07K 129M 47.2M 94.9M 11.8K 2.74 0.04 ZFS delete >>> queue >> Under normal circumstances, the delete queue should be empty. >> Here, the delete queue *itself* is 129M, which means that it''s >> probably describing many GB of data to be deleted. > > Aha... I''ll go and study the source a bit more carefully. > > I know that zdb is private and totally unstable, but could we get > the manpage for it to at least say what the LSIZE, PSIZE and ASIZE > columns mean please?See Section 2.6 of the ZFS On-Disk Specification: -- BEGIN QUOTE -- lsize: Logical size. The size of the data without compression, raidz or gang overheard. psize: Physical size of the blok on disk after compression. asize: Allocated size, totall size of all blocks allocated to hold this data including any gang headers or raidz parity information. If compression is turned off and ZFS is not on raidz storage, lsize, asize and psize will all be equal. All sizes are stored as the number of 512 byte sectors (minus one) needed to represent the size of this block. -- END QUOTE -- -- Darren J Moffat
James C. McPherson
2006-May-22 10:53 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
Hi Darren Darren J Moffat wrote:> James C. McPherson wrote:...>> I know that zdb is private and totally unstable, but could we get >> the manpage for it to at least say what the LSIZE, PSIZE and ASIZE >> columns mean please? > See Section 2.6 of the ZFS On-Disk Specification: > -- BEGIN QUOTE -- > lsize: Logical size. The size of the data without compression, raidz or > gang overheard. > > psize: Physical size of the blok on disk after compression. > > asize: Allocated size, total size of all blocks allocated to hold this > data including any gang headers or raidz parity information. > > If compression is turned off and ZFS is not on raidz storage, lsize, > asize and psize will all be equal. > > All sizes are stored as the number of 512 byte sectors (minus one) > needed to represent the size of this block. > -- END QUOTE --Thankyou for the pointer. /me slaps self on side of head for research failure I''ll go and do some more reading :) cheers, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Tabriz Leman
2006-May-22 17:24 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
Jeff Bonwick wrote:>> I''ve had a "zdb -bv root_pool" running for about 30 minutes now.. it >> just finished and of course told me that everything adds up: >> > > This is definitely the delete queue problem: > > >> Blocks LSIZE PSIZE ASIZE avg comp %Total Type >> 4.18M 357G 222G 223G 53.2K 1.61 99.68 ZFS plain file >> 8.07K 129M 47.2M 94.9M 11.8K 2.74 0.04 ZFS delete queue >> > > Under normal circumstances, the delete queue should be empty. > Here, the delete queue *itself* is 129M, which means that it''s > probably describing many GB of data to be deleted. > > The problem is described here: > > 6420204 root filesystem''s delete queue is not running > > Tabriz, any update on this? > >Well, unfortunately, the update is no update. I have not had a chance to fix this yet. I will try to get to it this week or early next week. The workaround for this bug is to issue to following command... # zfs set readonly=off <pool>/<fs_name> This will cause the delete queue to start up and should flush your queue. I apologize for the inconvenience. Tabriz> Jeff > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Jeff Bonwick
2006-May-22 19:30 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
> > 6420204 root filesystem''s delete queue is not running> The workaround for this bug is to issue to following command... > > # zfs set readonly=off <pool>/<fs_name> > > This will cause the delete queue to start up and should flush your queue.Tabriz, Thanks for the update. James, please let us know if this solves your problem. Jeff
James C. McPherson
2006-May-22 21:13 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
Jeff Bonwick wrote:>>> 6420204 root filesystem''s delete queue is not running > >> The workaround for this bug is to issue to following command... >> >> # zfs set readonly=off <pool>/<fs_name> >> >> This will cause the delete queue to start up and should flush your queue. > > Tabriz, > > Thanks for the update. James, please let us know if this solves your problem.Hi all, yes, I''ve tried that several times and it didn''t work for me at all. One thing that worked a *little* bit was to set readonly=on, then go in with mdb -kw and set the drained flag on root_pool to 0 and then re-set readonly=off. But that only freed up about 2Gb. thanks again, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
James C. McPherson
2006-Jun-22 23:05 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
James C. McPherson wrote:> Jeff Bonwick wrote: >>>> 6420204 root filesystem''s delete queue is not running >>> The workaround for this bug is to issue to following command... >>> # zfs set readonly=off <pool>/<fs_name> >>> This will cause the delete queue to start up and should flush your >>> queue. >> Thanks for the update. James, please let us know if this solves your >> problem. > yes, I''ve tried that several times and it didn''t work for me at all. > One thing that worked a *little* bit was to set readonly=on, then > go in with mdb -kw and set the drained flag on root_pool to 0 and > then re-set readonly=off. But that only freed up about 2Gb.Here''s the next installment in the saga. I bfu''d to include Mark''s recent putback, rebooted, re-ran the "set readonly=off" op on the root pool and root filesystem, and waited. Nothing. Nada. Not a sausage. Here''s my root filesystem delete head: > ::fsinfo ! head -2 VFSP FS MOUNT fffffffffbcaa4e0 zfs / > fffffffffbcaa4e0::print struct vfs vfs_data |::print struct zfsvfs z_delete_head { z_delete_head.z_mutex = { _opaque = [ 0 ] } z_delete_head.z_cv = { _opaque = 0 } z_delete_head.z_quiesce_cv = { _opaque = 0 } z_delete_head.z_drained = 0x1 z_delete_head.z_draining = 0 z_delete_head.z_thread_target = 0 z_delete_head.z_thread_count = 0 z_delete_head.z_znode_count = 0x5ce4 z_delete_head.z_znodes = { list_size = 0xc0 list_offset = 0x10 list_head = { list_next = 0xffffffff9232ded0 list_prev = 0xfffffe820d2c16b0 } } } I also went in with mdb -kw and set z_drained to 0, then re-set the readonly flag... still nothing. Pool usage is now up to ~93%, and a zdb run shows lots of leaked space too: ....[snip bazillions of entries re leakage].... block traversal size 273838116352 != alloc 274123164672 (leaked 285048320) bp count: 5392224 bp logical: 454964635136 avg: 84374 bp physical: 272756334592 avg: 50583 compression: 1.67 bp allocated: 273838116352 avg: 50783 compression: 1.66 SPA allocated: 274123164672 used: 91.83% Blocks LSIZE PSIZE ASIZE avg comp %Total Type 3 48.0K 8K 24.0K 8K 6.00 0.00 L1 deferred free 5 44.0K 14.5K 37.0K 7.40K 3.03 0.00 L0 deferred free 8 92.0K 22.5K 61.0K 7.62K 4.09 0.00 deferred free 1 512 512 1K 1K 1.00 0.00 object directory 3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array 1 16K 1.50K 3.00K 3.00K 10.67 0.00 packed nvlist - - - - - - - packed nvlist size 1 16K 1K 3.00K 3.00K 16.00 0.00 L1 bplist 1 16K 16K 32K 32K 1.00 0.00 L0 bplist 2 32K 17.0K 35.0K 17.5K 1.88 0.00 bplist - - - - - - - bplist header - - - - - - - SPA space map header 140 2.19M 364K 1.06M 7.79K 6.16 0.00 L1 SPA space map 5.01K 20.1M 15.4M 30.7M 6.13K 1.31 0.01 L0 SPA space map 5.15K 22.2M 15.7M 31.8M 6.17K 1.42 0.01 SPA space map 1 28.0K 28.0K 28.0K 28.0K 1.00 0.00 ZIL intent log 2 32K 2K 6.00K 3.00K 16.00 0.00 L6 DMU dnode 2 32K 2K 6.00K 3.00K 16.00 0.00 L5 DMU dnode 2 32K 2K 6.00K 3.00K 16.00 0.00 L4 DMU dnode 2 32K 2.50K 7.50K 3.75K 12.80 0.00 L3 DMU dnode 15 240K 50.5K 152K 10.1K 4.75 0.00 L2 DMU dnode 594 9.28M 3.88M 11.6M 20.1K 2.39 0.00 L1 DMU dnode 68.7K 1.07G 274M 549M 7.99K 4.00 0.21 L0 DMU dnode 69.3K 1.08G 278M 561M 8.09K 3.98 0.21 DMU dnode 3 3.00K 1.50K 4.50K 1.50K 2.00 0.00 DMU objset - - - - - - - DSL directory 3 1.50K 1.50K 3.00K 1K 1.00 0.00 DSL directory child map 2 1K 1K 2K 1K 1.00 0.00 DSL dataset snap map 5 64.5K 7.50K 15.0K 3.00K 8.60 0.00 DSL props - - - - - - - DSL dataset - - - - - - - ZFS znode - - - - - - - ZFS ACL 2.82K 45.1M 2.93M 5.85M 2.08K 15.41 0.00 L2 ZFS plain file 564K 8.81G 612M 1.19G 2.17K 14.76 0.47 L1 ZFS plain file 4.40M 414G 253G 253G 57.5K 1.63 99.21 L0 ZFS plain file 4.95M 422G 254G 254G 51.4K 1.67 99.68 ZFS plain file 1 16K 1K 3.00K 3.00K 16.00 0.00 L2 ZFS directory 261 4.08M 280K 839K 3.21K 14.94 0.00 L1 ZFS directory 113K 108M 63.2M 126M 1.11K 1.72 0.05 L0 ZFS directory 114K 113M 63.5M 127M 1.12K 1.77 0.05 ZFS directory 2 1K 1K 2K 1K 1.00 0.00 ZFS master node 1 16K 4.50K 13.5K 13.5K 3.56 0.00 L2 ZFS delete queue 66 1.03M 524K 1.54M 23.8K 2.02 0.00 L1 ZFS delete queue 8.21K 131M 53.9M 108M 13.1K 2.44 0.04 L0 ZFS delete queue 8.28K 132M 54.4M 109M 13.2K 2.43 0.04 ZFS delete queue - - - - - - - zvol object - - - - - - - zvol prop - - - - - - - other uint8[] - - - - - - - other uint64[] - - - - - - - other ZAP - - - - - - - persistent error log 2 32K 2K 6.00K 3.00K 16.00 0.00 L6 Total 2 32K 2K 6.00K 3.00K 16.00 0.00 L5 Total 2 32K 2K 6.00K 3.00K 16.00 0.00 L4 Total 2 32K 2.50K 7.50K 3.75K 12.80 0.00 L3 Total 2.83K 45.4M 2.98M 6.02M 2.12K 15.22 0.00 L2 Total 565K 8.83G 617M 1.21G 2.19K 14.66 0.47 L1 Total 4.59M 415G 253G 254G 55.3K 1.64 99.52 L0 Total 5.14M 424G 254G 255G 49.6K 1.67 100.00 Total Tabriz, how''s your fix coming along? James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Mark Shellenbaum
2006-Jun-22 23:46 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
James C. McPherson wrote:> James C. McPherson wrote: >> Jeff Bonwick wrote: >>>>> 6420204 root filesystem''s delete queue is not running >>>> The workaround for this bug is to issue to following command... >>>> # zfs set readonly=off <pool>/<fs_name> >>>> This will cause the delete queue to start up and should flush your >>>> queue. >>> Thanks for the update. James, please let us know if this solves your >>> problem. >> yes, I''ve tried that several times and it didn''t work for me at all. >> One thing that worked a *little* bit was to set readonly=on, then >> go in with mdb -kw and set the drained flag on root_pool to 0 and >> then re-set readonly=off. But that only freed up about 2Gb. > > Here''s the next installment in the saga. I bfu''d to include Mark''s > recent putback, rebooted, re-ran the "set readonly=off" op on the > root pool and root filesystem, and waited. Nothing. Nada. Not a > sausage. > > Here''s my root filesystem delete head: > > > > ::fsinfo ! head -2 > VFSP FS MOUNT > fffffffffbcaa4e0 zfs / > > fffffffffbcaa4e0::print struct vfs vfs_data |::print struct zfsvfs > z_delete_head > { > z_delete_head.z_mutex = { > _opaque = [ 0 ] > } > z_delete_head.z_cv = { > _opaque = 0 > } > z_delete_head.z_quiesce_cv = { > _opaque = 0 > } > z_delete_head.z_drained = 0x1 > z_delete_head.z_draining = 0 > z_delete_head.z_thread_target = 0 > z_delete_head.z_thread_count = 0 > z_delete_head.z_znode_count = 0x5ce4So we have a bunch of stuff in the in-core delete queue, but no threads to process them. The fact that we don''t have the threads is related to the bug that Tabriz is working on. -Mark
James C. McPherson
2006-Jun-23 13:42 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
Mark Shellenbaum wrote: ...> So we have a bunch of stuff in the in-core delete queue, but no threads > to process them. The fact that we don''t have the threads is related to > the bug that Tabriz is working on.Hi Mark, after installing your fixes from three days ago and (cough!) ensuring that my boot archive contained them, I then spent the next 7 or so hours waiting for the delete queue to be flushed. In that time my root disk (a Maxtor) decided it didn''t like me much (I was asking it to do too much io) so zfs paniced... then a few single- user boots later (where each time the boot process was stuck in the fs-usr service, flushing the queue) and I''m finally back to having the disk space that I think I should have. My one remaining concern is that I''m not sure that I''ve got all my zfs bits totally sync''d with my kernel so I''ll be bfuing again tomorrow just to make sure. Thanks for your help with this, I reallyreally appreciate it. best regards, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Tabriz
2006-Jun-26 16:29 UTC
[zfs-discuss] Re: where has all my space gone? (with zfs mountroot + b38)
James C. McPherson wrote:> James C. McPherson wrote: >> Jeff Bonwick wrote: >>>>> 6420204 root filesystem''s delete queue is not running >>>> The workaround for this bug is to issue to following command... >>>> # zfs set readonly=off <pool>/<fs_name> >>>> This will cause the delete queue to start up and should flush your >>>> queue. >>> Thanks for the update. James, please let us know if this solves >>> your problem. >> yes, I''ve tried that several times and it didn''t work for me at all. >> One thing that worked a *little* bit was to set readonly=on, then >> go in with mdb -kw and set the drained flag on root_pool to 0 and >> then re-set readonly=off. But that only freed up about 2Gb. > > Here''s the next installment in the saga. I bfu''d to include Mark''s > recent putback, rebooted, re-ran the "set readonly=off" op on the > root pool and root filesystem, and waited. Nothing. Nada. Not a > sausage. > > Here''s my root filesystem delete head: > > > > ::fsinfo ! head -2 > VFSP FS MOUNT > fffffffffbcaa4e0 zfs / > > fffffffffbcaa4e0::print struct vfs vfs_data |::print struct zfsvfs > z_delete_head > { > z_delete_head.z_mutex = { > _opaque = [ 0 ] > } > z_delete_head.z_cv = { > _opaque = 0 > } > z_delete_head.z_quiesce_cv = { > _opaque = 0 > } > z_delete_head.z_drained = 0x1 > z_delete_head.z_draining = 0 > z_delete_head.z_thread_target = 0 > z_delete_head.z_thread_count = 0 > z_delete_head.z_znode_count = 0x5ce4 > z_delete_head.z_znodes = { > list_size = 0xc0 > list_offset = 0x10 > list_head = { > list_next = 0xffffffff9232ded0 > list_prev = 0xfffffe820d2c16b0 > } > } > } > > > > > I also went in with mdb -kw and set z_drained to 0, then re-set the > readonly flag... still nothing. Pool usage is now up to ~93%, and a > zdb run shows lots of leaked space too: > > ....[snip bazillions of entries re leakage].... > > block traversal size 273838116352 != alloc 274123164672 (leaked > 285048320) > > bp count: 5392224 > bp logical: 454964635136 avg: 84374 > bp physical: 272756334592 avg: 50583 compression: > 1.67 > bp allocated: 273838116352 avg: 50783 compression: > 1.66 > SPA allocated: 274123164672 used: 91.83% > > Blocks LSIZE PSIZE ASIZE avg comp %Total Type > 3 48.0K 8K 24.0K 8K 6.00 0.00 L1 > deferred free > 5 44.0K 14.5K 37.0K 7.40K 3.03 0.00 L0 > deferred free > 8 92.0K 22.5K 61.0K 7.62K 4.09 0.00 deferred free > 1 512 512 1K 1K 1.00 0.00 object directory > 3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array > 1 16K 1.50K 3.00K 3.00K 10.67 0.00 packed nvlist > - - - - - - - packed nvlist > size > 1 16K 1K 3.00K 3.00K 16.00 0.00 L1 bplist > 1 16K 16K 32K 32K 1.00 0.00 L0 bplist > 2 32K 17.0K 35.0K 17.5K 1.88 0.00 bplist > - - - - - - - bplist header > - - - - - - - SPA space map > header > 140 2.19M 364K 1.06M 7.79K 6.16 0.00 L1 SPA > space map > 5.01K 20.1M 15.4M 30.7M 6.13K 1.31 0.01 L0 SPA > space map > 5.15K 22.2M 15.7M 31.8M 6.17K 1.42 0.01 SPA space map > 1 28.0K 28.0K 28.0K 28.0K 1.00 0.00 ZIL intent log > 2 32K 2K 6.00K 3.00K 16.00 0.00 L6 DMU dnode > 2 32K 2K 6.00K 3.00K 16.00 0.00 L5 DMU dnode > 2 32K 2K 6.00K 3.00K 16.00 0.00 L4 DMU dnode > 2 32K 2.50K 7.50K 3.75K 12.80 0.00 L3 DMU dnode > 15 240K 50.5K 152K 10.1K 4.75 0.00 L2 DMU dnode > 594 9.28M 3.88M 11.6M 20.1K 2.39 0.00 L1 DMU dnode > 68.7K 1.07G 274M 549M 7.99K 4.00 0.21 L0 DMU dnode > 69.3K 1.08G 278M 561M 8.09K 3.98 0.21 DMU dnode > 3 3.00K 1.50K 4.50K 1.50K 2.00 0.00 DMU objset > - - - - - - - DSL directory > 3 1.50K 1.50K 3.00K 1K 1.00 0.00 DSL directory > child map > 2 1K 1K 2K 1K 1.00 0.00 DSL dataset > snap map > 5 64.5K 7.50K 15.0K 3.00K 8.60 0.00 DSL props > - - - - - - - DSL dataset > - - - - - - - ZFS znode > - - - - - - - ZFS ACL > 2.82K 45.1M 2.93M 5.85M 2.08K 15.41 0.00 L2 ZFS > plain file > 564K 8.81G 612M 1.19G 2.17K 14.76 0.47 L1 ZFS > plain file > 4.40M 414G 253G 253G 57.5K 1.63 99.21 L0 ZFS > plain file > 4.95M 422G 254G 254G 51.4K 1.67 99.68 ZFS plain file > 1 16K 1K 3.00K 3.00K 16.00 0.00 L2 ZFS > directory > 261 4.08M 280K 839K 3.21K 14.94 0.00 L1 ZFS > directory > 113K 108M 63.2M 126M 1.11K 1.72 0.05 L0 ZFS > directory > 114K 113M 63.5M 127M 1.12K 1.77 0.05 ZFS directory > 2 1K 1K 2K 1K 1.00 0.00 ZFS master node > 1 16K 4.50K 13.5K 13.5K 3.56 0.00 L2 ZFS > delete queue > 66 1.03M 524K 1.54M 23.8K 2.02 0.00 L1 ZFS > delete queue > 8.21K 131M 53.9M 108M 13.1K 2.44 0.04 L0 ZFS > delete queue > 8.28K 132M 54.4M 109M 13.2K 2.43 0.04 ZFS delete queue > - - - - - - - zvol object > - - - - - - - zvol prop > - - - - - - - other uint8[] > - - - - - - - other uint64[] > - - - - - - - other ZAP > - - - - - - - persistent > error log > 2 32K 2K 6.00K 3.00K 16.00 0.00 L6 Total > 2 32K 2K 6.00K 3.00K 16.00 0.00 L5 Total > 2 32K 2K 6.00K 3.00K 16.00 0.00 L4 Total > 2 32K 2.50K 7.50K 3.75K 12.80 0.00 L3 Total > 2.83K 45.4M 2.98M 6.02M 2.12K 15.22 0.00 L2 Total > 565K 8.83G 617M 1.21G 2.19K 14.66 0.47 L1 Total > 4.59M 415G 253G 254G 55.3K 1.64 99.52 L0 Total > 5.14M 424G 254G 255G 49.6K 1.67 100.00 Total > > > > > Tabriz, how''s your fix coming along? > >I am sorry that my response if so delayed, I took Friday off. Anyhow, let me work on this today. I have talked with the ZFS team and we have come to the consensus that if the readonly property is not explicitly given on a remount operation (ie mount -F zfs -o remount,ro pool/dataset, or zfs mount -o remount,ro pool/dataset) that we will remount rw by default. This is more or less what UFS does, except that it does NOT allow explicit readonly remounts (a UFS remount operation will always cause the filesystem to be remounted rw). ZFS is capable of handling readonly remount, so we will continue to allow them. Will work on this today and let you know as more info becomes available. Tabriz> > > James C. McPherson > -- > Solaris Datapath Engineering > Data Management Group > Sun Microsystems