Daniel Smedegaard Buus
2010-Aug-13 06:30 UTC
[zfs-code] Reboot, pool ok, zfs filesystem disappeared, its used space still "in use"
Hello all! Disclaimer: I''m using zfs-fuse (0.6.0, equals r16 AFAICT) on Linux Mint 9, but I''m hoping I''m not too far off asking for help in the "real" zfs forum? Hope not :) Okay, so I have this massive RAIDZ2 pool, ''titanic'', which until recently just had one large fs called ''titanic/archive'', mounted in /home/daniel/Archive. The day before yesterday, I added another fs, ''titanic/backups'', this time mounted in /titanic/backups, and a new striped (non-parity) pool on two new disks, ''home'' mounted in ''/home''. Yesterday, before going to bed, I did some organizing of files on the archive fs, then shut down the machine. This morning, I turn on the computer to listen to some radio, and Gnome complains about not being able to restore two Nautilus sessions on /home/daniel/Archive, as "the location does not exist". Slight panic. Okay, so a df -h shows, lnxsrv daniel # df -h Filesystem Size Used Avail Use% Mounted on /dev/md21p1 12G 4,2G 7,1G 38% / none 2,0G 592K 2,0G 1% /dev none 2,0G 1,4M 2,0G 1% /dev/shm none 2,0G 220K 2,0G 1% /var/run none 2,0G 0 2,0G 0% /var/lock none 2,0G 0 2,0G 0% /lib/init/rw none 12G 4,2G 7,1G 38% /var/lib/ureadahead/debugfs home 524G 34G 491G 7% /home titanic 500G 45K 500G 1% /titanic titanic/backups 500G 45K 500G 1% /titanic/backups And zpool status says, lnxsrv daniel # zpool status pool: home state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM home ONLINE 0 0 0 sda5 ONLINE 0 0 0 sdb5 ONLINE 0 0 0 errors: No known data errors pool: titanic state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM titanic ONLINE 0 0 0 raidz2 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 md0p1 ONLINE 0 0 0 md1p1 ONLINE 0 0 0 md2p1 ONLINE 0 0 0 md3p1 ONLINE 0 0 0 md4p1 ONLINE 0 0 0 errors: No known data errors But then a zfs list -r: lnxsrv daniel # zfs list -r NAME USED AVAIL REFER MOUNTPOINT home 33,1G 491G 33,1G /home titanic 8,35T 499G 44,8K /titanic titanic/backups 44,8K 499G 44,8K /titanic/backups So, I''m wondering where did titanic/archive go? I read 499 gigs out of 3.35 teras used, which off the top of my head matches okay with the space used by the archive fs, so the space seems to still be allocated. So in my head I''m hearing zfs go, "Got a problem, Skip, waiting for you to resolve it. Till then, I''m just gonna cup my hands over this part of the pool." I hope so, anyway ;) I thought of the documentation stating that by default mounts fail if the mount point is non-empty, so I checked /home/daniel/Archive, and it''s fully empty, and also zsf mount -a does nothing new (well, I didn''t expect that either, as the fs list doesn''t mention the archive fs). I thought that maybe the fact that I''m mounting the archive fs in the scope of another pool''s mounted fs might cause some sort of conflict, or the mount failed because at boot time it might be ordered unfavorable (trying to mount /home/daniel/Archive from the titanic pool before trying to start the second "home" pool), but still I think I should be able to see the fs in the list afterwards? I really hope someone has a tip of where to go from here. I''m afraid to do anything yet, so I haven''t tried exporting/importing, rebooting, or anything. I did a grep on my logs, and I''ve copy/pasted relevant parts below. I forget which logs contain zfs messages, so I did a cat /var/log/* | grep -i zfs. Anyway, here goes (not sure if I should panic about the "Failed to flush write cache" bits): Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md0p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md1p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md2p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md3p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md4p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:28 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 13 07:01:28 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 13 07:01:28 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 13 07:01:28 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 13 07:01:28 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 13 07:01:34 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 13 07:01:35 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 13 07:01:35 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:01:38 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:38 lnxsrv zfs-fuse: mount options: fsname=home,allow_other,suid,dev,big_writes Aug 13 07:01:40 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:40 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 13 07:01:41 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:41 lnxsrv zfs-fuse: mount options: fsname=titanic/backups,allow_other,suid,dev,big_writes Aug 13 07:47:23 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:48:16 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:55:50 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:47:23 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:48:16 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:55:50 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:01:28 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 13 07:01:28 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 13 07:01:28 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 13 07:01:28 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 13 07:01:28 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md0p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md1p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md2p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md3p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md4p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 13 07:01:34 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 13 07:01:35 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 13 07:01:35 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:01:38 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:38 lnxsrv zfs-fuse: mount options: fsname=home,allow_other,suid,dev,big_writes Aug 13 07:01:40 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:40 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 13 07:01:41 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:41 lnxsrv zfs-fuse: mount options: fsname=titanic/backups,allow_other,suid,dev,big_writes Aug 13 07:01:28 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 13 07:01:28 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 13 07:01:28 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 13 07:01:28 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 13 07:01:28 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 13 07:01:34 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 13 07:01:35 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 13 07:01:35 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:01:38 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:38 lnxsrv zfs-fuse: mount options: fsname=home,allow_other,suid,dev,big_writes Aug 13 07:01:40 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:40 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 13 07:01:41 lnxsrv zfs-fuse: enabling fuse big_writes Aug 13 07:01:41 lnxsrv zfs-fuse: mount options: fsname=titanic/backups,allow_other,suid,dev,big_writes Aug 13 07:47:23 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:48:16 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 13 07:55:50 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 And two days ago, working, with titanic/archive being mounted, too: Aug 11 13:13:47 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md0p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:13:47 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md1p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:13:47 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md2p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:13:47 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md3p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:13:47 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md4p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:33:27 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md0p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:33:27 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md1p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:33:27 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md2p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:33:27 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md3p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 13:33:27 lnxsrv zfs-fuse: WARNING: Failed to flush write cache on device ''/dev/md4p1''. Data on pool ''titanic'' may be lost if power fails. No further warnings will be given. Aug 11 07:05:12 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 11 07:05:12 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 11 07:05:12 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 11 07:05:13 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 11 07:05:13 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 11 07:05:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 07:05:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 07:05:21 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 07:05:21 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 07:05:21 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 07:05:21 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 12:49:38 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 11 12:49:38 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 11 12:49:38 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 11 12:49:38 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 11 12:49:38 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 11 12:49:42 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 12:49:42 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 12:49:45 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 12:49:45 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 12:49:45 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 12:49:45 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 13:00:12 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 11 13:00:12 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 11 13:00:12 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 11 13:00:12 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 11 13:00:12 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 11 13:00:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 13:00:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 13:00:18 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:00:18 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 13:00:19 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:00:19 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 13:18:06 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 13:18:06 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 13:28:14 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:28:14 lnxsrv zfs-fuse: mount options: fsname=home,allow_other,suid,dev,big_writes Aug 11 13:33:26 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 13:33:26 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 13:33:29 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:33:29 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 13:33:30 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:33:30 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 13:39:21 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:39:21 lnxsrv zfs-fuse: mount options: fsname=titanic/backups,allow_other,suid,dev,big_writes Aug 11 13:40:51 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 Aug 11 07:05:12 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 11 07:05:12 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 11 07:05:12 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 11 07:05:13 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 11 07:05:13 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 11 07:05:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 07:05:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 07:05:21 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 07:05:21 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 07:05:21 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 07:05:21 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 12:49:38 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 11 12:49:38 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 11 12:49:38 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 11 12:49:38 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 11 12:49:38 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 11 12:49:42 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 12:49:42 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 12:49:45 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 12:49:45 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 12:49:45 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 12:49:45 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 13:00:12 lnxsrv zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1 Aug 11 13:00:12 lnxsrv zfs-fuse: ARC caching: maximum ARC size: compiled-in default Aug 11 13:00:12 lnxsrv zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000 Aug 11 13:00:12 lnxsrv zfs-fuse: ARC setup: min ARC size set to 16777216 bytes Aug 11 13:00:12 lnxsrv zfs-fuse: ARC setup: max ARC size set to 134217728 bytes Aug 11 13:00:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 13:00:17 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 13:00:18 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:00:18 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 13:00:19 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:00:19 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 13:18:06 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 13:18:06 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 13:28:14 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:28:14 lnxsrv zfs-fuse: mount options: fsname=home,allow_other,suid,dev,big_writes Aug 11 13:33:26 lnxsrv zfs-fuse: put_nvlist: out of memory 4260 > 2920 Aug 11 13:33:26 lnxsrv zfs-fuse: put_nvlist: out of memory 4240 > 4112 Aug 11 13:33:29 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:33:29 lnxsrv zfs-fuse: mount options: fsname=titanic/archive,allow_other,suid,dev,big_writes Aug 11 13:33:30 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:33:30 lnxsrv zfs-fuse: mount options: fsname=titanic,allow_other,suid,dev,big_writes Aug 11 13:39:21 lnxsrv zfs-fuse: enabling fuse big_writes Aug 11 13:39:21 lnxsrv zfs-fuse: mount options: fsname=titanic/backups,allow_other,suid,dev,big_writes Aug 11 13:40:51 lnxsrv zfs-fuse: put_nvlist: out of memory 1828 > 1688 I don''t like the cache flush WARNINGs, and I [b]really[/b] don''t like the "out of memory" notices... -- This message posted from opensolaris.org