Wow -- I had seen Joe Little''s blog about i-ram and slog and was using it that way, and mine just failed also. I''m not at the point he is with data loss, but I''ve booted the OS 2008.05 livecd and I can see all the disks (minus the log) in an zpool import. Please, Please, Please, tell me there is a way to recover from this.... Jeb This message posted from opensolaris.org
Meant to add that zpool import -f pool doesn''t work b/c of the missing log vdev. All the other disks are there and show up with "zpool import", but it won''t import. Is there anyway a util could clear the log device vdev from the remaining raidz2 devices? Then I could import just a standard raidz2 pool. I really love zfs (and had recently upgraded to 6 disks in raidz2), but this is *really* gonna hurt to lose all this stuff (yeah, the work stuff is backed up, but I have/had tons of personal stuff on there). I definitely would prefer to just sit tight, and see if there is any way to get this going (read only would be fine). Jeb This message posted from opensolaris.org
On Thu, May 29, 2008 at 7:25 PM, Jeb Campbell <jebc at c4solutions.net> wrote:> Meant to add that zpool import -f pool doesn''t work b/c of the missing log vdev. > > All the other disks are there and show up with "zpool import", but it won''t import. > > Is there anyway a util could clear the log device vdev from the remaining raidz2 devices? > > Then I could import just a standard raidz2 pool. > > I really love zfs (and had recently upgraded to 6 disks in raidz2), but this is *really* gonna hurt to lose all this stuff (yeah, the work stuff is backed up, but I have/had tons of personal stuff on there). > > I definitely would prefer to just sit tight, and see if there is any way to get this going (read only would be fine). >You can mount all those filesystems, and then zfs send/recv them off to another box. Its sucks, but as of now, there is no re-importing of the pool UNTIL the log can be removed. Sadly, I think that log removal will at least require importation of the pool in question first. For some reason you already can''t import your pool. In my case, I was running B70 and could import the pool still, but just degraded. I think that once you are at a higher rev (which I do not know, but inclusive of B82 and B85), you won''t be able to import it anymore when it fails.> Jeb > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On Thu, May 29, 2008 at 8:59 PM, Joe Little <jmlittle at gmail.com> wrote:> On Thu, May 29, 2008 at 7:25 PM, Jeb Campbell <jebc at c4solutions.net> wrote: >> Meant to add that zpool import -f pool doesn''t work b/c of the missing log vdev. >> >> All the other disks are there and show up with "zpool import", but it won''t import. >> >> Is there anyway a util could clear the log device vdev from the remaining raidz2 devices? >> >> Then I could import just a standard raidz2 pool. >> >> I really love zfs (and had recently upgraded to 6 disks in raidz2), but this is *really* gonna hurt to lose all this stuff (yeah, the work stuff is backed up, but I have/had tons of personal stuff on there). >> >> I definitely would prefer to just sit tight, and see if there is any way to get this going (read only would be fine). >>More to the point, does it say there are any permanent errors that you find? Again, I was able to import it after reassigning the log device so it thinks its there. I got to this point: root at rome:~# zpool status -v pool: data state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 24 raidz1 ONLINE 0 0 24 c2t0d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 logs ONLINE 0 0 24 c3t1d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: data/home:<0x0> Yes, because of the "error" I can no longer have any mounts created at import, but the "zfs mount data/proj" or other filesystem, but not data/home, is still possible. Again, I think that you will want to use "-o ro" as an option to that mount command to not have the system go bonkers. Check my blog for more info on reseting the log device for a "zfs replace" action -- which itself puts you into more troubling position of possibly having corruptions from the resilver, but at least for me allowed me to mount the pool for read-only mounts of the remaining filesystems.> > You can mount all those filesystems, and then zfs send/recv them off > to another box. Its sucks, but as of now, there is no re-importing of > the pool UNTIL the log can be removed. Sadly, I think that log removal > will at least require importation of the pool in question first. For > some reason you already can''t import your pool. > > In my case, I was running B70 and could import the pool still, but > just degraded. I think that once you are at a higher rev (which I do > not know, but inclusive of B82 and B85), you won''t be able to import > it anymore when it fails. > > >> Jeb >> >> >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >
Ok, here is where I''m at: My install of OS 2008.05 (snv_86?) will not even come up in single user. The OS 2008.05 live cd comes up fine, but I can''t import my old pool b/c of the missing log (and I have to import to fix the log ...). So I think I''ll boot from the live cd, import my rootpool, mount it, and copy /rootpool/etc/zfs.cache to zfs.cache.save. Then I''ll stick the zfs.cache from the live cd onto the rootpool, update boot bits, and cross my fingers. The goal of this is to get my installed OS to finish a boot, then I can try using the saved zfs.cache to load the degraded pool w/o import. As long as I can get to it read-only, I''ll copy what I can off. Any tips, comments, or suggestions would be welcome, Jeb This message posted from opensolaris.org
On Fri, May 30, 2008 at 6:30 AM, Jeb Campbell <jebc at c4solutions.net> wrote:> Ok, here is where I''m at: > > My install of OS 2008.05 (snv_86?) will not even come up in single user. > > The OS 2008.05 live cd comes up fine, but I can''t import my old pool b/c of the missing log (and I have to import to fix the log ...). > > So I think I''ll boot from the live cd, import my rootpool, mount it, and copy /rootpool/etc/zfs.cache to zfs.cache.save. Then I''ll stick the zfs.cache from the live cd onto the rootpool, update boot bits, and cross my fingers. > > The goal of this is to get my installed OS to finish a boot, then I can try using the saved zfs.cache to load the degraded pool w/o import. As long as I can get to it read-only, I''ll copy what I can off. > > Any tips, comments, or suggestions would be welcome, >It seems we are in our own little echo chamber here. Well, these are the bugs/resolutions that need to be addressed: 1) and l2arc or log device needs to evacuation-possible 2) any failure of a l2arc or log device should never prevent importation of a pool. It is an additional device for cache/log purposes, and failures of these devices should be correctly handled, but not at the scope of failing the volume/losing already stored data. Yes, this means that data in the intent log may be lost, but I''d rather lose that 0.01% vs the whole volume 3) The failure to iterate filesystems bug is also quite annoying. If any sub-FS in a zfs pool is not iteratable (data/home in my example), the importation of the pool should note the error but proceed. Mounts to data/proj and data/* except for data/home should continue. Again, always attempt to do as much as possible to provide access to the data that''s still available. Giving up on all mounts because of a fault on one is not reasonable behavior There are more things here, and perhaps one can argue with the above. However, my stance is being overly conservative to _DATA ACCESS_ -- faulting a pool until you can contact support and get a hack to recover otherwise available data is just not good form :) You''ll lose customers left and right because of the cost of the "downtime"> Jeb > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
> 1) and l2arc or log device needs to evacuation-possiblehow about evacuation of any vdev? (pool shrink!) > 2) any failure of a l2arc or log device should never prevent > importation of a pool. how about import or creation of any kinda degraded pool? Rob