zfs discuss - May 2010 - can you recover a pool if you lose the zil (b134+)

I was messing around with a ramdisk on a pool and I forgot to remove it
before I shut down the server.  Now I am not able to mount the pool.  I am
not concerned with the data in this pool, but I would like to try to figure
out how to recover it.  

I am running Nexenta 3.0 NCP (b134+).

I have tried a couple of the commands (zpool import -f and zpool import -FX
llift) 

root at zfs1:/export/home/gnordli# zpool import -f
  pool: llift
    id: 15946357767934802606
 state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://www.sun.com/msg/ZFS-8000-6X
config:

        llift        UNAVAIL  missing device
          mirror-0   ONLINE
            c4t8d0   ONLINE
            c4t9d0   ONLINE
          mirror-1   ONLINE
            c4t10d0  ONLINE
            c4t11d0  ONLINE

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.


root at zfs1:/export/home/gnordli# zpool import -FX llift
cannot import ''llift'': no such pool or dataset
        Destroy and re-create the pool from
        a backup source.



I do not have a copy of the "zpool.cache" file.

Any other commands I could try to recover it or is it just unrecoverable?  

Thanks,

Geoff

Hi Geoff,
I also tested a ram disk as a zil and found I could recover the pool:-
ramdiskadm -a zil 1g
zpool create -f tank c1t3d0 c1t4d0 log /dev/ramdisk/zil
zpool status tank

reboot

zpool status tank
ramdiskadm -a zil 1g
zpool replace -f tank /dev/ramdisk/zil
zpool status tank


Cheers
Richard.
-- 
This message posted from opensolaris.org

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Geoff Nordli
> 
> I was messing around with a ramdisk on a pool and I forgot to remove it
> before I shut down the server.  Now I am not able to mount the pool.  I
> am
> not concerned with the data in this pool, but I would like to try to
> figure
> out how to recover it.
> 
> I am running Nexenta 3.0 NCP (b134+).
Try this:
	zpool upgrade
By default, it will just tell you the current versions of zpools, without
actually doing any upgrades.  If your zpool is 19 or greater, then the loss
of a ZIL is not fatal to the pool.  You should be able to "zpool
import" and
then you''ll see a message about "zpool import -F"

If you have zpool < 19, then it''s lost.

BTW, just to make sure you know ... Having a ZIL in RAM makes no sense
whatsoever, except for academic purposes.  For a system in actual usage, you
should either implement nonvolatile ZIL device, or disable ZIL (to be used
with caution.)

On May 17, 2010, at 5:29 PM, Edward Ned Harvey wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Geoff Nordli
>> 
>> I was messing around with a ramdisk on a pool and I forgot to remove it
>> before I shut down the server.  Now I am not able to mount the pool.  I
>> am
>> not concerned with the data in this pool, but I would like to try to
>> figure
>> out how to recover it.
>> 
>> I am running Nexenta 3.0 NCP (b134+).
> 
> Try this:
> 	zpool upgrade
> By default, it will just tell you the current versions of zpools, without
> actually doing any upgrades.  If your zpool is 19 or greater, then the loss
> of a ZIL is not fatal to the pool.  You should be able to "zpool
import" and
> then you''ll see a message about "zpool import -F"
> 
> If you have zpool < 19, then it''s lost.
If you have zpool.cache, then it is not lost, if you  do not have it -
there''s still a chance, as ZIL device details are reflected in the
in-pool config, and it may be possible to extract config copy out of the pool.

> BTW, just to make sure you know ... Having a ZIL in RAM makes no sense
> whatsoever, except for academic purposes.  For a system in actual usage,
you
> should either implement nonvolatile ZIL device, or disable ZIL (to be used
> with caution.)
Be aware that zil_disable has been removed recently and replaced with
''sync'' dataset property.

regards
victor
> 
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

>-----Original Message-----
>From: Edward Ned Harvey [mailto:solaris2 at nedharvey.com]
>Sent: Monday, May 17, 2010 6:29 AM
>>
>> I was messing around with a ramdisk on a pool and I forgot to remove
>> it before I shut down the server.  Now I am not able to mount the
>> pool.  I am not concerned with the data in this pool, but I would like
>> to try to figure out how to recover it.
>>
>> I am running Nexenta 3.0 NCP (b134+).
>
>Try this:
>	zpool upgrade
>By default, it will just tell you the current versions of zpools, without
actually
>doing any upgrades.  If your zpool is 19 or greater, then the loss of a ZIL
is not
>fatal to the pool.  You should be able to "zpool import" and then
you''ll
see a
>message about "zpool import -F"
>
>If you have zpool < 19, then it''s lost.
>
>BTW, just to make sure you know ... Having a ZIL in RAM makes no sense
>whatsoever, except for academic purposes.  For a system in actual usage,
you
>should either implement nonvolatile ZIL device, or disable ZIL (to be used
with
>caution.)
>
Thanks Edward.

The syspool is sitting at level 18 so I assume the old pool is toast.  I was
more curious why nothing was working because there are reports that you can
do it, but it wasn''t working for me.  

This system isn''t in production, I was just testing to see if the zil
was
being used or not.  

Have a great day!

Geoff

I even have this problem on my (productive) backup server. I lost my system-hdd
and my separate ZIL-device while the system crashes and now I''m in
trouble. The old system was running under the least version of osol/dev with zfs
v22.
10 days ago after the servers crashs I was very optimistc of solving the
problems the same day. It''s a long time ago.
After seting up a new systen (osol 2009.06 and updating to the lastest version
of osol/dev with deduplication) I tried to import my backup zpool, but I
can''t.
# zpool import
  pool: tank1
    id: 5048704328421749681
 state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:

        tank1        UNAVAIL  missing device
          raidz2-0   ONLINE
            c7t5d0   ONLINE
            c7t0d0   ONLINE
            c7t6d0   ONLINE
            c7t3d0   ONLINE
            c7t1d0   ONLINE
            c7t4d0   ONLINE
            c7t2d0   ONLINE

# zpool import -f tank1
cannot import ''tank1'': one or more devices is currently
unavailable
        Destroy and re-create the pool from
        a backup source

Any other option (-F, -X, -V, -D) and any combination of them doesn''t
helps.
I can not add / attach / detach / remove a vdev and the ZIL-device either.
In the last ten days I read a lot of threads, guides to solve problems with ZFS
and so on, but not found a solution for my problem. I created a fake-zpool with
separate ZIL-device to combine the ZIL-file with my old zpool for importing
them, but it doesn''t work in course of the different GUID and checksum
(the name I was modifiing by an binary editor).
The output of:
eee at opensolaris:~# zdb -e tank1

Configuration for import:
        vdev_children: 2
        version: 22
        pool_guid: 5048704328421749681
        name: ''tank1''
        state: 0
        hostid: 946038
        hostname: ''opensolaris''
        vdev_tree:
            type: ''root''
            id: 0
            guid: 5048704328421749681
            children[0]:
                type: ''raidz''
                id: 0
                guid: 16723866123388081610
                nparity: 2
                metaslab_array: 23
                metaslab_shift: 30
                ashift: 9
                asize: 7001340903424
                is_log: 0
                create_txg: 4
                children[0]:
                    type: ''disk''
                    id: 0
                    guid: 6858138566678362598
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 0,0:a''
                    whole_disk: 1
                    DTL: 4345
                    create_txg: 4
                    path: ''/dev/dsk/c7t5d0s0''
                    devid: ''id1,sd at
SATA_____SAMSUNG_HD103UJ_______S13PJ1BQ709050/a''
                children[1]:
                    type: ''disk''
                    id: 1
                    guid: 16136237447458434520
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 1,0:a''
                    whole_disk: 1
                    DTL: 4344
                    create_txg: 4
                    path: ''/dev/dsk/c7t0d0s0''
                    devid: ''id1,sd at
SATA_____SAMSUNG_HD103UJ_______S13PJDWQ317311/a''
                children[2]:
                    type: ''disk''
                    id: 2
                    guid: 10876853602231471126
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 2,0:a''
                    whole_disk: 1
                    DTL: 4343
                    create_txg: 4
                    path: ''/dev/dsk/c7t6d0s0''
                    devid: ''id1,sd at
SATA_____Hitachi_HDT72101______STF604MH14S56W/a''
                children[3]:
                    type: ''disk''
                    id: 3
                    guid: 2384677379114262201
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 3,0:a''
                    whole_disk: 1
                    DTL: 4342
                    create_txg: 4
                    path: ''/dev/dsk/c7t3d0s0''
                    devid: ''id1,sd at
SATA_____SAMSUNG_HD103UJ_______S13PJ1NQ811135/a''
                children[4]:
                    type: ''disk''
                    id: 4
                    guid: 15143849195434333247
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 4,0:a''
                    whole_disk: 1
                    DTL: 4341
                    create_txg: 4
                    path: ''/dev/dsk/c7t1d0s0''
                    devid: ''id1,sd at
SATA_____Hitachi_HDT72101______STF604MH16V73W/a''
                children[5]:
                    type: ''disk''
                    id: 5
                    guid: 11627603446133164653
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 5,0:a''
                    whole_disk: 1
                    DTL: 4340
                    create_txg: 4
                    path: ''/dev/dsk/c7t4d0s0''
                    devid: ''id1,sd at
SATA_____SAMSUNG_HD103UJ_______S13PJDWQ317308/a''
                children[6]:
                    type: ''disk''
                    id: 6
                    guid: 15036924286456611863
                    phys_path: ''/pci at 0,0/pci8086,244e at
1e/pci11ab,11ab at 9/disk at 6,0:a''
                    whole_disk: 1
                    DTL: 4338
                    create_txg: 4
                    path: ''/dev/dsk/c7t2d0s0''
                    devid: ''id1,sd at
SATA_____Hitachi_HDS72101______JP2921HQ0KMEZA/a''
            children[1]:
                type: ''missing''
                id: 1
                guid: 0
^[[D^[[D
doesn''t gave me the GUID of the old ZIL-device and ends without a
prompt.

I need some help, pleaaaaaaaaaase!
Thanks for all replies.
Ron
-- 
This message posted from opensolaris.org

Is there a best practice on keeping a backup of the zpool.cache file? Is it
possible? Does it change with changes to vdevs?
-- 
This message posted from opensolaris.org

On May 25, 2010, at 7:46 AM, thomas wrote:
> Is there a best practice on keeping a backup of the zpool.cache file?
Same as anything else, but a little bit easier because you can
snapshot the root pool.  Thus far, the only real use for the backups
is for a manual recovery of missing top-level vdevs -- a rare event.
> Is it possible?
Yes
> Does it change with changes to vdevs?
Yes
 -- richard

-- 
ZFS and NexentaStor training, Rotterdam, July 13-15, 2010
http://nexenta-rotterdam.eventbrite.com/

> a manual recovery of missing top-level vdevs
> -- a rare event.
Yes, but so rare that I never thought troubling me. In my mind it was only the
slog and loosing the last few seconds doesn''t wrong. So I
don''t have a backup, a snapshot neither the original zpool.cache file.
Is there any solution for my problem?

Thanks
Ron
-- 
This message posted from opensolaris.org

On May 25, 2010, at 12:33 PM, R. Eulenberg wrote:
>> a manual recovery of missing top-level vdevs
>> -- a rare event.
> Yes, but so rare that I never thought troubling me. In my mind it was only
the slog and loosing the last few seconds doesn''t wrong. So I
don''t have a backup, a snapshot neither the original zpool.cache file.
> Is there any solution for my problem?
The description that Peter Woodman put together is a good reference.
http://github.com/pjjw/logfix

If you don''t know the GUID, then it is a rather long trial-and-error
process.
Or you might recompile the source and rip out the parts looking for the
separate log.
 -- richard

-- 
ZFS and NexentaStor training, Rotterdam, July 13-15, 2010
http://nexenta-rotterdam.eventbrite.com/