zfs discuss - Jan 2006 - Using same ZFS under different kernel versions

I build two zfs filesystems using b29 (from brandz).

I then re-installed solaris express b28, preserving the zfs filesystems.
When I tried to "zpool import" my zfs filesystems I got a kernel
panic:
> debugging crash dump vmcore.0 (32-bit) from blackbird
> operating system: 5.11 snv_28 (i86pc)
> panic message:
> ZFS: bad checksum (read on /dev/dsk/c1d0p0 off 24d5e000: zio d3d9d800 
> [L0 DMU objset] vdev=0 offset=2495e000 size=400L/200P/200A fletcher4 
> lzjb LE contiguous birth=2174 fill=17 cksum=82ee33d1a:38baba
> dump content: kernel pages only

I then bfu-ed up to b29 and tried to zpool import and get another kernel 
panic.
> debugging crash dump vmcore.9 (32-bit) from blackbird
> operating system: 5.11 brandz_release-2005-12-23 (i86pc)
> panic message:
> ZFS: bad checksum (read on /dev/dsk/c1d0p0 off 24d5e000: zio c8436940 
> [L0 DMU objset] vdev=0 offset=2495e000 size=400L/200P/200A fletcher4 
> lzjb LE contiguous birth=2174 fill=17 cksum=82ee33d1a:38baba
> dump content: kernel pages only

Was I incorrectly using the same zfs filesystem under different versions 
of the kernel?

Is recovery possible?

Does zdb only work when the zfs pool has been imported?

Regards,
Thomas

>I build two zfs filesystems using b29 (from brandz).
>
>I then re-installed solaris express b28, preserving the zfs filesystems.
>When I tried to "zpool import" my zfs filesystems I got a kernel
panic:
>
>> debugging crash dump vmcore.0 (32-bit) from blackbird
>> operating system: 5.11 snv_28 (i86pc)
>> panic message:
>> ZFS: bad checksum (read on /dev/dsk/c1d0p0 off 24d5e000: zio d3d9d800 
>> [L0 DMU objset] vdev=0 offset=2495e000 size=400L/200P/200A fletcher4 
>> lzjb LE contiguous birth=2174 fill=17 cksum=82ee33d1a:38baba
>> dump content: kernel pages only
>
>
>I then bfu-ed up to b29 and tried to zpool import and get another kernel 
>panic.
>
>> debugging crash dump vmcore.9 (32-bit) from blackbird
>> operating system: 5.11 brandz_release-2005-12-23 (i86pc)
>> panic message:
>> ZFS: bad checksum (read on /dev/dsk/c1d0p0 off 24d5e000: zio c8436940 
>> [L0 DMU objset] vdev=0 offset=2495e000 size=400L/200P/200A fletcher4 
>> lzjb LE contiguous birth=2174 fill=17 cksum=82ee33d1a:38baba
>> dump content: kernel pages only
>
>
>Was I incorrectly using the same zfs filesystem under different versions 
>of the kernel?
No; I have moved the same filesystem forward from 27 to 30 (in steps
and between systems).
>Is recovery possible?
In this particular case, you seem to be having a checksum error which
means there''s on disk data corruption.

I''ll let the zfs dive deeper into this.  What type disk is it?
How was it given to Solaris?  What type of system is it in?

Casper

Casper.Dik at Sun.COM wrote:
>>I build two zfs filesystems using b29 (from brandz).
>>
>>I then re-installed solaris express b28, preserving the zfs filesystems.
>>When I tried to "zpool import" my zfs filesystems I got a
kernel panic:
>>
>>    
>>
>>>debugging crash dump vmcore.0 (32-bit) from blackbird
>>>operating system: 5.11 snv_28 (i86pc)
>>>panic message:
>>>ZFS: bad checksum (read on /dev/dsk/c1d0p0 off 24d5e000: zio
d3d9d800
>>>[L0 DMU objset] vdev=0 offset=2495e000 size=400L/200P/200A fletcher4
>>>lzjb LE contiguous birth=2174 fill=17 cksum=82ee33d1a:38baba
>>>dump content: kernel pages only
>>>      
>>>
>>I then bfu-ed up to b29 and tried to zpool import and get another kernel
>>panic.
>>
>>    
>>
>>>debugging crash dump vmcore.9 (32-bit) from blackbird
>>>operating system: 5.11 brandz_release-2005-12-23 (i86pc)
>>>panic message:
>>>ZFS: bad checksum (read on /dev/dsk/c1d0p0 off 24d5e000: zio
c8436940
>>>[L0 DMU objset] vdev=0 offset=2495e000 size=400L/200P/200A fletcher4
>>>lzjb LE contiguous birth=2174 fill=17 cksum=82ee33d1a:38baba
>>>dump content: kernel pages only
>>>      
>>>
>>Was I incorrectly using the same zfs filesystem under different versions
>>of the kernel?
>>    
>>
>
>No; I have moved the same filesystem forward from 27 to 30 (in steps
>and between systems).
>
>  
>
>>Is recovery possible?
>>    
>>
>
>In this particular case, you seem to be having a checksum error which
>means there''s on disk data corruption.
>
>I''ll let the zfs dive deeper into this.  What type disk is it?
>How was it given to Solaris?  What type of system is it in?
>  
>
I have made dozens of zfs file systems on a dozen different
drives and have been having quite a lot of fun mixing and matching
them on 5 different x86 and x86/64 computers.  Everything was
rock solid until I re-installed solaris express b28 and "preserved"
the two zfs filesystems on my IBM Thinkpad x31 laptop.  When the
kernel started up after the re-install, "format" reported that zfs
was using the two partitions.  I then ran "zfs import" on either
of the drives and would get the same kernel panic.  I tried to
import on 2 different machines with both b28 and b29 kernels
and also got the same kernel panic.

Both drives are laptop ATA drives.  One of the zfs filesystems
was created on a partition and the other was on a whole drive.

Did I err in using a partition for zfs?  I know the documentation
recommended using the whole disk. 

I am curious as why the crash reported "read on /dev/dsk/c1d0p0"
when the zfs filesystem was created on /dev/dsk/c1d0p7?

Regards,
Thomas

Your last comment -- about p0 instead of p7 -- makes me suspect that
you''re hitting an import/export bug that can arise when a disk has
multiple slices that start or end at the same place.  Eric Schrock
fixed this in build 31, under this bugid:

6344272 re-think how whole disks are stored

Eric, are there any interim workarounds you can suggest?

Jeff
> I have made dozens of zfs file systems on a dozen different
> drives and have been having quite a lot of fun mixing and matching
> them on 5 different x86 and x86/64 computers.  Everything was
> rock solid until I re-installed solaris express b28 and
"preserved"
> the two zfs filesystems on my IBM Thinkpad x31 laptop.  When the
> kernel started up after the re-install, "format" reported that
zfs
> was using the two partitions.  I then ran "zfs import" on either
> of the drives and would get the same kernel panic.  I tried to
> import on 2 different machines with both b28 and b29 kernels
> and also got the same kernel panic.
> 
> Both drives are laptop ATA drives.  One of the zfs filesystems
> was created on a partition and the other was on a whole drive.
> 
> Did I err in using a partition for zfs?  I know the documentation
> recommended using the whole disk. 
> 
> I am curious as why the crash reported "read on /dev/dsk/c1d0p0"
> when the zfs filesystem was created on /dev/dsk/c1d0p7?

Hello Jeff,

Thursday, January 5, 2006, 9:11:38 AM, you wrote:

JB> Your last comment -- about p0 instead of p7 -- makes me suspect that
JB> you''re hitting an import/export bug that can arise when a disk
has
JB> multiple slices that start or end at the same place.  Eric Schrock
JB> fixed this in build 31, under this bugid:

JB> 6344272 re-think how whole disks are stored

JB> Eric, are there any interim workarounds you can suggest?

I''m not Eric but... changing other slices so they do not start/end at
the same place solves a problem (it''s not necessary to have overlaping
slices when one of them is dedicated to zfs, the only exception
normally is s2 when you put zfs on other slice and it starts at 0 -
just changing s2 in this case helps).


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl

On Thu, Jan 05, 2006 at 12:11:38AM -0800, Jeff Bonwick
wrote:
> Your last comment -- about p0 instead of p7 -- makes me suspect that
> you''re hitting an import/export bug that can arise when a disk has
> multiple slices that start or end at the same place.  Eric Schrock
> fixed this in build 31, under this bugid:
> 
> 6344272 re-think how whole disks are stored
> 
> Eric, are there any interim workarounds you can suggest?
Actually, the above didn''t quite fix it.  The bug specific to this
problem is likely:

6362672 import gets confused about overlapping slices

But if you didn''t do an explicit import or export, it may be something
else.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Eric Schrock wrote:
>On Thu, Jan 05, 2006 at 12:11:38AM -0800, Jeff Bonwick wrote:
>  
>
>>Your last comment -- about p0 instead of p7 -- makes me suspect that
>>you''re hitting an import/export bug that can arise when a disk
has
>>multiple slices that start or end at the same place.  Eric Schrock
>>fixed this in build 31, under this bugid:
>>
>>6344272 re-think how whole disks are stored
>>
>>Eric, are there any interim workarounds you can suggest?
>>    
>>
>
>Actually, the above didn''t quite fix it.  The bug specific to this
>problem is likely:
>
>6362672 import gets confused about overlapping slices
>
>But if you didn''t do an explicit import or export, it may be
something
>else.
>  
>
I tried to copy the partition to a file in /tmp and
then ran "zfs import -d /tmp" and I got:

      FAULTED insufficient replicas
      FAULTED cannot open.

I then used lofiadm to mount the file as a loopback
device and issued a "zfs import /dev/lofi -f -R/path name"
and it worked!  I was able to recover the zfs completely.

Thanks,
Thomas

Robert Milkowski wrote:
>Hello Jeff,
>
>Thursday, January 5, 2006, 9:11:38 AM, you wrote:
>
>JB> Your last comment -- about p0 instead of p7 -- makes me suspect that
>JB> you''re hitting an import/export bug that can arise when a
disk has
>JB> multiple slices that start or end at the same place.  Eric Schrock
>JB> fixed this in build 31, under this bugid:
>
>JB> 6344272 re-think how whole disks are stored
>
>JB> Eric, are there any interim workarounds you can suggest?
>
>I''m not Eric but... changing other slices so they do not start/end
at
>the same place solves a problem (it''s not necessary to have
overlaping
>slices when one of them is dedicated to zfs, the only exception
>normally is s2 when you put zfs on other slice and it starts at 0 -
>just changing s2 in this case helps).
>  
>
My zfs was the last slice on the disk and overlapped s2.

How do you change the s2 slice size when the drive is mounted?

I was scared to use prtvtoc and fmthd.

Regards,
Thomas
>
>  
>

Hello Thomas,

Thursday, January 5, 2006, 10:38:28 PM, you wrote:

TM> Robert Milkowski wrote:
>>Hello Jeff,
>>
>>Thursday, January 5, 2006, 9:11:38 AM, you wrote:
>>
>>JB> Your last comment -- about p0 instead of p7 -- makes me suspect
that
>>JB> you''re hitting an import/export bug that can arise when
a disk has
>>JB> multiple slices that start or end at the same place.  Eric
Schrock
>>JB> fixed this in build 31, under this bugid:
>>
>>JB> 6344272 re-think how whole disks are stored
>>
>>JB> Eric, are there any interim workarounds you can suggest?
>>
>>I''m not Eric but... changing other slices so they do not
start/end at
>>the same place solves a problem (it''s not necessary to have
overlaping
>>slices when one of them is dedicated to zfs, the only exception
>>normally is s2 when you put zfs on other slice and it starts at 0 -
>>just changing s2 in this case helps).
>>  
>>
TM> My zfs was the last slice on the disk and overlapped s2.

TM> How do you change the s2 slice size when the drive is mounted?

TM> I was scared to use prtvtoc and fmthd.

You can use format from s10-ga - it will work.
Or, as pointed by Eric, you can: "You should be able to work around this
by setting ''NOINUSE_CHECK'' in your environment." and then
run format.

I did use format from s10-ga.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                   http://milek.blogspot.com