Hi all,
We have a server with a LSI 9240-4i controller configured in JBOD with
4 SATA disks. Running FreeBSD 9.1-Beta1:
Relevant dmesg:
FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012
root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a
Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 17179869184 (16384 MB)
avail memory = 16471670784 (15708 MB)
...
mfi0: <Drake Skinny> port 0xe000-0xe0ff mem
0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on
pci1
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
...
mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host
mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0073/1000/9241/1000)
mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0073/1000/9241/1000)
mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107
mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A
mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3)
...
mfisyspd0 on mfi0
mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd0: SYSPD volume attached
mfisyspd1 on mfi0
mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd1: SYSPD volume attached
mfisyspd2 on mfi0
mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd2: SYSPD volume attached
mfisyspd3 on mfi0
mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd3: SYSPD volume attached
...
mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info:
enclPd=ffff, scsiType=0, portMap=00,
sasAddr=4433221100000000,0000000000000000
mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1)
mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info:
enclPd=ffff, scsiType=0, portMap=02,
sasAddr=4433221102000000,0000000000000000
mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2)
mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info:
enclPd=ffff, scsiType=0, portMap=03,
sasAddr=4433221101000000,0000000000000000
mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0)
mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info:
enclPd=ffff, scsiType=0, portMap=01,
sasAddr=4433221103000000,0000000000000000
mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12
16:32:56; (28 seconds since power on)
The problem:
When trying to create a RaidZ pool using gpart and perform a 4K
alignment using gnop, we get the follwoing error immediately after
exporting the pool and destroying the .nop devices:
id: 8043746387654554958
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-5E
config:
Pool FAULTED corrupted data
raidz1-0 ONLINE
13283347160590042564 UNAVAIL corrupted data
16981727992215676534 UNAVAIL corrupted data
6607570030658834339 UNAVAIL corrupted data
3435463242860701988 UNAVAIL corrupted data
When we use glabel for the same purpose with the combination of gnop,
the pool imports fine.
Any suggestions?
--
George Kontostanos
Aicom telecoms ltd
http://www.aisecure.net
Hi, Works fine for me with 9.0-p3 with the same controller, slight firmware differences: mfi0: 2709 (boot + 4s/0x0020/info) - Firmware version 2.120.244-1482 mfi0: 2710 (boot + 5s/0x0020/info) - Package version 20.10.1-0077 mfi0: 2711 (boot + 5s/0x0020/info) - Board Revision 04A I will be trying out 9.1-BETA on a new batch of machines next week ... Regards, Jan. On 08/08/2012, at 7:25 AM, George Kontostanos <gkontos.mail@gmail.com> wrote:> Hi all, > > We have a server with a LSI 9240-4i controller configured in JBOD with > 4 SATA disks. Running FreeBSD 9.1-Beta1: > > Relevant dmesg: > > FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012 > root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX> > AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> > AMD Features2=0x1<LAHF> > TSC: P-state invariant, performance statistics > real memory = 17179869184 (16384 MB) > avail memory = 16471670784 (15708 MB) > ... > mfi0: <Drake Skinny> port 0xe000-0xe0ff mem > 0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on > pci1 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > ... > mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host > mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started > (PCI ID 0073/1000/9241/1000) > mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 > mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started > (PCI ID 0073/1000/9241/1000) > mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 > mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107 > mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A > mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) > ... > mfisyspd0 on mfi0 > mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd0: SYSPD volume attached > mfisyspd1 on mfi0 > mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd1: SYSPD volume attached > mfisyspd2 on mfi0 > mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd2: SYSPD volume attached > mfisyspd3 on mfi0 > mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd3: SYSPD volume attached > ... > mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info: > enclPd=ffff, scsiType=0, portMap=00, > sasAddr=4433221100000000,0000000000000000 > mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) > mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info: > enclPd=ffff, scsiType=0, portMap=02, > sasAddr=4433221102000000,0000000000000000 > mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) > mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info: > enclPd=ffff, scsiType=0, portMap=03, > sasAddr=4433221101000000,0000000000000000 > mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) > mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info: > enclPd=ffff, scsiType=0, portMap=01, > sasAddr=4433221103000000,0000000000000000 > mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12 > 16:32:56; (28 seconds since power on) > > The problem: > > When trying to create a RaidZ pool using gpart and perform a 4K > alignment using gnop, we get the follwoing error immediately after > exporting the pool and destroying the .nop devices: > > id: 8043746387654554958 > state: FAULTED > status: One or more devices contains corrupted data. > action: The pool cannot be imported due to damaged devices or data. > The pool may be active on another system, but can be imported using > the '-f' flag. > see: http://illumos.org/msg/ZFS-8000-5E > config: > > Pool FAULTED corrupted data > raidz1-0 ONLINE > 13283347160590042564 UNAVAIL corrupted data > 16981727992215676534 UNAVAIL corrupted data > 6607570030658834339 UNAVAIL corrupted data > 3435463242860701988 UNAVAIL corrupted data > > When we use glabel for the same purpose with the combination of gnop, > the pool imports fine. > > Any suggestions? > > -- > George Kontostanos > Aicom telecoms ltd > http://www.aisecure.net > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
On Wed, Aug 8, 2012 at 10:05 AM, Jan Mikkelsen <janm@transactionware.com> wrote:> Hi, > > Works fine for me with 9.0-p3 with the same controller, slight firmware differences: > > mfi0: 2709 (boot + 4s/0x0020/info) - Firmware version 2.120.244-1482 > mfi0: 2710 (boot + 5s/0x0020/info) - Package version 20.10.1-0077 > mfi0: 2711 (boot + 5s/0x0020/info) - Board Revision 04A > > I will be trying out 9.1-BETA on a new batch of machines next week ... > > Regards, > > Jan. >Hi can you clarify if this was done using gpart and gnop? Thanks
On 8 Aug, George Kontostanos wrote:> Hi all, > > We have a server with a LSI 9240-4i controller configured in JBOD with > 4 SATA disks. Running FreeBSD 9.1-Beta1: > > Relevant dmesg: > > FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012 > root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX> > AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> > AMD Features2=0x1<LAHF> > TSC: P-state invariant, performance statistics > real memory = 17179869184 (16384 MB) > avail memory = 16471670784 (15708 MB) > ... > mfi0: <Drake Skinny> port 0xe000-0xe0ff mem > 0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on > pci1 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > ... > mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host > mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started > (PCI ID 0073/1000/9241/1000) > mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 > mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started > (PCI ID 0073/1000/9241/1000) > mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 > mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107 > mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A > mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) > ... > mfisyspd0 on mfi0 > mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd0: SYSPD volume attached > mfisyspd1 on mfi0 > mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd1: SYSPD volume attached > mfisyspd2 on mfi0 > mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd2: SYSPD volume attached > mfisyspd3 on mfi0 > mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd3: SYSPD volume attached > ... > mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info: > enclPd=ffff, scsiType=0, portMap=00, > sasAddr=4433221100000000,0000000000000000 > mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) > mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info: > enclPd=ffff, scsiType=0, portMap=02, > sasAddr=4433221102000000,0000000000000000 > mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) > mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info: > enclPd=ffff, scsiType=0, portMap=03, > sasAddr=4433221101000000,0000000000000000 > mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) > mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info: > enclPd=ffff, scsiType=0, portMap=01, > sasAddr=4433221103000000,0000000000000000 > mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12 > 16:32:56; (28 seconds since power on) > > The problem: > > When trying to create a RaidZ pool using gpart and perform a 4K > alignment using gnop, we get the follwoing error immediately after > exporting the pool and destroying the .nop devices: > > id: 8043746387654554958 > state: FAULTED > status: One or more devices contains corrupted data. > action: The pool cannot be imported due to damaged devices or data. > The pool may be active on another system, but can be imported using > the '-f' flag. > see: http://illumos.org/msg/ZFS-8000-5E > config: > > Pool FAULTED corrupted data > raidz1-0 ONLINE > 13283347160590042564 UNAVAIL corrupted data > 16981727992215676534 UNAVAIL corrupted data > 6607570030658834339 UNAVAIL corrupted data > 3435463242860701988 UNAVAIL corrupted dataI'm planning on doing something similar, but I'm curious about how gnop and GPT labels interact. I want to partition the drives for my pool slightly on the small side so that I'm less likely to run into problems if I have to replace a drive in the future. If I used gpart to create and label a GPT partition on the drive, the partition will show up as /dev/gpt/label. The gnop man page says that running gnop on dev creates /dev/dev.nop. What happens if you gnop /dev/gpt/label? Is this what you are doing?> When we use glabel for the same purpose with the combination of gnop, > the pool imports fine. > > Any suggestions?It should be sufficient to only gnop one of the devices. You should be able to create the pool with only one gnop device to get the ashift value that you desire, export the pool, destroy the .nop device, and import the pool. If it things the device is corrupted (which seems like a bug of some sort), then ZFS should be able to resilver it.
On 8 Aug, George Kontostanos wrote:> The problem: > > When trying to create a RaidZ pool using gpart and perform a 4K > alignment using gnop, we get the follwoing error immediately after > exporting the pool and destroying the .nop devices: > > id: 8043746387654554958 > state: FAULTED > status: One or more devices contains corrupted data. > action: The pool cannot be imported due to damaged devices or data. > The pool may be active on another system, but can be imported using > the '-f' flag. > see: http://illumos.org/msg/ZFS-8000-5E > config: > > Pool FAULTED corrupted data > raidz1-0 ONLINE > 13283347160590042564 UNAVAIL corrupted data > 16981727992215676534 UNAVAIL corrupted data > 6607570030658834339 UNAVAIL corrupted data > 3435463242860701988 UNAVAIL corrupted data > > When we use glabel for the same purpose with the combination of gnop, > the pool imports fine.Might kern/170945 have something to do with this?