Hi.
ZFS based on b36, filled SDR-0202.
It could be related to SDR-0169 (large sequential writing problems thread)
however I didn''t notice such problems then.
I have a v240 with 3511 JBODs directly connected to hosts via two links with
MPxIO. Now if I create a pool write performance is bad - looking at iostat I can
see the some of a disks have very long service times. Now if I create a pool
using different disks the result is the same. Issuing IOs directly to selected
disks using dd looks fine.
bash-3.00# zpool create -f large c6t22E4000A3300FC31d0 c6t22E4000A3300FC32d0
c6t22E4000A3300FC33d0 c6t22E4000A3300FC34d0
bash-3.00# dd if=/dev/zero of=/large/q1 bs=1024k
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 88.0 0.0 11264.9 89.0 26.0 1011.3 295.4 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 44.0 0.0 5632.4 33.0 2.0 749.9 45.4 100 100
c6t22E4000A3300FC32d0
0.0 44.0 0.0 5632.5 26.0 9.0 590.8 204.5 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
bash-3.00# zpool create -f large c6t22E4000A3300FC31d0 c6t22E4000A3300FC32d0
c6t22E4000A3300FC33d0 c6t22E4000A3300FC34d0
bash-3.00# dd if=/dev/zero of=/large/q1 bs=1024k
bash-3.00# iostat -xnzC 1
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 88.0 0.0 11264.9 89.0 26.0 1011.3 295.4 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 44.0 0.0 5632.4 33.0 2.0 749.9 45.4 100 100
c6t22E4000A3300FC32d0
0.0 44.0 0.0 5632.5 26.0 9.0 590.8 204.5 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 88.0 0.0 11264.8 89.0 26.0 1011.3 295.4 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 44.0 0.0 5632.4 33.0 2.0 749.9 45.4 100 100
c6t22E4000A3300FC32d0
0.0 44.0 0.0 5632.4 26.0 9.0 590.8 204.5 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 87.0 0.0 11134.1 89.0 26.0 1023.1 298.9 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 43.0 0.0 5503.1 33.0 2.0 767.5 46.5 100 100
c6t22E4000A3300FC32d0
0.0 44.0 0.0 5631.0 26.0 9.0 591.0 204.6 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 87.0 0.0 11136.2 89.0 26.0 1022.9 298.8 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 44.0 0.0 5632.1 33.0 2.0 750.0 45.5 100 100
c6t22E4000A3300FC32d0
0.0 43.0 0.0 5504.1 26.0 9.0 604.6 209.3 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 4.0 0.0 2.0 0.0 0.0 0.0 12.3 0 5 c1
0.0 2.0 0.0 1.0 0.0 0.0 0.0 14.9 0 3 c1t1d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 9.7 0 2 c1t0d0
0.0 88.0 0.0 11267.1 89.0 26.0 1011.1 295.4 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 44.0 0.0 5633.5 33.0 2.0 749.8 45.4 100 100
c6t22E4000A3300FC32d0
0.0 44.0 0.0 5633.5 26.0 9.0 590.7 204.5 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 88.0 0.0 11259.9 89.0 26.0 1011.7 295.6 0 400 c6
0.0 0.0 0.0 0.0 30.0 5.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 44.0 0.0 5630.0 33.0 2.0 750.2 45.5 100 100
c6t22E4000A3300FC32d0
0.0 44.0 0.0 5630.0 26.0 9.0 591.1 204.6 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0 100
c6t22E4000A3300FC34d0
^C
bash-3.00#
sometimes other disks have the same problem.
bash-3.00# zpool destroy large
bash-3.00# dd if=/dev/zero of=/dev/rdsk/c6t22E4000A3300FC31d0s0 bs=1024k
bash-3.00# iostat -xnzC 1
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 29.5 0.0 30236.1 0.0 1.0 0.0 33.2 0 98 c6
0.0 29.5 0.0 30236.4 0.0 1.0 0.0 33.2 0 98
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 30.0 0.0 30722.9 0.0 1.0 0.0 32.6 0 98 c6
0.0 30.0 0.0 30722.6 0.0 1.0 0.0 32.6 0 98
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 29.0 0.0 29705.0 0.0 1.0 0.0 33.8 0 98 c6
0.0 29.0 0.0 29704.8 0.0 1.0 0.0 33.8 0 98
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 30.0 0.0 30709.2 0.0 1.0 0.0 32.6 0 98 c6
0.0 30.0 0.0 30709.5 0.0 1.0 0.0 32.6 0 98
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 29.0 0.0 29700.6 0.0 1.0 0.0 33.8 0 98 c6
0.0 29.0 0.0 29700.5 0.0 1.0 0.0 33.8 0 98
c6t22E4000A3300FC31d0
so this disk looks ok.
However if I try to write to a block device instead of raw device then I can see
problems.
bash-3.00# dd if=/dev/zero of=/dev/dsk/c6t22E4000A3300FC31d0s0 bs=1024k
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
5.0 24.0 17.1 176.0 148.1 150.2 5108.9 5182.2 0 59 c6
5.0 24.0 17.1 176.0 148.1 150.2 5108.9 5182.2 58 59
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 256.0 256.0 0.0 0.0 0 100 c6
0.0 0.0 0.0 0.0 256.0 256.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 10.0 0.0 80.0 268.1 243.8 26815.8 24378.2 0 100 c6
0.0 10.0 0.0 80.0 268.1 243.8 26815.8 24378.2 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 276.0 236.0 0.0 0.0 0 100 c6
0.0 0.0 0.0 0.0 276.0 236.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 97.0 0.0 775.9 326.0 186.0 3361.3 1917.4 0 100 c6
0.0 97.0 0.0 775.9 326.0 186.0 3361.3 1917.4 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 379.0 133.0 0.0 0.0 0 100 c6
0.0 0.0 0.0 0.0 379.0 133.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 112.0 173.0 0.0 0.0 0 100 c6
0.0 0.0 0.0 0.0 112.0 173.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 11.0 0.0 88.0 118.6 159.2 10778.6 14471.3 0 100 c6
0.0 11.0 0.0 88.0 118.6 159.2 10778.6 14471.3 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 122.0 152.0 0.0 0.0 0 100 c6
0.0 0.0 0.0 0.0 122.0 152.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 12.0 0.0 96.0 128.6 137.4 10720.5 11453.2 0 100 c6
0.0 12.0 0.0 96.0 128.6 137.4 10720.5 11453.2 100 100
c6t22E4000A3300FC31d0
^C
bash-3.00#
And I get warnings in system log (only if i write to dsk, no warnings if I write
to rdsk).
Apr 24 15:13:36 XXX scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci
(scsi_vhci0):
Apr 24 15:13:36 XXX /scsi_vhci/ssd at g22e4000a3300fc31 (ssd227): Command
Timeout on path /pci at 1d,700000/SUNW,qlc at 1/fp at 0,0 (fp1)
Apr 24 15:13:37 XXX scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci
(scsi_vhci0):
Apr 24 15:13:37 XXX /scsi_vhci/ssd at g22e4000a3300fc31 (ssd227): Command
Timeout on path /pci at 1d,700000/SUNW,qlc at 1/fp at 0,0 (fp1)
Apr 24 15:14:36 XXX scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci
(scsi_vhci0):
Apr 24 15:14:36 XXX /scsi_vhci/ssd at g22e4000a3300fc31 (ssd227): Command
Timeout on path /pci at 1e,600000/SUNW,qlc at 2/fp at 0,0 (fp4)
Apr 24 15:14:47 XXX scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci
(scsi_vhci0):
Apr 24 15:14:47 XXX /scsi_vhci/ssd at g22e4000a3300fc31 (ssd227): Command
Timeout on path /pci at 1e,600000/SUNW,qlc at 2/fp at 0,0 (fp4)
This message posted from opensolaris.org
bash-3.00# fcinfo hba-port -l
HBA Port WWN: 210000e08b825c30
OS Device Name: /dev/cfg/c0
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Type: L-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b825c30
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 210100e08ba25c30
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb
Current Speed: not established
Node WWN: 200100e08ba25c30
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 210000e08b82cf2e
OS Device Name: /dev/cfg/c2
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Type: L-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b82cf2e
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 210100e08ba2cf2e
OS Device Name: /dev/cfg/c5
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb
Current Speed: not established
Node WWN: 200100e08ba2cf2e
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
bash-3.00#
This message posted from opensolaris.org
I create SVM stripe on the same disks and then tried to write to /dev/md/dsk and
/dev/md/rdsk
bash-3.00# metainit d100 1 4 c6t22E4000A3300FC31d0s0 c6t22E4000A3300FC32d0s0
c6t22E4000A3300FC33d0s0 c6t22E4000A3300FC34d0s0
d100: Concat/Stripe is setup
bash-3.00# dd if=/dev/zero of=/dev/md/rdsk/d100 bs=8192k
^C157+0 records in
157+0 records out
iostat:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 986.0 0.0 15776.3 0.0 25.1 0.0 25.5 0 368 c6
0.0 246.0 0.0 3936.1 0.0 6.1 0.0 24.7 0 91
c6t22E4000A3300FC31d0
0.0 247.0 0.0 3952.1 0.0 6.3 0.0 25.5 0 93
c6t22E4000A3300FC32d0
0.0 246.0 0.0 3936.1 0.0 6.4 0.0 26.2 0 91
c6t22E4000A3300FC33d0
0.0 247.0 0.0 3952.1 0.0 6.3 0.0 25.5 0 93
c6t22E4000A3300FC34d0
0.0 15.0 0.0 15360.3 0.0 1.0 0.0 65.7 0 98 d100
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 980.0 0.0 15679.8 0.0 25.9 0.0 26.4 0 367 c6
0.0 247.0 0.0 3951.9 0.0 6.5 0.0 26.4 0 91
c6t22E4000A3300FC31d0
0.0 243.0 0.0 3887.9 0.0 6.5 0.0 26.6 0 93
c6t22E4000A3300FC32d0
0.0 247.0 0.0 3951.9 0.0 6.5 0.0 26.3 0 93
c6t22E4000A3300FC33d0
0.0 243.0 0.0 3887.9 0.0 6.4 0.0 26.4 0 90
c6t22E4000A3300FC34d0
0.0 16.0 0.0 16383.2 0.0 1.0 0.0 61.5 0 98 d100
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 999.0 0.0 15983.4 0.0 25.8 0.0 25.8 0 372 c6
0.0 246.0 0.0 3935.9 0.0 6.3 0.0 25.4 0 91
c6t22E4000A3300FC31d0
0.0 252.0 0.0 4031.9 0.0 6.6 0.0 26.1 0 95
c6t22E4000A3300FC32d0
0.0 249.0 0.0 3983.9 0.0 6.3 0.0 25.5 0 93
c6t22E4000A3300FC33d0
0.0 252.0 0.0 4031.9 0.0 6.6 0.0 26.2 0 93
c6t22E4000A3300FC34d0
0.0 15.0 0.0 15359.3 0.0 1.0 0.0 65.6 0 98 d100
now dsk
bash-3.00# dd if=/dev/zero of=/dev/md/dsk/d100 bs=8192k
^C239+0 records in
239+0 records out
[hungs here for many minutes]
iostat:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 1024.0 1024.0 0.0 0.0 0 400 c6
0.0 0.0 0.0 0.0 256.0 256.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 0.0 0.0 0.0 256.0 256.0 0.0 0.0 100 100
c6t22E4000A3300FC32d0
0.0 0.0 0.0 0.0 256.0 256.0 0.0 0.0 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 256.0 256.0 0.0 0.0 100 100
c6t22E4000A3300FC34d0
0.0 0.0 0.0 0.0 0.0 41516.1 0.0 0.0 3 100 d100
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 601.8 0.0 4814.5 1165.4 881.8 1936.5 1465.2 0 400 c6
0.0 178.9 0.0 1431.6 305.4 206.4 1706.5 1153.4 100 100
c6t22E4000A3300FC31d0
0.0 15.0 0.0 120.0 275.8 236.1 18391.8 15744.4 100 100
c6t22E4000A3300FC32d0
0.0 182.9 0.0 1463.6 302.7 209.1 1654.7 1142.8 100 100
c6t22E4000A3300FC33d0
0.0 224.9 0.0 1799.5 281.5 230.2 1251.7 1023.4 100 100
c6t22E4000A3300FC34d0
0.0 601.8 0.0 4814.4 0.0 73905.4 0.1 122806.3 3 100 d100
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 1141.0 907.0 0.0 0.0 0 400 c6
0.0 0.0 0.0 0.0 303.0 209.0 0.0 0.0 100 100
c6t22E4000A3300FC31d0
0.0 0.0 0.0 0.0 277.0 235.0 0.0 0.0 100 100
c6t22E4000A3300FC32d0
0.0 0.0 0.0 0.0 299.0 213.0 0.0 0.0 100 100
c6t22E4000A3300FC33d0
0.0 0.0 0.0 0.0 262.0 250.0 0.0 0.0 100 100
c6t22E4000A3300FC34d0
0.0 0.0 0.0 0.0 0.0 107352.1 0.0 0.0 4 100 d100
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
1.0 0.0 1.0 0.0 0.0 0.0 0.0 18.8 0 2 c1
1.0 0.0 1.0 0.0 0.0 0.0 0.0 18.8 0 2 c1t1d0
0.0 974.6 0.0 7797.2 1250.2 796.9 1282.7 817.6 0 400 c6
0.0 306.2 0.0 2449.6 345.5 166.2 1128.3 542.9 100 100
c6t22E4000A3300FC31d0
0.0 14.0 0.0 112.1 297.4 214.6 21230.8 15316.8 100 100
c6t22E4000A3300FC32d0
0.0 319.2 0.0 2553.7 336.5 175.3 1054.1 549.1 100 100
c6t22E4000A3300FC33d0
0.0 335.2 0.0 2681.8 270.8 240.8 807.9 718.3 100 100
c6t22E4000A3300FC34d0
1.0 0.0 1.0 0.0 0.0 0.0 0.0 18.9 0 2 d0
1.0 0.0 1.0 0.0 0.0 0.0 0.0 18.9 0 2 d2
0.0 974.7 0.0 7797.7 0.0 137219.2 0.0 140778.6 3 100 d100
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 121.0 0.0 968.4 1246.0 802.0 10293.0 6625.4 0 400 c6
0.0 2.0 0.0 16.0 347.0 165.0 173429.8 82466.7 100 100
c6t22E4000A3300FC31d0
0.0 0.0 0.0 0.0 298.0 214.0 0.0 0.0 100 100
c6t22E4000A3300FC32d0
0.0 13.0 0.0 104.0 338.0 174.0 25988.4 13378.9 100 100
c6t22E4000A3300FC33d0
0.0 106.0 0.0 848.3 263.0 249.0 2479.8 2348.1 100 100
c6t22E4000A3300FC34d0
0.0 121.0 0.0 968.4 0.0 164217.3 0.3 1356614.0 3 100 d100
^C
bash-3.00#
There''s definitely some kind of a problem with accessing those disks -
any idea?
This message posted from opensolaris.org
Robert, Could you check the write_cache on some of these drives: # format -e format> cache cache> write_cache write_cache> display Thanks, George Robert Milkowski wrote:> bash-3.00# fcinfo hba-port -l > HBA Port WWN: 210000e08b825c30 > OS Device Name: /dev/cfg/c0 > Manufacturer: QLogic Corp. > Model: 375-3108-xx > Type: L-port > State: online > Supported Speeds: 1Gb 2Gb > Current Speed: 2Gb > Node WWN: 200000e08b825c30 > Link Error Statistics: > Link Failure Count: 0 > Loss of Sync Count: 0 > Loss of Signal Count: 0 > Primitive Seq Protocol Error Count: 0 > Invalid Tx Word Count: 0 > Invalid CRC Count: 0 > HBA Port WWN: 210100e08ba25c30 > OS Device Name: /dev/cfg/c3 > Manufacturer: QLogic Corp. > Model: 375-3108-xx > Type: unknown > State: offline > Supported Speeds: 1Gb 2Gb > Current Speed: not established > Node WWN: 200100e08ba25c30 > Link Error Statistics: > Link Failure Count: 0 > Loss of Sync Count: 0 > Loss of Signal Count: 0 > Primitive Seq Protocol Error Count: 0 > Invalid Tx Word Count: 0 > Invalid CRC Count: 0 > HBA Port WWN: 210000e08b82cf2e > OS Device Name: /dev/cfg/c2 > Manufacturer: QLogic Corp. > Model: 375-3108-xx > Type: L-port > State: online > Supported Speeds: 1Gb 2Gb > Current Speed: 2Gb > Node WWN: 200000e08b82cf2e > Link Error Statistics: > Link Failure Count: 0 > Loss of Sync Count: 0 > Loss of Signal Count: 0 > Primitive Seq Protocol Error Count: 0 > Invalid Tx Word Count: 0 > Invalid CRC Count: 0 > HBA Port WWN: 210100e08ba2cf2e > OS Device Name: /dev/cfg/c5 > Manufacturer: QLogic Corp. > Model: 375-3108-xx > Type: unknown > State: offline > Supported Speeds: 1Gb 2Gb > Current Speed: not established > Node WWN: 200100e08ba2cf2e > Link Error Statistics: > Link Failure Count: 0 > Loss of Sync Count: 0 > Loss of Signal Count: 0 > Primitive Seq Protocol Error Count: 0 > Invalid Tx Word Count: 0 > Invalid CRC Count: 0 > bash-3.00# > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Hello George,
Monday, April 24, 2006, 4:13:41 PM, you wrote:
GW> Robert,
GW> Could you check the write_cache on some of these drives:
All 4 drives have write cache enabled.
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
One thing ZFS does differently is to issue (via DKIOCFLUSHWRITECACHE) the SCSI synchronize_cache command, which could cause something like this, especially if they happen to be older drives (which I kinda doubt in your case). If you feel comfortable with mdb, run ::spa -v and set vdev_nowritecache for each vdev to turn off this logic (it''s in vdev_disk_io_start()) and see if it helps. If this isn''t clear enough, let us know and we''ll get you more specific instructions. --Bill On Mon, Apr 24, 2006 at 04:57:56PM +0200, Robert Milkowski wrote:> Hello George, > > Monday, April 24, 2006, 4:13:41 PM, you wrote: > > GW> Robert, > > GW> Could you check the write_cache on some of these drives: > > All 4 drives have write cache enabled. > > -- > Best regards, > Robert mailto:rmilkowski at task.gda.pl > http://milek.blogspot.com > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Hi. Sorry for delay - too much going on. bash-3.00# uname -a SunOS XXX 5.10 zfs-s10-0315 sun4u sparc SUNW,Sun-Fire-V240 bash-3.00# bash-3.00# mdb -kw Loading modules: [ unix krtld genunix specfs dtrace ufs sd md ip sctp usba fcp fctl lofs zfs random logindmux ptm cpc fcip crypto ssd nfs ]> ::spa -vADDR STATE NAME 000006000354f680 ACTIVE large1 ADDR STATE AUX DESCRIPTION 0000060003b9efc0 HEALTHY - root 00000600037a1040 HEALTHY - /dev/dsk/c6t22E4000A3300FB7Dd0s0 00000600037a0b00 HEALTHY - /dev/dsk/c6t22E4000A3300FB7Ed0s0 0000060003b9e540 HEALTHY - /dev/dsk/c6t22E4000A3300FB7Fd0s0 0000060003b9e000 HEALTHY - /dev/dsk/c6t22E4000A3300FB80d0s0 000003004fe51a80 HEALTHY - /dev/dsk/c6t22E4000A3300FB81d0s0 000003004fe51540 HEALTHY - /dev/dsk/c6t22E4000A3300FB82d0s0 000003004fe51000 HEALTHY - /dev/dsk/c6t22E4000A3300FB83d0s0 000003004fe50ac0 HEALTHY - /dev/dsk/c6t22E4000A3300FB84d0s0 000003004fe50580 HEALTHY - /dev/dsk/c6t22E4000A3300FB85d0s0 000003004fe50040 HEALTHY - /dev/dsk/c6t22E4000A3300FB86d0s0 000003004fe4fac0 HEALTHY - /dev/dsk/c6t22E4000A3300FB87d0s0 000003004fe4f580 HEALTHY - /dev/dsk/c6t22E4000A3300FB88d0s0> > 00000600037a1040::print struct vdev vdev_nowritecachemdb: failed to find member vdev_nowritecache of struct vdev: no such member of structure or union> 00000600037a1040::print struct vdev vdev_last_tryvdev_last_try = 0>Well there''s deinitely vdev_nowritecache in current source. http://cvs.opensolaris.org/source/diff/on/usr/src/uts/common/fs/zfs/sys/vdev_impl.h?r2=1.8&r1=1.7 So it was introduced lately - definitely after zfs-s10-0315. Any idea what to do with that build? Where there any other changes which could change observed behaviour so my config would work? (of course I can upgrade to b37 or b38). This message posted from opensolaris.org
I can also say it''s not MPxIO - I turned it off imported pool and have the same bahaviour. I also tried to turn off write cache using format -e on all disks in a pool - no effect (however I''m not sure how ZFS would deal with this - I did it while pool was already imported). These are 500GB SATA disks from Sun in 3511s JBODs so they are quite new. This message posted from opensolaris.org
James C. McPherson
2006-Apr-30 10:40 UTC
[zfs-discuss] Re: Problems with ZFS and 3511 JBODs
Robert Milkowski wrote:> I can also say it''s not MPxIO - I turned it off imported pool and have > the same bahaviour. I also tried to turn off write cache using format -e > on all disks in a pool - no effect (however I''m not sure how ZFS would > deal with this - I did it while pool was already imported). > These are 500GB SATA disks from Sun in 3511s JBODs so they are quite new.Hi Robert, could you provide some details of your san modules please: modinfo |egrep "SunFC|SCSI" And are you up to date with your 3511 firmware? thanks in advance, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
Hello James, Sunday, April 30, 2006, 12:40:09 PM, you wrote: JCM> Robert Milkowski wrote:>> I can also say it''s not MPxIO - I turned it off imported pool and have >> the same bahaviour. I also tried to turn off write cache using format -e >> on all disks in a pool - no effect (however I''m not sure how ZFS would >> deal with this - I did it while pool was already imported). >> These are 500GB SATA disks from Sun in 3511s JBODs so they are quite new.JCM> Hi Robert, JCM> could you provide some details of your san modules please: JCM> modinfo |egrep "SunFC|SCSI" bash-3.00# uname -a SunOS XXXXX 5.10 zfs-s10-0315 sun4u sparc SUNW,Sun-Fire-V240 bash-3.00# modinfo | egrep "SunFC|SCSI" 19 123dc50 e368 189 1 scsi_vhci (SCSI VHCI Driver 1.42) 20 124aac8 f2b8 - 1 scsi (SCSI Bus Utility Routines) 21 12513c0 24c80 32 1 sd (SCSI Disk Driver 1.493) 22 126e988 11418 50 1 glm (GLM SCSI HBA Driver 1.206.) 37 1316950 c59c8 108 1 qlc (SunFC Qlogic FCA v20051013-2.08) 38 1356190 88a8 - 1 fctl (SunFC Transport v20060208-1.50) 39 135d150 183c8 94 1 fp (SunFC Port v20060208-1.69) 40 1374df0 180b8 99 1 fcp (SunFC FCP v20060208-1.97) 41 7bb44000 255d0 118 1 ssd (SCSI SSA/FCAL Disk Driver 1.493) 42 7bae8000 6bc0 80 1 ses (SCSI Enclosure Services Driver ) 45 7be54000 2a870 86 1 iscsi (Sun iSCSI Initiator v20060208-0) bash-3.00# JCM> And are you up to date with your 3511 firmware? bash-3.00# luxadm display -v /dev/rdsk/c0t122d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/c0t122d0s2 Vendor: HITACHI Product ID: HDS7250SASUN500G Revision: A20A Serial Num: KRVN63ZAGPE1 Unformatted capacity: 476936.000 MBytes Read Cache: Enabled Minimum prefetch: 0x0 Maximum prefetch: 0x0 Device Type: Disk device Path(s): /dev/rdsk/c0t122d0s2 /devices/pci at 1e,600000/SUNW,qlc at 2/fp at 0,0/ssd at w22e5000a3300fb63,0:c,raw LUN path port WWN: 22e5000a3300fb63 Host controller port WWN: 210000e08b825c30 Path status: O.K. /dev/rdsk/c2t122d0s2 /devices/pci at 1d,700000/SUNW,qlc at 1/fp at 0,0/ssd at w22e6000a3300fb63,0:c,raw LUN path port WWN: 22e6000a3300fb63 Host controller port WWN: 210000e08b82cf2e Path status: O.K. bash-3.00# sccli: * ses Id Chassis Vendor/Product ID Rev PLD WWNN WWPN ---------------------------------------------------------------------------- 28 092627 SUN StorEdge 3511F D 0413 1000 205000C0FF092627 215000C0FF092627 Topology: loop(a) * enclosure-status Id Chassis Vendor/Product ID Rev PLD WWNN WWPN ---------------------------------------------------------------------------- 28 092627 SUN StorEdge 3511F D 0413 1000 205000C0FF092627 215000C0FF092627 Topology: loop(a) Status: OK So it''s not the latest - I''ll try to upgrade and see if it helps. ps. sorry for late response -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com