freebsd stable - May 2014 - gmirror + ZFS issues

Hello,

I've got a new server and I installed FreeBSD 10 on it. I have a problem 
to create a new zfs pool. The command stalls on IO wait (D state / 
zio->io_cv).

The device for the pool is a 1.7T partition (index 4) of a gmirror device.

root at daedalus:~ # uname -a
FreeBSD daedalus 10.0-RELEASE-p3 FreeBSD 10.0-RELEASE-p3 #0: Tue May 13 
18:31:10 UTC 2014 
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


root at daedalus:~ # dmesg | grep ada
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <HGST HUS724020ALA640 MF6OAA70> ATA-8 SATA 3.x device
ada0: Serial Number PN2134P6HE028P
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <HGST HUS724020ALA640 MF6OAA70> ATA-8 SATA 3.x device
ada1: Serial Number PN2134P6HAZTVP
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6


root at daedalus:~ # gmirror status
       Name    Status  Components
mirror/gm0  COMPLETE  ada0 (ACTIVE)
                       ada1 (ACTIVE)


root at daedalus:~ # gpart show
=>        63  3907029104  mirror/gm0  MBR  (1.8T)
           63  3907029042           1  freebsd  [active]  (1.8T)
   3907029105          62              - free -  (31K)

=>         0  3907029042  mirror/gm0s1  BSD  (1.8T)
            0   204800000             1  freebsd-ufs  (98G)
    204800000    20971520             2  freebsd-swap  (10G)
    225771520  3681257522             4  freebsd-ufs  (1.7T)

So the blocked command is the following:
root at daedalus:~ # zpool create tank /dev/mirror/gm0s1d

root at daedalus:~ # ps -O comm,mwchan
   PID COMMAND MWCHAN   TT  STAT    TIME COMMAND
  1104 getty   ttyin    v0  Is+  0:00.00 /usr/libexec/getty Pc ttyv0
  1105 getty   ttyin    v1  Is+  0:00.00 /usr/libexec/getty Pc ttyv1
  1106 getty   ttyin    v2  Is+  0:00.00 /usr/libexec/getty Pc ttyv2
  1107 getty   ttyin    v3  Is+  0:00.00 /usr/libexec/getty Pc ttyv3
  1108 getty   ttyin    v4  Is+  0:00.00 /usr/libexec/getty Pc ttyv4
  1109 getty   ttyin    v5  Is+  0:00.00 /usr/libexec/getty Pc ttyv5
  1110 getty   ttyin    v6  Is+  0:00.00 /usr/libexec/getty Pc ttyv6
  1111 getty   ttyin    v7  Is+  0:00.00 /usr/libexec/getty Pc ttyv7
85215 csh     pause     2  Ss   0:00.12 /bin/csh
86210 ps      -         2  R+   0:00.00 ps -O comm,mwchan
85081 csh     pause     0  Is   0:00.01 -csh (csh)
85084 screen  pause     0  S+   0:00.11 screen
85086 csh     pause     1  Is   0:00.12 /bin/csh
85159 zpool   zio->io_  1  D+   0:00.00 zpool create tank /dev/mirror/gm0s1d

root at daedalus:/proc/85159 # cat status
zpool 85159 1 85159 85086 pts/1 ctty 1400242849,897702 0,0 0,1876 
zio->io_cv 0 0 0,0,5 -


There is no activity on disks:
root at daedalus:~ # iostat 1
        tty            ada0             ada1            pass0 
   cpu
  tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy 
in id
    0    41 124.91 457 55.80  125.92 454 55.80   0.00   0  0.00   0  0 
0  0 99
    0   465  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0 
  0 100
    0   155  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0 
  0 100
    0   151  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0 
  0 100

I have also a FreeBSD 9.0-RELEASE-p3 setup the same way and it works 
like a charm. (but not the same hardware on this one)

Actually, I'm trying to zeroing the partition, we never know :-)

Anyone has an idea ?

Thanks,

-- 
Nicolas Haller

On 5/16/2014 8:23 AM, Nicolas Haller wrote:
> Hello,
>
> I've got a new server and I installed FreeBSD 10 on it. I have a 
> problem to create a new zfs pool. The command stalls on IO wait (D 
> state / zio->io_cv).
>
> The device for the pool is a 1.7T partition (index 4) of a gmirror 
> device.
Why?

ZFS provides its own mirroring and it is superior, as it checksums each 
block and thus does not rely on the drive returning an error to detect 
problems.  It can also rewrite a bad block (assuming the problem is 
transient) and scrub also relies on independent components.

You're destroying the data integrity advantage that ZFS gives you by 
using a gmirror under it.  Stop doing that and see if your problem 
disappears.

(In other words it sounds like the problem is real but you shouldn't be 
doing that anyway, so it also shouldn't bite you.)

-- 
-- Karl
karl at denninger.net


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140516/bff104a4/attachment.bin>