There are clearly issues with the controller driver on FreeBSD. Not only
zfs related:
I've did some testing under Centos 7 with ext4
Command line used: iozone -Rb /root/output_iops.wks -O -i 0 -i
1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -r 512K -s 8G
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
8388608 4 156982 0 859809 0 656572 208156
8388608 8 82545 0 503549 0 435005 111282
8388608 32 21620 0 129740 0 135738 31361
8388608 64 11070 0 66027 0 71660 16175
8388608 128 5558 0 31800 0 35361 8066
8388608 512 1383 0 7520 0 8472 2068
Same test on BSD with ufs
Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i
2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
1048576 4 120642 0 368188 0 252302 41650
1048576 8 61871 0 253074 0 189851 35015
1048576 32 15925 0 89190 0 80004 19357
1048576 64 7958 0 47190 0 59120 10070
1048576 128 4024 0 23619 0 22435 5542
ZFS test on BSD:
Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i
2 -e -+n -r4K -s 1G
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
1048576 4 69600 0 99750 0 92007 49556
Perftest while write is close to what it should be, read takes a 4x penalty
nvmecontrol perftest -n 32 -o write -s 4096 -t30 nvme0ns1
Threads: 32 Size: 4096 WRITE Time: 30 IO/s: 211294 MB/s: 825
nvmecontrol perftest -n 32 -o read -s 4096 -t30 nvme0ns1
Threads: 32 Size: 4096 READ Time: 30 IO/s: 221365 MB/s: 864
What i've tried for zfs:
vfs.zfs.vdev.sync_read_min_active: 32
vfs.zfs.vdev.sync_read_max_active: 32
vfs.zfs.vdev.sync_write_min_active: 32
vfs.zfs.vdev.sync_write_max_active: 32
vfs.zfs.vdev.async_read_min_active: 32
vfs.zfs.vdev.async_read_max_active: 32
vfs.zfs.vdev.async_write_min_active: 32
vfs.zfs.vdev.async_write_max_active: 32
No effect what so ever in tests.
increasing vfs.zfs.top_maxinflight - > no effect
While i've managed to make it a little bit more stable so that
controller doesn't just disappear by disabling ASPM in bios i still get :
nvme0: resetting controller
and after this i have to reboot.
Some other errors i had today which where fixed with ssd firmware update
and by forcing PCIeX to 16x :
1511.530545] Buffer I/O error on device nvme0n1, logical block 0
[ 1511.530594] lost page write due to I/O error on nvme0n1
[ 1517.694014] Buffer I/O error on device nvme1n1, logical block 0
[ 1517.694064] lost page write due to I/O error on nvme1n1
Best regards,
Vintila Mihai Alexandru
On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote:> On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote:
>
>> /etc/sysctl.conf
>>
>> net.inet.tcp.fast_finwait2_recycle=1
>> net.inet.ip.portrange.randomized=0
>> net.inet.ip.portrange.first=1024
>> net.inet.ip.portrange.last=65535
>> net.inet.tcp.recvbuf_max=16777216
>> net.inet.tcp.sendbuf_max=16777216
>> net.inet.tcp.recvbuf_inc=65536
>> vfs.zfs.vdev.trim_on_init=0
>> #close time_wait connections at 2*7500ms
>> net.inet.tcp.msl=7500
>> kern.ipc.somaxconn=4096
>> net.inet.icmp.icmplim=2000
>>
>> #zfs
>> vfs.zfs.txg.timeout=5
>> vfs.zfs.prefetch_disable=1
>
>> Any help to bring this device to proper speed will be welcomed.
> Do you try to increase vfs.zfs.top_maxinflight?