Eric Dombroski
2014-Jan-18 18:51 UTC
Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5
Hello: I believe there is a major performance regression between FreeBSD 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and handling incoming traffic. Below are the results of some iperf tests and large dd operations over NFS. Write throughput goes from ~40Gbps to ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable ("no buffer space available"), requiring the interface to be taken down/up. These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks on either system. I can't reproduce this using an Intel 1Gbps ethernet through PCIe passthrough, although I suspect the problem manifests itself over 1Gbps speeds anyway. Tests: Client (host): root at gogo:~# uname -a Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux root at gogo:~# kvm -version QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright (c) 2003-2008 Fabrice Bellard root at gogo:~# lsmod | grep vhost vhost_net 27436 3 tun 18337 8 vhost_net macvtap 17633 1 vhost_net Command: iperf -c 192.168.100.x -t 60 Server (FreeBSD 9.2 VM): root at umarotest:~ # uname -a FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan 11 03:25:02 UTC 2014 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 root at umarotest:~ # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58996 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58997 [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58998 [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58999 [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 59000 [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec While pinging out from the server to the client, I do not get any errors. root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 r260430: Wed Jan 8 05:10:04 UTC 2014 root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root at umaro:~ # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50264 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50265 [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50266 [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50267 [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50268 [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec *** While pinging out from the server to client, frequent "ping: sendto: No space left on device" errors *** After a while, I can also reliably re-produce more egregious "ping: sendto: No buffer space available" errors after doing a large sequential write over NFS: mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:/storage/shared /mnt/nfs dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 I am going to file a freebsd bug report as well. Thanks, Eric
Eric Dombroski
2014-Jan-18 19:04 UTC
Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5
PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=185864 On Sat, Jan 18, 2014 at 1:51 PM, Eric Dombroski <eric at edombroski.com> wrote:> Hello: > > I believe there is a major performance regression between FreeBSD > 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and > handling incoming traffic. Below are the results of some iperf tests and > large dd operations over NFS. Write throughput goes from ~40Gbps to > ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable > ("no buffer space available"), requiring the interface to be taken down/up. > > > These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks > on either system. > > I can't reproduce this using an Intel 1Gbps ethernet through PCIe > passthrough, although I suspect the problem manifests itself over 1Gbps > speeds anyway. > > Tests: > > Client (host): > root at gogo:~# uname -a > Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux > root at gogo:~# kvm -version > QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright > (c) 2003-2008 Fabrice Bellard > root at gogo:~# lsmod | grep vhost > vhost_net 27436 3 > tun 18337 8 vhost_net > macvtap 17633 1 vhost_net > > > Command: iperf -c 192.168.100.x -t 60 > > > Server (FreeBSD 9.2 VM): > > root at umarotest:~ # uname -a > FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan > 11 03:25:02 UTC 2014 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > root at umarotest:~ # iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58996 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec > [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58997 > [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58998 > [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec > [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58999 > [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec > [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 59000 > [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > > While pinging out from the server to the client, I do not get any > errors. > > > root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 > r260430: Wed Jan 8 05:10:04 UTC 2014 root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC > amd64 > root at umaro:~ # iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 > port 50264 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec > [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 > port 50265 > [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec > [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 > port 50266 > [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 > port 50267 > [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 > port 50268 > [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec > > *** While pinging out from the server to client, frequent "ping: > sendto: No space left on device" errors *** > > > After a while, I can also reliably re-produce more egregious "ping: > sendto: No buffer space available" errors after doing a large sequential > write over NFS: > > mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:/storage/shared > /mnt/nfs > dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 > > I am going to file a freebsd bug report as well. > > Thanks, > Eric >
Adrian Chadd
2014-Jan-18 21:06 UTC
Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5
Hi, Have you tried disabling tso? Adrian On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric at edombroski.com> wrote:> Hello: > > I believe there is a major performance regression between FreeBSD > 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and > handling incoming traffic. Below are the results of some iperf tests and > large dd operations over NFS. Write throughput goes from ~40Gbps to > ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable > ("no buffer space available"), requiring the interface to be taken down/up. > > > These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks on > either system. > > I can't reproduce this using an Intel 1Gbps ethernet through PCIe > passthrough, although I suspect the problem manifests itself over 1Gbps > speeds anyway. > > Tests: > > Client (host): > root at gogo:~# uname -a > Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux > root at gogo:~# kvm -version > QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright > (c) 2003-2008 Fabrice Bellard > root at gogo:~# lsmod | grep vhost > vhost_net 27436 3 > tun 18337 8 vhost_net > macvtap 17633 1 vhost_net > > > Command: iperf -c 192.168.100.x -t 60 > > > Server (FreeBSD 9.2 VM): > > root at umarotest:~ # uname -a > FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan > 11 03:25:02 UTC 2014 > root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > root at umarotest:~ # iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58996 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec > [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58997 > [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58998 > [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec > [ 5] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 58999 > [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec > [ 4] local 192.168.100.44 port 5001 connected with 192.168.100.1 > port 59000 > [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec > > While pinging out from the server to the client, I do not get any > errors. > > > root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 > r260430: Wed Jan 8 05:10:04 UTC 2014 > root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC > amd64 > root at umaro:~ # iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port > 50264 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec > [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port > 50265 > [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec > [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port > 50266 > [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > [ 5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port > 50267 > [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec > [ 4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port > 50268 > [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec > > *** While pinging out from the server to client, frequent "ping: > sendto: No space left on device" errors *** > > > After a while, I can also reliably re-produce more egregious "ping: > sendto: No buffer space available" errors after doing a large sequential > write over NFS: > > mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5: > /storage/shared > /mnt/nfs > dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000 > > I am going to file a freebsd bug report as well. > > Thanks, > Eric > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >