freebsd stable - Mar 2016 - Virtio network: poor network with KVM hypervisor (latest Proxmox)

Hi all! 

I am using the latest Proxmox 4.1 with all updates installed. 
I have several VM's with FreeBSD guests and 1 VM with Ubuntu 14 (all KVM). 
Host system file download speed: 60 MBps. 
FreeBSD guest download speed: 2 MBps on virtio network with TSO enabled, 5-9
MBps with TSO disabled; 12 MBps on e1000 network.
Ubuntu guest: 60 MBps with virtio. 

I've tried the following: 
1) Different FreeBSD versions: 9.3, 10.2, 10.3-BETA3. 
2) Different TSO settings, enabling/disabling RXCSUM. 
3) Different TSO settings on host system. 

The best results I got described above :( 

Does anyone have any ideas how to get full network performance inside FreeBSD
guests?

--
Alexey Tarasov

(\__/) 
(='.'=) 
E[: | | | | :]? 
(")_(")

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3764 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160301/74926148/attachment.bin>

Hi Alexey,

Can?t bring in much light, but here are our settings under QEMU-KVM 2.1.0
(qemu-system-x86_64):

kern.ipc.maxsockbuf=4194304
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvspace=262144
net.inet.tcp.sendspace=262144
net.inet.tcp.mssdflt=1460
net.inet.tcp.minmss=1300

net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288

We dind?t change any of the eth. settings:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex> (this is autodetected,
we?re not enforcing it in any way)
        status: active

We didn?t have as huge of a problem as you do, but we usually didn?t get over
10-15 Mbps both inbound and outbound for a single TCP connection. Now we easily
see 50Mbps ratios, which is way more reasonable, although not impressive.

IIRC the biggest change came after changing:

net.inet.tcp.mssdflt=1460
net.inet.tcp.minmss=1300

net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288

We?re on 9.3-RELEASE-p5.

I can?t seem to enable/disable TSO or LRO via ifconfig, neither I can find any
sysctl tunables related to this (man vtnet reads that there should be some):

# sysctl hw. | grep vtnet
# (no output)

Hope it helps. Please let us know if you find out anything else, as we?re also
interested in getting some more throughput out of those interfaces!

Best,
David
> El 1 mar? 2016, a les 12:30, Alexey Tarasov <me at lexasoft.ru> va
escriure:
> 
> Hi all! 
> 
> I am using the latest Proxmox 4.1 with all updates installed. 
> I have several VM's with FreeBSD guests and 1 VM with Ubuntu 14 (all
KVM).
> Host system file download speed: 60 MBps. 
> FreeBSD guest download speed: 2 MBps on virtio network with TSO enabled,
5-9 MBps with TSO disabled; 12 MBps on e1000 network.
> Ubuntu guest: 60 MBps with virtio. 
> 
> I've tried the following: 
> 1) Different FreeBSD versions: 9.3, 10.2, 10.3-BETA3. 
> 2) Different TSO settings, enabling/disabling RXCSUM. 
> 3) Different TSO settings on host system. 
> 
> The best results I got described above :( 
> 
> Does anyone have any ideas how to get full network performance inside
FreeBSD guests?
> 
> --
> Alexey Tarasov
> 
> (\__/) 
> (='.'=) 
> E[: | | | | :]? 
> (")_(")
>