On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote:> Hi Michael, > > I am playing with vhost multiqueue capability and have a question about > vhost multiqueue and RSS (receive side steering). My setup has Mellanox > ConnectX-3 NIC which supports multiqueue and RSS. Network related > parameters for qemu are: > > -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4 > -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10 > > In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue. > > I am running one tcp stream into the guest using iperf. Since there is > only one tcp stream I expect it to be handled by one queue only but > this seams to be not the case. ethtool -S on a host shows that the > stream is handled by one queue in the NIC, just like I would expect, > but in a guest all 4 virtio-input interrupt are incremented. Am I > missing any configuration?I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors? To see, can you try dumping struct kvm_irqfd that's passed to kvm?> -- > Gleb.
I have a question related this topic. So How do you set the RSS Key on the Mellanox NIc? I mean from your Guest? If it being set as part of Host driver, is there a way to set it from Guest? I mean my guest will choose a RSS Key and will try to set on the Physical NIC. Thanks, Venkatesh -----Original Message----- From: kvm-owner at vger.kernel.org [mailto:kvm-owner at vger.kernel.org] On Behalf Of Michael S. Tsirkin Sent: Monday, November 17, 2014 12:26 AM To: Gleb Natapov Cc: kvm at vger.kernel.org; Jason Wang; virtualization at lists.linux-foundation.org Subject: Re: vhost + multiqueue + RSS question. On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote:> Hi Michael, > > I am playing with vhost multiqueue capability and have a question > about vhost multiqueue and RSS (receive side steering). My setup has > Mellanox > ConnectX-3 NIC which supports multiqueue and RSS. Network related > parameters for qemu are: > > -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4 > -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10 > > In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue. > > I am running one tcp stream into the guest using iperf. Since there is > only one tcp stream I expect it to be handled by one queue only but > this seams to be not the case. ethtool -S on a host shows that the > stream is handled by one queue in the NIC, just like I would expect, > but in a guest all 4 virtio-input interrupt are incremented. Am I > missing any configuration?I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors? To see, can you try dumping struct kvm_irqfd that's passed to kvm?> -- > Gleb.-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 11/17/2014 02:56 AM, Michael S. Tsirkin wrote:> On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote: >> Hi Michael, >> >> I am playing with vhost multiqueue capability and have a question about >> vhost multiqueue and RSS (receive side steering). My setup has Mellanox >> ConnectX-3 NIC which supports multiqueue and RSS. Network related >> parameters for qemu are: >> >> -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4 >> -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10 >> >> In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue. >> >> I am running one tcp stream into the guest using iperf. Since there is >> only one tcp stream I expect it to be handled by one queue only but >> this seams to be not the case. ethtool -S on a host shows that the >> stream is handled by one queue in the NIC, just like I would expect, >> but in a guest all 4 virtio-input interrupt are incremented. Am I >> missing any configuration? > I don't see anything obviously wrong with what you describe. > Maybe, somehow, same irqfd got bound to multiple MSI vectors? > To see, can you try dumping struct kvm_irqfd that's passed to kvm? > > >> -- >> Gleb.This sounds like a regression, which kernel/qemu version did you use?
On 11/17/2014 12:54 PM, Venkateswara Rao Nandigam wrote:> I have a question related this topic. So How do you set the RSS Key on the Mellanox NIc? I mean from your Guest?I believe it's possible but not implemented currently. The issue is the implementation should not be vendor specific. TUN/TAP has its own automatic flow steering implementation (flow caches).> > If it being set as part of Host driver, is there a way to set it from Guest? I mean my guest will choose a RSS Key and will try to set on the Physical NIC.Flow caches can co-operate with RFS/aRFS now, so there's indeed some kind of co-operation between host card and guest I believe.> > Thanks, > Venkatesh >
On Mon, Nov 17, 2014 at 01:30:06PM +0800, Jason Wang wrote:> On 11/17/2014 02:56 AM, Michael S. Tsirkin wrote: > > On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote: > >> Hi Michael, > >> > >> I am playing with vhost multiqueue capability and have a question about > >> vhost multiqueue and RSS (receive side steering). My setup has Mellanox > >> ConnectX-3 NIC which supports multiqueue and RSS. Network related > >> parameters for qemu are: > >> > >> -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4 > >> -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10 > >> > >> In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue. > >> > >> I am running one tcp stream into the guest using iperf. Since there is > >> only one tcp stream I expect it to be handled by one queue only but > >> this seams to be not the case. ethtool -S on a host shows that the > >> stream is handled by one queue in the NIC, just like I would expect, > >> but in a guest all 4 virtio-input interrupt are incremented. Am I > >> missing any configuration? > > I don't see anything obviously wrong with what you describe. > > Maybe, somehow, same irqfd got bound to multiple MSI vectors? > > To see, can you try dumping struct kvm_irqfd that's passed to kvm? > > > > > >> -- > >> Gleb. > > This sounds like a regression, which kernel/qemu version did you use?Sorry, should have mentioned it from the start. Host is a fedora 20 with kernel 3.16.6-200.fc20.x86_64 and qemu-system-x86-1.6.2-9.fc20.x86_64. Guest is also fedora 20 but with an older kernel 3.11.10-301. -- Gleb.
On Sun, Nov 16, 2014 at 08:56:04PM +0200, Michael S. Tsirkin wrote:> On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote: > > Hi Michael, > > > > I am playing with vhost multiqueue capability and have a question about > > vhost multiqueue and RSS (receive side steering). My setup has Mellanox > > ConnectX-3 NIC which supports multiqueue and RSS. Network related > > parameters for qemu are: > > > > -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4 > > -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10 > > > > In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue. > > > > I am running one tcp stream into the guest using iperf. Since there is > > only one tcp stream I expect it to be handled by one queue only but > > this seams to be not the case. ethtool -S on a host shows that the > > stream is handled by one queue in the NIC, just like I would expect, > > but in a guest all 4 virtio-input interrupt are incremented. Am I > > missing any configuration? > > I don't see anything obviously wrong with what you describe. > Maybe, somehow, same irqfd got bound to multiple MSI vectors?It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes. -- Gleb.
On Mon, Nov 17, 2014 at 09:44:23AM +0200, Gleb Natapov wrote:> On Sun, Nov 16, 2014 at 08:56:04PM +0200, Michael S. Tsirkin wrote: > > On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote: > > > Hi Michael, > > > > > > I am playing with vhost multiqueue capability and have a question about > > > vhost multiqueue and RSS (receive side steering). My setup has Mellanox > > > ConnectX-3 NIC which supports multiqueue and RSS. Network related > > > parameters for qemu are: > > > > > > -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4 > > > -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10 > > > > > > In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue. > > > > > > I am running one tcp stream into the guest using iperf. Since there is > > > only one tcp stream I expect it to be handled by one queue only but > > > this seams to be not the case. ethtool -S on a host shows that the > > > stream is handled by one queue in the NIC, just like I would expect, > > > but in a guest all 4 virtio-input interrupt are incremented. Am I > > > missing any configuration? > > > > I don't see anything obviously wrong with what you describe. > > Maybe, somehow, same irqfd got bound to multiple MSI vectors? > It does not look like this is what is happening judging by the way > interrupts are distributed between queues. They are not distributed > uniformly and often I see one queue gets most interrupt and others get > much less and then it changes.Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?> -- > Gleb.