thr3ads.net - libvirt users - Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface [Oct 2017]

If this information is useful, please help other people find it:
Share via:

Yalan Zhang

2017-Oct-26 08:01 UTC

Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface

Hi Ashish,

IMO, it is yes, no way to increase tx_queue_size for direct type interface






-------
Best Regards,
Yalan Zhang
IRC: yalzhang
Internal phone: 8389413

On Thu, Oct 26, 2017 at 3:38 PM, Ashish Kurian <ashishbnv@gmail.com>
wrote:
> Hi Yalan,
>
> In the previous email you mentioned "tx_queue_size='512' will
not work in
> the guest with direct type interface, in fact, no matter what you set, it
> will not work and guest will get the default '256'. "
>
> So if I am using macvtap for my interfaces, then the device type will
> always be direct type. Does it mean that there is no way I can increase the
> buffer size with the macvtap interfaces?
>
>
>
> Best Regards,
> Ashish Kurian
>
> On Thu, Oct 26, 2017 at 9:04 AM, Ashish Kurian <ashishbnv@gmail.com>
> wrote:
>
>> Hi Yalan,
>>
>> Thank you for your comment on qemu-kvm-rhev
>>
>> I am waiting for a response about my previous email with the logs
>> attached. I do not understand what is the problem.
>>
>>
>> On Oct 26, 2017 8:58 AM, "Yalan Zhang"
<yalzhang@redhat.com> wrote:
>>
>> Hi Ashish,
>>
>> Please never mind for qemu-kvm-rhev.
>> qemu with the code base 2.10.0 will support the tx_queue_size and
>> rx_queue_size.
>>
>> Thank you~
>>
>>
>>
>>
>>
>> -------
>> Best Regards,
>> Yalan Zhang
>> IRC: yalzhang
>> Internal phone: 8389413
>>
>> On Thu, Oct 26, 2017 at 2:22 PM, Yalan Zhang
<yalzhang@redhat.com> wrote:
>>
>>> Hi Ashish,
>>>
>>> Are these packages available for free? How can I install them?
>>> => You did have vhost backend driver. Do not set <driver
>>> name='qemu'...>, by default it will use vhost as backend
driver.
>>>
>>>  Is it possible to have my interfaces with an IP address inside the
VM
>>> to be bridged to the physical interfaces on the host?
>>> => Yes, you can create a linux bridge with physical interface
connected,
>>> and use bridge type interface. Refer to https://libvirt.org/formatd
>>> omain.html#elementsNICSBridge
>>> direct type is also ok (but your host and guest have no access to
each
>>> other).
>>>
>>> Is it also a possibility that I change the rx and tx buffer on the
>>> physical interface on the host and it is reflected automatically
inside the
>>> VM as you said it will always receive the default value of the
host?
>>> => No, it do not receive the default value of the host. It's
the default
>>> value related with the virtual device driver on the guest.
>>> hostdev type interface will passthrough the physical interface or
VF of
>>> the host to guest, it will get the device's parameters for rx
and tx buffer.
>>>
>>>
>>>
>>> -------
>>> Best Regards,
>>> Yalan Zhang
>>> IRC: yalzhang
>>> Internal phone: 8389413
>>>
>>> On Thu, Oct 26, 2017 at 1:30 PM, Ashish Kurian
<ashishbnv@gmail.com>
>>> wrote:
>>>
>>>> Hi Yalan,
>>>>
>>>> Thank you for your response. I do not have the following
packages
>>>> installed
>>>>
>>>> vhost backend driver
>>>> qemu-kvm-rhev package
>>>>
>>>> Are these packages available for free? How can I install them?
>>>>
>>>> In my KVM VM, I must have an IP address to the interfaces that
I am
>>>> trying to increasing the buffers. That is the reason I was
using macvtap
>>>> (direct type interface). Is it possible to have my interfaces
with an IP
>>>> address inside the VM to be bridged to the physical interfaces
on the host?
>>>>
>>>> Is it also a possibility that I change the rx and tx buffer on
the
>>>> physical interface on the host and it is reflected
automatically inside the
>>>> VM as you said it will always receive the default value of the
host?
>>>>
>>>>
>>>> Best Regards,
>>>> Ashish Kurian
>>>>
>>>> On Thu, Oct 26, 2017 at 6:45 AM, Yalan Zhang
<yalzhang@redhat.com>
>>>> wrote:
>>>>
>>>>> Hi Ashish,
>>>>>
>>>>> I have tested with your xml in the first mail, and it works
for rx_queue_size(see
>>>>> below).
>>>>> multiqueue need to work with vhost backend driver. And when
you set
>>>>> "queues=1" it will ignored.
>>>>>
>>>>> Please check your qemu-kvm-rhev package, should be newer
than
>>>>> qemu-kvm-rhev-2.9.0-16.el7_4.2
>>>>> And the logs?
>>>>>
>>>>> tx_queue_size='512' will not work in the guest with
direct type
>>>>> interface, in fact, no matter what you set, it will not
work and guest will
>>>>> get the default '256'.
>>>>> We only support vhost-user backend to have more than 256.
refer to
>>>>> https://libvirt.org/formatdomain.html#elementsNICSEthernet
>>>>>
>>>>> tx_queue_size
>>>>> The optional tx_queue_size attribute controls the size of
virtio ring
>>>>> for each queue as described above. The default value is
hypervisor
>>>>> dependent and may change across its releases. Moreover,
some hypervisors
>>>>> may pose some restrictions on actual value. For instance,
QEMU v2.9
>>>>> requires value to be a power of two from [256, 1024] range.
In addition to
>>>>> that, this may work only for a subset of interface types,
e.g.
>>>>> aforementioned QEMU enables this option only for vhostuser
type. Since
>>>>> 3.7.0 (QEMU and KVM only)
>>>>> multiqueue only supports vhost as backend driver.
>>>>>
>>>>> # rpm -q libvirt qemu-kvm-rhev
>>>>> libvirt-3.2.0-14.el7_4.3.x86_64
>>>>> qemu-kvm-rhev-2.9.0-16.el7_4.9.x86_64
>>>>>
>>>>> 1. the xml as below
>>>>>    <interface type='direct'>
>>>>>       <mac address='52:54:00:00:b5:99'/>
>>>>>       <source dev='eno1'
mode='vepa'/>
>>>>>       <model type='virtio'/>
>>>>>       <driver name='vhost' queues='5'
rx_queue_size='512'
>>>>> tx_queue_size='512'>
>>>>>         <host csum='off' gso='off'
tso4='off' tso6='off' ecn='off'
>>>>> ufo='off' mrg_rxbuf='off'/>
>>>>>         <guest csum='off' tso4='off'
tso6='off' ecn='off' ufo='off'/>
>>>>>       </driver>
>>>>>       <address type='pci'
domain='0x0000' bus='0x00' slot='0x03'
>>>>> function='0x0'/>
>>>>>     </interface>
>>>>>
>>>>> 2. after start the vm, check the qemu command line:
>>>>> *-netdev
>>>>>
tap,fds=26:28:29:30:31,id=hostnet0,vhost=on,vhostfds=32:33:34:35:36*
>>>>> -device
virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off,
>>>>>
host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest
>>>>> _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,
>>>>>
*mq=on,vectors=12,rx_queue_size=512,tx_queue_size=512*,netdev=hostnet
>>>>> 0,id=net0,mac=52:54:00:00:b5:99,bus=pci.0,addr=0x3
>>>>>
>>>>> 3. check on guest
>>>>> # ethtool -g eth0
>>>>> Ring parameters for eth0:
>>>>> Pre-set maximums:
>>>>> RX: *512 ==> rx_queue_size works*
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: *256   ===> no change*
>>>>> Current hardware settings:
>>>>> RX: *512 **==> rx_queue_size works*
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: *256 ===> no change*
>>>>>
>>>>> # ethtool -l eth0
>>>>> Channel parameters for eth0:
>>>>> Pre-set maximums:
>>>>> RX: 0
>>>>> TX: 0
>>>>> Other: 0
>>>>> Combined: *5  ==> queues what we set*
>>>>> Current hardware settings:
>>>>> RX: 0
>>>>> TX: 0
>>>>> Other: 0
>>>>> Combined: 1
>>>>>
>>>>>
>>>>> If change to qemu as driver,
>>>>> # virsh edit rhel7
>>>>> ..
>>>>>   <interface type='direct'>
>>>>>       <mac address='52:54:00:00:b5:99'/>
>>>>>       <source dev='eno1'
mode='vepa'/>
>>>>>       <model type='virtio'/>
>>>>>       <driver name='qemu' queues='5'
rx_queue_size='512'
>>>>> tx_queue_size='512'>
>>>>>         <host csum='off' gso='off'
tso4='off' tso6='off' ecn='off'
>>>>> ufo='off' mrg_rxbuf='off'/>
>>>>>         <guest csum='off' tso4='off'
tso6='off' ecn='off' ufo='off'/>
>>>>>       </driver>
>>>>>       <address type='pci'
domain='0x0000' bus='0x00' slot='0x03'
>>>>> function='0x0'/>
>>>>>     </interface>
>>>>> ..
>>>>> Domain rhel7 XML configuration edited. ==> the xml can
validate and
>>>>> save
>>>>>
>>>>> # virsh start rhel7
>>>>> Domain rhel7 started
>>>>>
>>>>>
>>>>> # virsh dumpxml rhel7 | grep /interface -B9
>>>>>       <source dev='eno1'
mode='vepa'/>
>>>>>       <target dev='macvtap0'/>
>>>>>       <model type='virtio'/>
>>>>>       *<driver name='qemu' queues='5'
rx_queue_size='512'
>>>>> tx_queue_size='512'>*
>>>>>         <host csum='off' gso='off'
tso4='off' tso6='off' ecn='off'
>>>>> ufo='off' mrg_rxbuf='off'/>
>>>>>         <guest csum='off' tso4='off'
tso6='off' ecn='off' ufo='off'/>
>>>>>       </driver>
>>>>>       <alias name='net0'/>
>>>>>       <address type='pci'
domain='0x0000' bus='0x00' slot='0x03'
>>>>> function='0x0'/>
>>>>>     </interface>
>>>>>
>>>>>
>>>>> * -netdev tap,fds=26:28:29:30:31*,id=hostnet0 -device
>>>>>
virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off,
>>>>>
host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest
>>>>> _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,
>>>>>
*rx_queue_size=512,tx_queue_size=512*,netdev=hostnet0,id=net0,mac=52:
>>>>> 54:00:00:b5:99,bus=pci.0,addr=0x3
>>>>>
>>>>> *"mq=on,vectors=12" is missing*, indicates there
is no multiqueue
>>>>>
>>>>> and check on guest
>>>>>
>>>>> # ethtool -l eth0
>>>>> Channel parameters for eth0:
>>>>> Pre-set maximums:
>>>>> RX: 0
>>>>> TX: 0
>>>>> Other: 0
>>>>> Combined: 1  ==> no multiqueue
>>>>> Current hardware settings:
>>>>> RX: 0
>>>>> TX: 0
>>>>> Other: 0
>>>>> Combined: 1
>>>>>
>>>>> # ethtool -g eth0
>>>>> Ring parameters for eth0:
>>>>> Pre-set maximums:
>>>>> RX: *512*
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 256
>>>>> Current hardware settings:
>>>>> RX: *512*
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 256
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -------
>>>>> Best Regards,
>>>>> Yalan Zhang
>>>>> IRC: yalzhang
>>>>> Internal phone: 8389413
>>>>>
>>>>> On Thu, Oct 26, 2017 at 2:33 AM, Ashish Kurian
<ashishbnv@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Michal,
>>>>>>
>>>>>> An update to what I have already said : when I try
adding <driver
>>>>>> name='qemu' txmode='iothread'
ioeventfd='on' event_idx='off' queues='1'
>>>>>> rx_queue_size='512'
tx_queue_size='512'> although it showed me the error as
>>>>>> mentioned, when I checked the xml again I saw that
<driver
>>>>>> name='qemu' txmode='iothread'
ioeventfd='on' event_idx='off' > is added to
>>>>>> the interface.
>>>>>>
>>>>>> The missing parameters are : queues='1'
rx_queue_size='512'
>>>>>> tx_queue_size='512'
>>>>>>
>>>>>> Best Regards,
>>>>>> Ashish Kurian
>>>>>>
>>>>>> On Wed, Oct 25, 2017 at 5:07 PM, Ashish Kurian
<ashishbnv@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Michal,
>>>>>>>
>>>>>>> What I found was that when I restarted the machine
and did a virsh
>>>>>>> edit command to see the xml config, I see that it
is was not actually
>>>>>>> changed. This suggests why I saw 256 again after
restarting.
>>>>>>>
>>>>>>> So now I tried again to edit the xml via virsh edit
command and used
>>>>>>> the following to set the parameters.
>>>>>>>
>>>>>>> <driver name='qemu'
txmode='iothread' ioeventfd='on' event_idx='off'
>>>>>>> queues='1' rx_queue_size='512'
tx_queue_size='512'>
>>>>>>> </driver>
>>>>>>>
>>>>>>> It was not accepted and I got the error saying :
>>>>>>>
>>>>>>>
>>>>>>> error: XML document failed to validate against
schema: Unable to
>>>>>>> validate doc against
/usr/share/libvirt/schemas/domain.rng
>>>>>>> Extra element devices in interleave
>>>>>>> Element domain failed to validate content
>>>>>>>
>>>>>>> What does this imply? I have two more other
interfaces and do I have
>>>>>>> to the same to them also?
>>>>>>>
>>>>>>> Btw, there are now logs generated now in the domain
log or libvirtd
>>>>>>> log
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Ashish Kurian
>>>>>>>
>>>>>>> On Wed, Oct 25, 2017 at 2:50 PM, Michal Privoznik
<
>>>>>>> mprivozn@redhat.com> wrote:
>>>>>>>
>>>>>>>> On 10/25/2017 01:53 PM, Ashish Kurian wrote:
>>>>>>>> > Dear Users/Developers,
>>>>>>>> >
>>>>>>>> > I am using a KVM Ubuntu VM as a degrader
to apply specific delays
>>>>>>>> to
>>>>>>>> > incoming packets. As the delay for my
packets can be higher than
>>>>>>>> 7.5
>>>>>>>> > seconds, there is not enough buffer on my
interface to buffer all
>>>>>>>> the
>>>>>>>> > packets. Therefore those overflowing
packets are dropped in the
>>>>>>>> machine and
>>>>>>>> > not forwarded.
>>>>>>>> >
>>>>>>>> > When I tried to use the command  ethtool
-G ens8 rx 512 to
>>>>>>>> increase the
>>>>>>>> > buffer size, I get the following error.
>>>>>>>> >
>>>>>>>> > Cannot set device ring parameters:
Operation not permitted
>>>>>>>> >
>>>>>>>> > I have kept the VM xml files as specified
in the link :
>>>>>>>> > https://libvirt.org/formatdomain.html. The
value that I kept in
>>>>>>>> my xml file
>>>>>>>> > is as follows.
>>>>>>>> >
>>>>>>>> > <interface type='direct'>
>>>>>>>> >       <mac
address='52:54:00:72:f9:eb'/>
>>>>>>>> >       <source dev='enp7s0f0'
mode='vepa'/>
>>>>>>>> >       <model type='virtio'/>
>>>>>>>> >       <driver name='vhost'
queues='5' rx_queue_size='512'
>>>>>>>> > tx_queue_size='512'>
>>>>>>>> >       <host csum='off'
gso='off' tso4='off' tso6='off' ecn='off'
>>>>>>>> ufo='off'
>>>>>>>> > mrg_rxbuf='off'/>
>>>>>>>> >       <guest csum='off'
tso4='off' tso6='off' ecn='off'
>>>>>>>> ufo='off'/>
>>>>>>>> >       </driver>
>>>>>>>> >       <address type='pci'
domain='0x0000' bus='0x00' slot='0x08'
>>>>>>>> > function='0x0'/>
>>>>>>>> >     </interface>
>>>>>>>> >     <interface
type='direct'>
>>>>>>>> >       <mac
address='52:54:00:00:b5:99'/>
>>>>>>>> >       <source dev='enp7s0f1'
mode='vepa'/>
>>>>>>>> >       <model type='virtio'/>
>>>>>>>> >       <driver name='vhost'
queues='5' rx_queue_size='512'
>>>>>>>> > tx_queue_size='512'>
>>>>>>>> >       <host csum='off'
gso='off' tso4='off' tso6='off' ecn='off'
>>>>>>>> ufo='off'
>>>>>>>> > mrg_rxbuf='off'/>
>>>>>>>> >       <guest csum='off'
tso4='off' tso6='off' ecn='off'
>>>>>>>> ufo='off'/>
>>>>>>>> >       </driver>
>>>>>>>> >       <address type='pci'
domain='0x0000' bus='0x00' slot='0x09'
>>>>>>>> > function='0x0'/>
>>>>>>>>
>>>>>>>> So what does the qemu command line look like?
You can find it in
>>>>>>>> either
>>>>>>>> libvirtd log or domain log.
>>>>>>>>
>>>>>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>>>>>
>>>>>>>> Michal
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> libvirt-users mailing list
>>>>>> libvirt-users@redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>

Ashish Kurian

2017-Oct-26 08:38 UTC

head link

Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface

Hi Yalan and Michal,

Thank you for your response. So what I understand is that I can change
rx_queue size even if I use direct type interface and qemu driver as long
as the driver is virtio. Am I right? If that is the case why am I getting
the error saying that

error: XML document failed to validate against schema: Unable to validate
doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content

for the following xml config? Btw I changed vepa to bridged mode hoping it
would help me. Also note that I tried the following without any driver name
and it still failed.


<interface type='direct'>
      <mac address='52:54:00:72:f9:eb'/>
      <source dev='enp7s0f0' mode='bridge'/>
      <model type='virtio'/>
      <driver name='qemu' queues='5'
rx_queue_size='512'>
        <host csum='off' gso='off' tso4='off'
tso6='off' ecn='off'
ufo='off' mrg_rxbuf='off'/>
        <guest csum='off' tso4='off' tso6='off'
ecn='off' ufo='off'/>
      </driver>


Best Regards,
Ashish Kurian

On Thu, Oct 26, 2017 at 10:01 AM, Yalan Zhang <yalzhang@redhat.com> wrote:
> Hi Ashish,
>
> IMO, it is yes, no way to increase tx_queue_size for direct type interface
>
>
>
>
>
>
> -------
> Best Regards,
> Yalan Zhang
> IRC: yalzhang
> Internal phone: 8389413
>
> On Thu, Oct 26, 2017 at 3:38 PM, Ashish Kurian <ashishbnv@gmail.com>
> wrote:
>
>> Hi Yalan,
>>
>> In the previous email you mentioned "tx_queue_size='512'
will not work
>> in the guest with direct type interface, in fact, no matter what you
set,
>> it will not work and guest will get the default '256'. "
>>
>> So if I am using macvtap for my interfaces, then the device type will
>> always be direct type. Does it mean that there is no way I can increase
the
>> buffer size with the macvtap interfaces?
>>
>>
>>
>> Best Regards,
>> Ashish Kurian
>>
>> On Thu, Oct 26, 2017 at 9:04 AM, Ashish Kurian
<ashishbnv@gmail.com>
>> wrote:
>>
>>> Hi Yalan,
>>>
>>> Thank you for your comment on qemu-kvm-rhev
>>>
>>> I am waiting for a response about my previous email with the logs
>>> attached. I do not understand what is the problem.
>>>
>>>
>>> On Oct 26, 2017 8:58 AM, "Yalan Zhang"
<yalzhang@redhat.com> wrote:
>>>
>>> Hi Ashish,
>>>
>>> Please never mind for qemu-kvm-rhev.
>>> qemu with the code base 2.10.0 will support the tx_queue_size and
>>> rx_queue_size.
>>>
>>> Thank you~
>>>
>>>
>>>
>>>
>>>
>>> -------
>>> Best Regards,
>>> Yalan Zhang
>>> IRC: yalzhang
>>> Internal phone: 8389413
>>>
>>> On Thu, Oct 26, 2017 at 2:22 PM, Yalan Zhang
<yalzhang@redhat.com>
>>> wrote:
>>>
>>>> Hi Ashish,
>>>>
>>>> Are these packages available for free? How can I install them?
>>>> => You did have vhost backend driver. Do not set <driver
>>>> name='qemu'...>, by default it will use vhost as
backend driver.
>>>>
>>>>  Is it possible to have my interfaces with an IP address inside
the VM
>>>> to be bridged to the physical interfaces on the host?
>>>> => Yes, you can create a linux bridge with physical
interface
>>>> connected, and use bridge type interface. Refer to
>>>> https://libvirt.org/formatdomain.html#elementsNICSBridge
>>>> direct type is also ok (but your host and guest have no access
to each
>>>> other).
>>>>
>>>> Is it also a possibility that I change the rx and tx buffer on
the
>>>> physical interface on the host and it is reflected
automatically inside the
>>>> VM as you said it will always receive the default value of the
host?
>>>> => No, it do not receive the default value of the host.
It's the
>>>> default value related with the virtual device driver on the
guest.
>>>> hostdev type interface will passthrough the physical interface
or VF of
>>>> the host to guest, it will get the device's parameters for
rx and tx buffer.
>>>>
>>>>
>>>>
>>>> -------
>>>> Best Regards,
>>>> Yalan Zhang
>>>> IRC: yalzhang
>>>> Internal phone: 8389413
>>>>
>>>> On Thu, Oct 26, 2017 at 1:30 PM, Ashish Kurian
<ashishbnv@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Yalan,
>>>>>
>>>>> Thank you for your response. I do not have the following
packages
>>>>> installed
>>>>>
>>>>> vhost backend driver
>>>>> qemu-kvm-rhev package
>>>>>
>>>>> Are these packages available for free? How can I install
them?
>>>>>
>>>>> In my KVM VM, I must have an IP address to the interfaces
that I am
>>>>> trying to increasing the buffers. That is the reason I was
using macvtap
>>>>> (direct type interface). Is it possible to have my
interfaces with an IP
>>>>> address inside the VM to be bridged to the physical
interfaces on the host?
>>>>>
>>>>> Is it also a possibility that I change the rx and tx buffer
on the
>>>>> physical interface on the host and it is reflected
automatically inside the
>>>>> VM as you said it will always receive the default value of
the host?
>>>>>
>>>>>
>>>>> Best Regards,
>>>>> Ashish Kurian
>>>>>
>>>>> On Thu, Oct 26, 2017 at 6:45 AM, Yalan Zhang
<yalzhang@redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Ashish,
>>>>>>
>>>>>> I have tested with your xml in the first mail, and it
works for rx_queue_size(see
>>>>>> below).
>>>>>> multiqueue need to work with vhost backend driver. And
when you set
>>>>>> "queues=1" it will ignored.
>>>>>>
>>>>>> Please check your qemu-kvm-rhev package, should be
newer than
>>>>>> qemu-kvm-rhev-2.9.0-16.el7_4.2
>>>>>> And the logs?
>>>>>>
>>>>>> tx_queue_size='512' will not work in the guest
with direct type
>>>>>> interface, in fact, no matter what you set, it will not
work and guest will
>>>>>> get the default '256'.
>>>>>> We only support vhost-user backend to have more than
256. refer to
>>>>>>
https://libvirt.org/formatdomain.html#elementsNICSEthernet
>>>>>>
>>>>>> tx_queue_size
>>>>>> The optional tx_queue_size attribute controls the size
of virtio
>>>>>> ring for each queue as described above. The default
value is hypervisor
>>>>>> dependent and may change across its releases. Moreover,
some hypervisors
>>>>>> may pose some restrictions on actual value. For
instance, QEMU v2.9
>>>>>> requires value to be a power of two from [256, 1024]
range. In addition to
>>>>>> that, this may work only for a subset of interface
types, e.g.
>>>>>> aforementioned QEMU enables this option only for
vhostuser type. Since
>>>>>> 3.7.0 (QEMU and KVM only)
>>>>>> multiqueue only supports vhost as backend driver.
>>>>>>
>>>>>> # rpm -q libvirt qemu-kvm-rhev
>>>>>> libvirt-3.2.0-14.el7_4.3.x86_64
>>>>>> qemu-kvm-rhev-2.9.0-16.el7_4.9.x86_64
>>>>>>
>>>>>> 1. the xml as below
>>>>>>    <interface type='direct'>
>>>>>>       <mac address='52:54:00:00:b5:99'/>
>>>>>>       <source dev='eno1'
mode='vepa'/>
>>>>>>       <model type='virtio'/>
>>>>>>       <driver name='vhost'
queues='5' rx_queue_size='512'
>>>>>> tx_queue_size='512'>
>>>>>>         <host csum='off' gso='off'
tso4='off' tso6='off' ecn='off'
>>>>>> ufo='off' mrg_rxbuf='off'/>
>>>>>>         <guest csum='off' tso4='off'
tso6='off' ecn='off' ufo='off'/>
>>>>>>       </driver>
>>>>>>       <address type='pci'
domain='0x0000' bus='0x00' slot='0x03'
>>>>>> function='0x0'/>
>>>>>>     </interface>
>>>>>>
>>>>>> 2. after start the vm, check the qemu command line:
>>>>>> *-netdev
>>>>>>
tap,fds=26:28:29:30:31,id=hostnet0,vhost=on,vhostfds=32:33:34:35:36*
>>>>>> -device
virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off,
>>>>>>
host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest
>>>>>> _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,
>>>>>>
*mq=on,vectors=12,rx_queue_size=512,tx_queue_size=512*,netdev=hostnet
>>>>>> 0,id=net0,mac=52:54:00:00:b5:99,bus=pci.0,addr=0x3
>>>>>>
>>>>>> 3. check on guest
>>>>>> # ethtool -g eth0
>>>>>> Ring parameters for eth0:
>>>>>> Pre-set maximums:
>>>>>> RX: *512 ==> rx_queue_size works*
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: *256   ===> no change*
>>>>>> Current hardware settings:
>>>>>> RX: *512 **==> rx_queue_size works*
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: *256 ===> no change*
>>>>>>
>>>>>> # ethtool -l eth0
>>>>>> Channel parameters for eth0:
>>>>>> Pre-set maximums:
>>>>>> RX: 0
>>>>>> TX: 0
>>>>>> Other: 0
>>>>>> Combined: *5  ==> queues what we set*
>>>>>> Current hardware settings:
>>>>>> RX: 0
>>>>>> TX: 0
>>>>>> Other: 0
>>>>>> Combined: 1
>>>>>>
>>>>>>
>>>>>> If change to qemu as driver,
>>>>>> # virsh edit rhel7
>>>>>> ..
>>>>>>   <interface type='direct'>
>>>>>>       <mac address='52:54:00:00:b5:99'/>
>>>>>>       <source dev='eno1'
mode='vepa'/>
>>>>>>       <model type='virtio'/>
>>>>>>       <driver name='qemu' queues='5'
rx_queue_size='512'
>>>>>> tx_queue_size='512'>
>>>>>>         <host csum='off' gso='off'
tso4='off' tso6='off' ecn='off'
>>>>>> ufo='off' mrg_rxbuf='off'/>
>>>>>>         <guest csum='off' tso4='off'
tso6='off' ecn='off' ufo='off'/>
>>>>>>       </driver>
>>>>>>       <address type='pci'
domain='0x0000' bus='0x00' slot='0x03'
>>>>>> function='0x0'/>
>>>>>>     </interface>
>>>>>> ..
>>>>>> Domain rhel7 XML configuration edited. ==> the xml
can validate and
>>>>>> save
>>>>>>
>>>>>> # virsh start rhel7
>>>>>> Domain rhel7 started
>>>>>>
>>>>>>
>>>>>> # virsh dumpxml rhel7 | grep /interface -B9
>>>>>>       <source dev='eno1'
mode='vepa'/>
>>>>>>       <target dev='macvtap0'/>
>>>>>>       <model type='virtio'/>
>>>>>>       *<driver name='qemu'
queues='5' rx_queue_size='512'
>>>>>> tx_queue_size='512'>*
>>>>>>         <host csum='off' gso='off'
tso4='off' tso6='off' ecn='off'
>>>>>> ufo='off' mrg_rxbuf='off'/>
>>>>>>         <guest csum='off' tso4='off'
tso6='off' ecn='off' ufo='off'/>
>>>>>>       </driver>
>>>>>>       <alias name='net0'/>
>>>>>>       <address type='pci'
domain='0x0000' bus='0x00' slot='0x03'
>>>>>> function='0x0'/>
>>>>>>     </interface>
>>>>>>
>>>>>>
>>>>>> * -netdev tap,fds=26:28:29:30:31*,id=hostnet0 -device
>>>>>>
virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off,
>>>>>>
host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest
>>>>>> _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off,
>>>>>>
*rx_queue_size=512,tx_queue_size=512*,netdev=hostnet0,id=net0,mac=52:
>>>>>> 54:00:00:b5:99,bus=pci.0,addr=0x3
>>>>>>
>>>>>> *"mq=on,vectors=12" is missing*, indicates
there is no multiqueue
>>>>>>
>>>>>> and check on guest
>>>>>>
>>>>>> # ethtool -l eth0
>>>>>> Channel parameters for eth0:
>>>>>> Pre-set maximums:
>>>>>> RX: 0
>>>>>> TX: 0
>>>>>> Other: 0
>>>>>> Combined: 1  ==> no multiqueue
>>>>>> Current hardware settings:
>>>>>> RX: 0
>>>>>> TX: 0
>>>>>> Other: 0
>>>>>> Combined: 1
>>>>>>
>>>>>> # ethtool -g eth0
>>>>>> Ring parameters for eth0:
>>>>>> Pre-set maximums:
>>>>>> RX: *512*
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: 256
>>>>>> Current hardware settings:
>>>>>> RX: *512*
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: 256
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -------
>>>>>> Best Regards,
>>>>>> Yalan Zhang
>>>>>> IRC: yalzhang
>>>>>> Internal phone: 8389413
>>>>>>
>>>>>> On Thu, Oct 26, 2017 at 2:33 AM, Ashish Kurian
<ashishbnv@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Michal,
>>>>>>>
>>>>>>> An update to what I have already said : when I try
adding <driver
>>>>>>> name='qemu' txmode='iothread'
ioeventfd='on' event_idx='off' queues='1'
>>>>>>> rx_queue_size='512'
tx_queue_size='512'> although it showed me the error as
>>>>>>> mentioned, when I checked the xml again I saw that
<driver
>>>>>>> name='qemu' txmode='iothread'
ioeventfd='on' event_idx='off' > is added to
>>>>>>> the interface.
>>>>>>>
>>>>>>> The missing parameters are : queues='1'
rx_queue_size='512'
>>>>>>> tx_queue_size='512'
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Ashish Kurian
>>>>>>>
>>>>>>> On Wed, Oct 25, 2017 at 5:07 PM, Ashish Kurian
<ashishbnv@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Michal,
>>>>>>>>
>>>>>>>> What I found was that when I restarted the
machine and did a virsh
>>>>>>>> edit command to see the xml config, I see that
it is was not actually
>>>>>>>> changed. This suggests why I saw 256 again
after restarting.
>>>>>>>>
>>>>>>>> So now I tried again to edit the xml via virsh
edit command and
>>>>>>>> used the following to set the parameters.
>>>>>>>>
>>>>>>>> <driver name='qemu'
txmode='iothread' ioeventfd='on'
>>>>>>>> event_idx='off' queues='1'
rx_queue_size='512' tx_queue_size='512'>
>>>>>>>> </driver>
>>>>>>>>
>>>>>>>> It was not accepted and I got the error saying
:
>>>>>>>>
>>>>>>>>
>>>>>>>> error: XML document failed to validate against
schema: Unable to
>>>>>>>> validate doc against
/usr/share/libvirt/schemas/domain.rng
>>>>>>>> Extra element devices in interleave
>>>>>>>> Element domain failed to validate content
>>>>>>>>
>>>>>>>> What does this imply? I have two more other
interfaces and do I
>>>>>>>> have to the same to them also?
>>>>>>>>
>>>>>>>> Btw, there are now logs generated now in the
domain log or libvirtd
>>>>>>>> log
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Ashish Kurian
>>>>>>>>
>>>>>>>> On Wed, Oct 25, 2017 at 2:50 PM, Michal
Privoznik <
>>>>>>>> mprivozn@redhat.com> wrote:
>>>>>>>>
>>>>>>>>> On 10/25/2017 01:53 PM, Ashish Kurian
wrote:
>>>>>>>>> > Dear Users/Developers,
>>>>>>>>> >
>>>>>>>>> > I am using a KVM Ubuntu VM as a
degrader to apply specific
>>>>>>>>> delays to
>>>>>>>>> > incoming packets. As the delay for my
packets can be higher than
>>>>>>>>> 7.5
>>>>>>>>> > seconds, there is not enough buffer on
my interface to buffer
>>>>>>>>> all the
>>>>>>>>> > packets. Therefore those overflowing
packets are dropped in the
>>>>>>>>> machine and
>>>>>>>>> > not forwarded.
>>>>>>>>> >
>>>>>>>>> > When I tried to use the command 
ethtool -G ens8 rx 512 to
>>>>>>>>> increase the
>>>>>>>>> > buffer size, I get the following
error.
>>>>>>>>> >
>>>>>>>>> > Cannot set device ring parameters:
Operation not permitted
>>>>>>>>> >
>>>>>>>>> > I have kept the VM xml files as
specified in the link :
>>>>>>>>> > https://libvirt.org/formatdomain.html.
The value that I kept in
>>>>>>>>> my xml file
>>>>>>>>> > is as follows.
>>>>>>>>> >
>>>>>>>>> > <interface
type='direct'>
>>>>>>>>> >       <mac
address='52:54:00:72:f9:eb'/>
>>>>>>>>> >       <source
dev='enp7s0f0' mode='vepa'/>
>>>>>>>>> >       <model
type='virtio'/>
>>>>>>>>> >       <driver name='vhost'
queues='5' rx_queue_size='512'
>>>>>>>>> > tx_queue_size='512'>
>>>>>>>>> >       <host csum='off'
gso='off' tso4='off' tso6='off' ecn='off'
>>>>>>>>> ufo='off'
>>>>>>>>> > mrg_rxbuf='off'/>
>>>>>>>>> >       <guest csum='off'
tso4='off' tso6='off' ecn='off'
>>>>>>>>> ufo='off'/>
>>>>>>>>> >       </driver>
>>>>>>>>> >       <address type='pci'
domain='0x0000' bus='0x00' slot='0x08'
>>>>>>>>> > function='0x0'/>
>>>>>>>>> >     </interface>
>>>>>>>>> >     <interface
type='direct'>
>>>>>>>>> >       <mac
address='52:54:00:00:b5:99'/>
>>>>>>>>> >       <source
dev='enp7s0f1' mode='vepa'/>
>>>>>>>>> >       <model
type='virtio'/>
>>>>>>>>> >       <driver name='vhost'
queues='5' rx_queue_size='512'
>>>>>>>>> > tx_queue_size='512'>
>>>>>>>>> >       <host csum='off'
gso='off' tso4='off' tso6='off' ecn='off'
>>>>>>>>> ufo='off'
>>>>>>>>> > mrg_rxbuf='off'/>
>>>>>>>>> >       <guest csum='off'
tso4='off' tso6='off' ecn='off'
>>>>>>>>> ufo='off'/>
>>>>>>>>> >       </driver>
>>>>>>>>> >       <address type='pci'
domain='0x0000' bus='0x00' slot='0x09'
>>>>>>>>> > function='0x0'/>
>>>>>>>>>
>>>>>>>>> So what does the qemu command line look
like? You can find it in
>>>>>>>>> either
>>>>>>>>> libvirtd log or domain log.
>>>>>>>>>
>>>>>>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>>>>>>
>>>>>>>>> Michal
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> libvirt-users mailing list
>>>>>>> libvirt-users@redhat.com
>>>>>>>
https://www.redhat.com/mailman/listinfo/libvirt-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>

Michal Privoznik

2017-Oct-26 08:46 UTC

head link

Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface

On 10/26/2017 10:38 AM, Ashish Kurian wrote:> Hi Yalan and Michal,
> 
> Thank you for your response. So what I understand is that I can change
> rx_queue size even if I use direct type interface and qemu driver as long
> as the driver is virtio. Am I right? 
Yes.
> If that is the case why am I getting
> the error saying that
> 
> error: XML document failed to validate against schema: Unable to validate
> doc against /usr/share/libvirt/schemas/domain.rng
> Extra element devices in interleave
> Element domain failed to validate content
What's your libvirt version? For rx_queue_size you need 2.3.0 or newer.
For tx_queue_size you need 3.7.0 or newer (not that there's one yet).
> 
> for the following xml config? Btw I changed vepa to bridged mode hoping it
> would help me. Also note that I tried the following without any driver name
> and it still failed.
> 
> 
> <interface type='direct'>
>       <mac address='52:54:00:72:f9:eb'/>
>       <source dev='enp7s0f0' mode='bridge'/>
>       <model type='virtio'/>
>       <driver name='qemu' queues='5'
rx_queue_size='512'>
>         <host csum='off' gso='off' tso4='off'
tso6='off' ecn='off'
> ufo='off' mrg_rxbuf='off'/>
>         <guest csum='off' tso4='off' tso6='off'
ecn='off' ufo='off'/>
>       </driver>
> 
This works for me. I'm running recent git.

Michal

Seemingly Similar Threads

Search for more seemingly similar threads

libvirt users - Oct 2017 - Re: Need to increase the rx and tx buffer size of my interface

Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface

Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface

Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface

Seemingly Similar Threads