Linux Virtualization - Dec 2018 - [PATCH v2 0/5] VSOCK: support mergeable rx buffer in vhost-vsock

Now vsock only support send/receive small packet, it can't achieve
high performance. As previous discussed with Jason Wang, I revisit the
idea of vhost-net about mergeable rx buffer and implement the mergeable
rx buffer in vhost-vsock, it can allow big packet to be scattered in
into different buffers and improve performance obviously.

This series of patches mainly did three things?
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big
packet(64K) included guest->Host and Host->Guest. I test performance
independently and the result as follows:

Before performance:
              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:
              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

In this case, max send pkt size is still limited to 4K, so Host->Guest
performance will worse than before.

After performance increase the max send pkt size to 64K:
              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:
              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s
>From the test results, the performance is improved obviously, and guest
memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock,
we need to add a qemu patch to support parse feature.

---
v1 -> v2:
 * Addressed comments from Jason Wang.
 * Add performance test result independently.
 * Use Skb_page_frag_refill() which can use high order page and reduce
   the stress of page allocator.
 * Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small
   size can't fill one full packet, we only 128 vq num now.
 * Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx
   consistency.
 * Add virtio_transport ops to get max pkt len, in order to be compatible
   with old version.
---

Yiwen Jiang (5):
  VSOCK: support fill mergeable rx buffer in guest
  VSOCK: support fill data to mergeable rx buffer in host
  VSOCK: support receive mergeable rx buffer in guest
  VSOCK: increase send pkt len in mergeable mode to improve performance
  VSOCK: batch sending rx buffer to increase bandwidth

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

-- 
1.8.3.1

On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen
wrote:
> Now vsock only support send/receive small packet, it can't achieve
> high performance. As previous discussed with Jason Wang, I revisit the
> idea of vhost-net about mergeable rx buffer and implement the mergeable
> rx buffer in vhost-vsock, it can allow big packet to be scattered in
> into different buffers and improve performance obviously.
> 
> This series of patches mainly did three things?
> - mergeable buffer implementation
> - increase the max send pkt size
> - add used and signal guest in a batch
> 
> And I write a tool to test the vhost-vsock performance, mainly send big
> packet(64K) included guest->Host and Host->Guest. I test performance
> independently and the result as follows:
> 
> Before performance:
>               Single socket            Multiple sockets(Max Bandwidth)
> Guest->Host   ~400MB/s                 ~480MB/s
> Host->Guest   ~1450MB/s                ~1600MB/s
> 
> After performance only use implement mergeable rx buffer:
>               Single socket            Multiple sockets(Max Bandwidth)
> Guest->Host   ~400MB/s                 ~480MB/s
> Host->Guest   ~1280MB/s                ~1350MB/s
> 
> In this case, max send pkt size is still limited to 4K, so Host->Guest
> performance will worse than before.
It's concerning though, what if application sends small packets?
What is the source of the slowdown? Do you know?
> After performance increase the max send pkt size to 64K:
>               Single socket            Multiple sockets(Max Bandwidth)
> Guest->Host   ~1700MB/s                ~2900MB/s
> Host->Guest   ~1500MB/s                ~2440MB/s
> 
> After performance all patches are used:
>               Single socket            Multiple sockets(Max Bandwidth)
> Guest->Host   ~1700MB/s                ~2900MB/s
> Host->Guest   ~1700MB/s                ~2900MB/s
> 
> >From the test results, the performance is improved obviously, and guest
> memory will not be wasted.
> 
> In addition, in order to support mergeable rx buffer in virtio-vsock,
> we need to add a qemu patch to support parse feature.
> 
> ---
> v1 -> v2:
>  * Addressed comments from Jason Wang.
>  * Add performance test result independently.
>  * Use Skb_page_frag_refill() which can use high order page and reduce
>    the stress of page allocator.
>  * Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small
>    size can't fill one full packet, we only 128 vq num now.
>  * Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx
>    consistency.
>  * Add virtio_transport ops to get max pkt len, in order to be compatible
>    with old version.
> ---
> 
> Yiwen Jiang (5):
>   VSOCK: support fill mergeable rx buffer in guest
>   VSOCK: support fill data to mergeable rx buffer in host
>   VSOCK: support receive mergeable rx buffer in guest
>   VSOCK: increase send pkt len in mergeable mode to improve performance
>   VSOCK: batch sending rx buffer to increase bandwidth
> 
>  drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
>  include/linux/virtio_vsock.h            |  13 +-
>  include/uapi/linux/virtio_vsock.h       |   5 +
>  net/vmw_vsock/virtio_transport.c        | 229
+++++++++++++++++++++++++++-----
>  net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
>  5 files changed, 411 insertions(+), 85 deletions(-)
> 
> -- 
> 1.8.3.1

On 2018/12/12 23:09, Michael S. Tsirkin wrote:
> On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen wrote:
>> Now vsock only support send/receive small packet, it can't achieve
>> high performance. As previous discussed with Jason Wang, I revisit the
>> idea of vhost-net about mergeable rx buffer and implement the mergeable
>> rx buffer in vhost-vsock, it can allow big packet to be scattered in
>> into different buffers and improve performance obviously.
>>
>> This series of patches mainly did three things?
>> - mergeable buffer implementation
>> - increase the max send pkt size
>> - add used and signal guest in a batch
>>
>> And I write a tool to test the vhost-vsock performance, mainly send big
>> packet(64K) included guest->Host and Host->Guest. I test
performance
>> independently and the result as follows:
>>
>> Before performance:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~400MB/s                 ~480MB/s
>> Host->Guest   ~1450MB/s                ~1600MB/s
>>
>> After performance only use implement mergeable rx buffer:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~400MB/s                 ~480MB/s
>> Host->Guest   ~1280MB/s                ~1350MB/s
>>
>> In this case, max send pkt size is still limited to 4K, so
Host->Guest
>> performance will worse than before.
> 
> It's concerning though, what if application sends small packets?
> What is the source of the slowdown? Do you know?
> 
Hi Michael,

Before performance is tested by me one month ago, I don't retest this time,
this result can have some fluctuations, today I will retest all of cases
included small and big packets, and try to find out the slowdown reason.

Thanks,
Yiwen.
>> After performance increase the max send pkt size to 64K:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~1700MB/s                ~2900MB/s
>> Host->Guest   ~1500MB/s                ~2440MB/s
>>
>> After performance all patches are used:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~1700MB/s                ~2900MB/s
>> Host->Guest   ~1700MB/s                ~2900MB/s
>>
>> >From the test results, the performance is improved obviously, and
guest
>> memory will not be wasted.
>>
>> In addition, in order to support mergeable rx buffer in virtio-vsock,
>> we need to add a qemu patch to support parse feature.
>>
>> ---
>> v1 -> v2:
>>  * Addressed comments from Jason Wang.
>>  * Add performance test result independently.
>>  * Use Skb_page_frag_refill() which can use high order page and reduce
>>    the stress of page allocator.
>>  * Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small
>>    size can't fill one full packet, we only 128 vq num now.
>>  * Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx
>>    consistency.
>>  * Add virtio_transport ops to get max pkt len, in order to be
compatible
>>    with old version.
>> ---
>>
>> Yiwen Jiang (5):
>>   VSOCK: support fill mergeable rx buffer in guest
>>   VSOCK: support fill data to mergeable rx buffer in host
>>   VSOCK: support receive mergeable rx buffer in guest
>>   VSOCK: increase send pkt len in mergeable mode to improve performance
>>   VSOCK: batch sending rx buffer to increase bandwidth
>>
>>  drivers/vhost/vsock.c                   | 183
++++++++++++++++++++-----
>>  include/linux/virtio_vsock.h            |  13 +-
>>  include/uapi/linux/virtio_vsock.h       |   5 +
>>  net/vmw_vsock/virtio_transport.c        | 229
+++++++++++++++++++++++++++-----
>>  net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
>>  5 files changed, 411 insertions(+), 85 deletions(-)
>>
>> -- 
>> 1.8.3.1
> 
> .
>

On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen
wrote:
> Now vsock only support send/receive small packet, it can't achieve
> high performance. As previous discussed with Jason Wang, I revisit the
> idea of vhost-net about mergeable rx buffer and implement the mergeable
> rx buffer in vhost-vsock, it can allow big packet to be scattered in
> into different buffers and improve performance obviously.
Sorry, I've been a bad maintainer.  I was focussed on other projects and
my email backlog is huge.

I like the idea of trying out optimizations on virtio-vsock, seeing if
code can be shared with virtio-net, and maybe later switching to a
virtio-net transport for vsock (if it turns out enough code can be
shared).

Another optimization that could be interesting:

Userspace processes reading from a socket sleep in
vsock_stream_recvmsg().  I wonder if we can bypass struct
virtio_vsock_pkt and copying the payload into pkt->buf in this case.
(This doesn't improve poll(2)/select(2) though!)

Imagine a userspace process waiting for data on a socket.  When the
virtqueue becomes ready, we can read in struct virtio_vsock_hdr and find
the socket for that connection.  Then we could copy the payload directly
to userspace instead of creating a virtio_vsock_pkt and copying to
pkt->buf first.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: not available
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20181213/358d534c/attachment.sig>

On 2018/12/14 0:34, Stefan Hajnoczi wrote:
> On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen wrote:
>> Now vsock only support send/receive small packet, it can't achieve
>> high performance. As previous discussed with Jason Wang, I revisit the
>> idea of vhost-net about mergeable rx buffer and implement the mergeable
>> rx buffer in vhost-vsock, it can allow big packet to be scattered in
>> into different buffers and improve performance obviously.
> 
> Sorry, I've been a bad maintainer.  I was focussed on other projects
and
> my email backlog is huge.
> 
> I like the idea of trying out optimizations on virtio-vsock, seeing if
> code can be shared with virtio-net, and maybe later switching to a
> virtio-net transport for vsock (if it turns out enough code can be
> shared).
> 
> Another optimization that could be interesting:
> 
> Userspace processes reading from a socket sleep in
> vsock_stream_recvmsg().  I wonder if we can bypass struct
> virtio_vsock_pkt and copying the payload into pkt->buf in this case.
> (This doesn't improve poll(2)/select(2) though!)
> 
> Imagine a userspace process waiting for data on a socket.  When the
> virtqueue becomes ready, we can read in struct virtio_vsock_hdr and find
> the socket for that connection.  Then we could copy the payload directly
> to userspace instead of creating a virtio_vsock_pkt and copying to
> pkt->buf first.
> 
Great, I also consider the optimization point later.
Then, I will send the next version based on your suggestions.

Thanks,
Yiwen.

On 2018/12/12 23:09, Michael S. Tsirkin wrote:
> On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen wrote:
>> Now vsock only support send/receive small packet, it can't achieve
>> high performance. As previous discussed with Jason Wang, I revisit the
>> idea of vhost-net about mergeable rx buffer and implement the mergeable
>> rx buffer in vhost-vsock, it can allow big packet to be scattered in
>> into different buffers and improve performance obviously.
>>
>> This series of patches mainly did three things?
>> - mergeable buffer implementation
>> - increase the max send pkt size
>> - add used and signal guest in a batch
>>
>> And I write a tool to test the vhost-vsock performance, mainly send big
>> packet(64K) included guest->Host and Host->Guest. I test
performance
>> independently and the result as follows:
>>
>> Before performance:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~400MB/s                 ~480MB/s
>> Host->Guest   ~1450MB/s                ~1600MB/s
>>
>> After performance only use implement mergeable rx buffer:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~400MB/s                 ~480MB/s
>> Host->Guest   ~1280MB/s                ~1350MB/s
>>
>> In this case, max send pkt size is still limited to 4K, so
Host->Guest
>> performance will worse than before.
> 
> It's concerning though, what if application sends small packets?
> What is the source of the slowdown? Do you know?
> 
Hi Michael,

To the two cases, I test the results included small and big packets as
follows:

64K packets performance comparison:
                                              Single socket    Multiple sockets
Host->Guest(before)                           1352.60MB/s      1436.33MB/s


Host->Guest(only use mergeable rx buffer)     1290.08MB/s      1212.67MB/s

4K packets performance comparison:
                                              Single socket    Multiple sockets
Host->Guest(before)                           535.47MB/s       688.67MB/s
Host->Guest(only use mergeable rx buffer)     522.33MB/s       599.00MB/s

3K packets performance comparison:
                                              Single socket    Multiple sockets
Host->Guest(before)                           359.74MB/s       442.00MB/s
Host->Guest(only use mergeable rx buffer)     374.47MB/s       452.33MB/s

We can see an interesting thing, for 64K and 4K packets,
using mergeable buffer has a poor performance, for 3K packet,
both have the same performance.

I guess in mergeable mode, when host send a 4k packet to guest, we
should call vhost_get_vq_desc() twice in host(hdr + 4k data),
and in guest we also should call virtqueue_get_buf() twice. So
when packet is smaller than (4k - hdr), it can be packed in a
single page, so the performance is the same as before.

So in the mergeable mode, the performance may be
worse in ((4k - hdr), 4k] than before.

Thanks,
Yiwen.
>> After performance increase the max send pkt size to 64K:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~1700MB/s                ~2900MB/s
>> Host->Guest   ~1500MB/s                ~2440MB/s
>>
>> After performance all patches are used:
>>               Single socket            Multiple sockets(Max Bandwidth)
>> Guest->Host   ~1700MB/s                ~2900MB/s
>> Host->Guest   ~1700MB/s                ~2900MB/s
>>
>> >From the test results, the performance is improved obviously, and
guest
>> memory will not be wasted.
>>
>> In addition, in order to support mergeable rx buffer in virtio-vsock,
>> we need to add a qemu patch to support parse feature.
>>
>> ---
>> v1 -> v2:
>>  * Addressed comments from Jason Wang.
>>  * Add performance test result independently.
>>  * Use Skb_page_frag_refill() which can use high order page and reduce
>>    the stress of page allocator.
>>  * Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small
>>    size can't fill one full packet, we only 128 vq num now.
>>  * Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx
>>    consistency.
>>  * Add virtio_transport ops to get max pkt len, in order to be
compatible
>>    with old version.
>> ---
>>
>> Yiwen Jiang (5):
>>   VSOCK: support fill mergeable rx buffer in guest
>>   VSOCK: support fill data to mergeable rx buffer in host
>>   VSOCK: support receive mergeable rx buffer in guest
>>   VSOCK: increase send pkt len in mergeable mode to improve performance
>>   VSOCK: batch sending rx buffer to increase bandwidth
>>
>>  drivers/vhost/vsock.c                   | 183
++++++++++++++++++++-----
>>  include/linux/virtio_vsock.h            |  13 +-
>>  include/uapi/linux/virtio_vsock.h       |   5 +
>>  net/vmw_vsock/virtio_transport.c        | 229
+++++++++++++++++++++++++++-----
>>  net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
>>  5 files changed, 411 insertions(+), 85 deletions(-)
>>
>> -- 
>> 1.8.3.1
> 
> .
>