Hi Stefan, Michael, Jason and everyone,
Several days ago, I discussed with jason about "Vsock over
Virtio-net".
This idea has two advantages:
First, it can use many great features of virtio-net, like batching,
mergeable rx buffer and multiqueue, etc.
Second, it can reduce many duplicate codes and make it easy to be
maintained.
Before the implement, I want to discuss with everyone again, and
want to know everyone's suggestions.
After the discussion, based on this point I will try to implement
this idea, but I am not familiar with the virtio-net, that is a
pity. :(
-------------------------Simple idea------------------------------
1. The packet layout will become as follows:
+---------------------------------+
| Virtio-net header |
|(struct virtio_net_hdr_mrg_rxbuf)|
+---------------------------------+
| Vsock header |
| (struct virtio_vsock_hdr) |
+---------------------------------+
| payload |
| (until end of packet) |
+---------------------------------+
2. The Guest->Host basic code flow as follow:
+------------+
| Client |
+------------+
|
|
+------------------------------------------------------------------+
|VSOCK Core Module |
|ops->sendmsg; (vsock_stream_sendmsg) |
| -> alloc_skb; /* it will packet a skb buffer, and include vsock |
| * hdr and payload */ |
| -> dev_queue_xmit(); /* it will call start_xmit(virtio-net.c) */|
|vsock hdr and payload, and then call |
+------------------------------------------------------------------+
|
|
+------------------------------------------------------------------+
|Virtio-net Module |
|start_xmit |
| -> add virtio_net_hdr and pack sg in ring desc, notify Host |
+------------------------------------------------------------------+
|
|
+------------------------------------------------------------------+
|Vhost-net Module |
|handle_tx |
| -> get tx buffer, skip virtio_net_hdr and call Vsock function. |
| /* This point has some differences, vhost-net use ->sendmsg to |
| * forward information, however vsock only need to notify server |
| * that data ready. */ |
+------------------------------------------------------------------+
|
|
+------------------------------------------------------------------+
|VSOCK Core Module |
|alloc_pkt, copy skb data to pkt. |
|add pkt to rx_queue and notify server to get data. |
+------------------------------------------------------------------+
3. To Host->Guest
I have a problem and difficult, mainly I know about virtio-net a little),
because I have been doing work related with storage and file system.
The problem as follows:
we should monitor all of socket of vsock in handle_rx, when there are
data coming, and copy data to vq desc. Vhost-net use ->recvmsg to
get data, it is different with socket. To vsock, I think host will
not call ->recvmsg when it need to send message to guest. To net,
vhost-net only as forwarding layer.
On 2018/11/15 ??11:56, jiangyiwen wrote:> Hi Stefan, Michael, Jason and everyone, > > Several days ago, I discussed with jason about "Vsock over Virtio-net". > This idea has two advantages: > First, it can use many great features of virtio-net, like batching, > mergeable rx buffer and multiqueue, etc. > Second, it can reduce many duplicate codes and make it easy to be > maintained. > > Before the implement, I want to discuss with everyone again, and > want to know everyone's suggestions. > > After the discussion, based on this point I will try to implement > this idea, but I am not familiar with the virtio-net, that is a > pity. :(I think we should have a new feature flag for this. E.g VIRTIO_NET_F_VSOCK. And host should fail the negotiation if guest doesn't support this to avoid confusion. When this feature is negotiated, we will use it only for VOSCK transport. This can simplify things somehow.> -------------------------Simple idea------------------------------ > > 1. The packet layout will become as follows: > > +---------------------------------+ > | Virtio-net header | > |(struct virtio_net_hdr_mrg_rxbuf)| > +---------------------------------+ > | Vsock header | > | (struct virtio_vsock_hdr) | > +---------------------------------+ > | payload | > | (until end of packet) | > +---------------------------------+ > > 2. The Guest->Host basic code flow as follow: > +------------+ > | Client | > +------------+ > | > | > +------------------------------------------------------------------+ > |VSOCK Core Module | > |ops->sendmsg; (vsock_stream_sendmsg) | > | -> alloc_skb; /* it will packet a skb buffer, and include vsock | > | * hdr and payload */ | > | -> dev_queue_xmit(); /* it will call start_xmit(virtio-net.c) */| > |vsock hdr and payload, and then call | > +------------------------------------------------------------------+Note, if we've negotiated the feature, virtio-net driver must not use register_netdev to register it to network core. This can avoid lots of confusion.> | > | > +------------------------------------------------------------------+ > |Virtio-net Module | > |start_xmit | > | -> add virtio_net_hdr and pack sg in ring desc, notify Host | > +------------------------------------------------------------------+ > | > | > +------------------------------------------------------------------+ > |Vhost-net Module | > |handle_tx | > | -> get tx buffer, skip virtio_net_hdr and call Vsock function. | > | /* This point has some differences, vhost-net use ->sendmsg to | > | * forward information, however vsock only need to notify server | > | * that data ready. */ | > +------------------------------------------------------------------+When VIRTIO_NET_F_VOSCK is negotiated, we know that it's a vsock transport, we can then forward it to vsock core.> | > | > +------------------------------------------------------------------+ > |VSOCK Core Module | > |alloc_pkt, copy skb data to pkt. | > |add pkt to rx_queue and notify server to get data. | > +------------------------------------------------------------------+ > > 3. To Host->Guest > I have a problem and difficult, mainly I know about virtio-net a little), > because I have been doing work related with storage and file system. > > The problem as follows: > we should monitor all of socket of vsock in handle_rx, when there are > data coming, and copy data to vq desc. Vhost-net use ->recvmsg to > get data, it is different with socket. To vsock, I think host will > not call ->recvmsg when it need to send message to guest. To net, > vhost-net only as forwarding layer.Know not much here, but is it possible to have a vsock(tap) to be passed to vhost_net and let vhost call its recvmgs()? Bascially it was a socket on host as well I believe? If this doesn't work, we can have vsock specific receiving routine in vhost_net if VIRTIO_NET_F_VOSCK is negotiated. Generally, I think we should try out best to keep the exist sendmsg()/recvmsg() interfaces and only consider the alternatives if we meet some real blocker. Thanks>
On 2018/11/15 12:19, Jason Wang wrote:> > On 2018/11/15 ??11:56, jiangyiwen wrote: >> Hi Stefan, Michael, Jason and everyone, >> >> Several days ago, I discussed with jason about "Vsock over Virtio-net". >> This idea has two advantages: >> First, it can use many great features of virtio-net, like batching, >> mergeable rx buffer and multiqueue, etc. >> Second, it can reduce many duplicate codes and make it easy to be >> maintained. >> >> Before the implement, I want to discuss with everyone again, and >> want to know everyone's suggestions. >> >> After the discussion, based on this point I will try to implement >> this idea, but I am not familiar with the virtio-net, that is a >> pity. :( > > > I think we should have a new feature flag for this. E.g VIRTIO_NET_F_VSOCK. And host should fail the negotiation if guest doesn't support this to avoid confusion. When this feature is negotiated, we will use it only for VOSCK transport. This can simplify things somehow. > > >> -------------------------Simple idea------------------------------ >> >> 1. The packet layout will become as follows: >> >> +---------------------------------+ >> | Virtio-net header | >> |(struct virtio_net_hdr_mrg_rxbuf)| >> +---------------------------------+ >> | Vsock header | >> | (struct virtio_vsock_hdr) | >> +---------------------------------+ >> | payload | >> | (until end of packet) | >> +---------------------------------+ >> >> 2. The Guest->Host basic code flow as follow: >> +------------+ >> | Client | >> +------------+ >> | >> | >> +------------------------------------------------------------------+ >> |VSOCK Core Module | >> |ops->sendmsg; (vsock_stream_sendmsg) | >> | -> alloc_skb; /* it will packet a skb buffer, and include vsock | >> | * hdr and payload */ | >> | -> dev_queue_xmit(); /* it will call start_xmit(virtio-net.c) */| >> |vsock hdr and payload, and then call | >> +------------------------------------------------------------------+ > > > Note, if we've negotiated the feature, virtio-net driver must not use register_netdev to register it to network core. This can avoid lots of confusion.Hi Jason, You mean we should not register netdev if use vsock, and in order to avoid confusion, then I think whether we should keep vsock and export some virtio-net's functions that can be shared. In this way, first, vsock may keep existing architecture and will not affect virtio-net. In addition, vsock doesn't need to use virtio_net header too, then it don't need to pack skb structure. Thanks, Yiwen.> > >> | >> | >> +------------------------------------------------------------------+ >> |Virtio-net Module | >> |start_xmit | >> | -> add virtio_net_hdr and pack sg in ring desc, notify Host | >> +------------------------------------------------------------------+ >> | >> | >> +------------------------------------------------------------------+ >> |Vhost-net Module | >> |handle_tx | >> | -> get tx buffer, skip virtio_net_hdr and call Vsock function. | >> | /* This point has some differences, vhost-net use ->sendmsg to | >> | * forward information, however vsock only need to notify server | >> | * that data ready. */ | >> +------------------------------------------------------------------+ > > > When VIRTIO_NET_F_VOSCK is negotiated, we know that it's a vsock transport, we can then forward it to vsock core. > > >> | >> | >> +------------------------------------------------------------------+ >> |VSOCK Core Module | >> |alloc_pkt, copy skb data to pkt. | >> |add pkt to rx_queue and notify server to get data. | >> +------------------------------------------------------------------+ >> >> 3. To Host->Guest >> I have a problem and difficult, mainly I know about virtio-net a little), >> because I have been doing work related with storage and file system. >> >> The problem as follows: >> we should monitor all of socket of vsock in handle_rx, when there are >> data coming, and copy data to vq desc. Vhost-net use ->recvmsg to >> get data, it is different with socket. To vsock, I think host will >> not call ->recvmsg when it need to send message to guest. To net, >> vhost-net only as forwarding layer. > > Know not much here, but is it possible to have a vsock(tap) to be passed to vhost_net and let vhost call its recvmgs()? Bascially it was a socket on host as well I believe?For vsock, Host->Guest, it's code flow as follows: Server call send() -> sendmsg(); (vsock_stream_sendmsg) -> virtio_trasnport_send_pkt_info -> alloc pkt, add pkt to send_pkt_list, wake up vhost_worker Vhost_worker -> vhost_transport_send_pkt_work -> get pkt from send_pkt_list -> get vq input desc and then fill data to desc addr -> update used ring and then signal guest In the whole process, host don't call recvmsg() because it is a net device, and it also receives any messages. For vhost-net, I understand it is a tap device, so it can receive messages from other net device. This is my understanding, it may have some errors. Thanks.> > If this doesn't work, we can have vsock specific receiving routine in vhost_net if VIRTIO_NET_F_VOSCK is negotiated. > > Generally, I think we should try out best to keep the exist sendmsg()/recvmsg() interfaces and only consider the alternatives if we meet some real blocker. > > Thanks > > >> > > . >
Michael S. Tsirkin
2018-Nov-15 07:04 UTC
[RFC] Discuss about an new idea "Vsock over Virtio-net"
On Thu, Nov 15, 2018 at 11:56:03AM +0800, jiangyiwen wrote:> Hi Stefan, Michael, Jason and everyone, > > Several days ago, I discussed with jason about "Vsock over Virtio-net". > This idea has two advantages: > First, it can use many great features of virtio-net, like batching, > mergeable rx buffer and multiqueue, etc. > Second, it can reduce many duplicate codes and make it easy to be > maintained.I'm not sure I get the motivation. Which features of virtio net are relevant to vsock? The ones that you mention all seem to be mostly of use to the networking stack.> Before the implement, I want to discuss with everyone again, and > want to know everyone's suggestions. > > After the discussion, based on this point I will try to implement > this idea, but I am not familiar with the virtio-net, that is a > pity. :( > > -------------------------Simple idea------------------------------ > > 1. The packet layout will become as follows: > > +---------------------------------+ > | Virtio-net header | > |(struct virtio_net_hdr_mrg_rxbuf)|Which fields in virtio_net_hdr_mrg_rxbuf are of interest to vsock?> +---------------------------------+ > | Vsock header | > | (struct virtio_vsock_hdr) | > +---------------------------------+ > | payload | > | (until end of packet) | > +---------------------------------+Thanks, -- MST
On 2018/11/15 15:04, Michael S. Tsirkin wrote:> On Thu, Nov 15, 2018 at 11:56:03AM +0800, jiangyiwen wrote: >> Hi Stefan, Michael, Jason and everyone, >> >> Several days ago, I discussed with jason about "Vsock over Virtio-net". >> This idea has two advantages: >> First, it can use many great features of virtio-net, like batching, >> mergeable rx buffer and multiqueue, etc. >> Second, it can reduce many duplicate codes and make it easy to be >> maintained. > > I'm not sure I get the motivation. Which features of > virtio net are relevant to vsock? The ones that you mention > all seem to be mostly of use to the networking stack. > > >> Before the implement, I want to discuss with everyone again, and >> want to know everyone's suggestions. >> >> After the discussion, based on this point I will try to implement >> this idea, but I am not familiar with the virtio-net, that is a >> pity. :( >> >> -------------------------Simple idea------------------------------ >> >> 1. The packet layout will become as follows: >> >> +---------------------------------+ >> | Virtio-net header | >> |(struct virtio_net_hdr_mrg_rxbuf)| > > Which fields in virtio_net_hdr_mrg_rxbuf are of interest to vsock? >Hi Michael, Yes, currently vsock has poor performance, first, it only support transport small packet, in order to make the balance between performance and guest memory. In order to solve this problem, there are two features vsock can used, mergeable rx buffer and multiqueue. Based on this, there are some shared codes vsock can use. Thanks, Yiwen.>> +---------------------------------+ >> | Vsock header | >> | (struct virtio_vsock_hdr) | >> +---------------------------------+ >> | payload | >> | (until end of packet) | >> +---------------------------------+ > > Thanks, >
On 2018/11/15 ??3:04, Michael S. Tsirkin wrote:> On Thu, Nov 15, 2018 at 11:56:03AM +0800, jiangyiwen wrote: >> Hi Stefan, Michael, Jason and everyone, >> >> Several days ago, I discussed with jason about "Vsock over Virtio-net". >> This idea has two advantages: >> First, it can use many great features of virtio-net, like batching, >> mergeable rx buffer and multiqueue, etc. >> Second, it can reduce many duplicate codes and make it easy to be >> maintained. > I'm not sure I get the motivation. Which features of > virtio net are relevant to vsock?Vsock is just a L2 (and above) protocol from the view of the device. So I think we should answer the question why we need two different paths for networking traffic? Or what is the fundamental reason that makes vsock does not go for virtio-net? I agree they could be different type of devices but codes could be shared in both guest and host (or even qemu) for not duplicating features(bugs). Thanks> The ones that you mention > all seem to be mostly of use to the networking stack. > >
Yan Vugenfirer
2018-Nov-15 09:36 UTC
[RFC] Discuss about an new idea "Vsock over Virtio-net"
> On 15 Nov 2018, at 05:56, jiangyiwen <jiangyiwen at huawei.com> wrote: > > Hi Stefan, Michael, Jason and everyone, > > Several days ago, I discussed with jason about "Vsock over Virtio-net". > This idea has two advantages: > First, it can use many great features of virtio-net, like batching, > mergeable rx buffer and multiqueue, etc. > Second, it can reduce many duplicate codes and make it easy to be > maintained.I would like to add that from Windows guest support perspective it makes more sense. To support vsock we most probably need NDIS protocol driver and if it will be binded to NDIS miniport driver (the driver that currently handles virtio-net in Windows) it will make our life much easier. Otherwise we need to ?plug? into NIDS on one hand, and handle transport outside of NDIS on other hand or in some strange way by creating mini port driver that is not really NIC miniport driver. Best regards, Yan.> > Before the implement, I want to discuss with everyone again, and > want to know everyone's suggestions. > > After the discussion, based on this point I will try to implement > this idea, but I am not familiar with the virtio-net, that is a > pity. :( > > -------------------------Simple idea------------------------------ > > 1. The packet layout will become as follows: > > +---------------------------------+ > | Virtio-net header | > |(struct virtio_net_hdr_mrg_rxbuf)| > +---------------------------------+ > | Vsock header | > | (struct virtio_vsock_hdr) | > +---------------------------------+ > | payload | > | (until end of packet) | > +---------------------------------+ > > 2. The Guest->Host basic code flow as follow: > +------------+ > | Client | > +------------+ > | > | > +------------------------------------------------------------------+ > |VSOCK Core Module | > |ops->sendmsg; (vsock_stream_sendmsg) | > | -> alloc_skb; /* it will packet a skb buffer, and include vsock | > | * hdr and payload */ | > | -> dev_queue_xmit(); /* it will call start_xmit(virtio-net.c) */| > |vsock hdr and payload, and then call | > +------------------------------------------------------------------+ > | > | > +------------------------------------------------------------------+ > |Virtio-net Module | > |start_xmit | > | -> add virtio_net_hdr and pack sg in ring desc, notify Host | > +------------------------------------------------------------------+ > | > | > +------------------------------------------------------------------+ > |Vhost-net Module | > |handle_tx | > | -> get tx buffer, skip virtio_net_hdr and call Vsock function. | > | /* This point has some differences, vhost-net use ->sendmsg to | > | * forward information, however vsock only need to notify server | > | * that data ready. */ | > +------------------------------------------------------------------+ > | > | > +------------------------------------------------------------------+ > |VSOCK Core Module | > |alloc_pkt, copy skb data to pkt. | > |add pkt to rx_queue and notify server to get data. | > +------------------------------------------------------------------+ > > 3. To Host->Guest > I have a problem and difficult, mainly I know about virtio-net a little), > because I have been doing work related with storage and file system. > > The problem as follows: > we should monitor all of socket of vsock in handle_rx, when there are > data coming, and copy data to vq desc. Vhost-net use ->recvmsg to > get data, it is different with socket. To vsock, I think host will > not call ->recvmsg when it need to send message to guest. To net, > vhost-net only as forwarding layer. >
On 2018/11/15 17:36, Yan Vugenfirer wrote:> > >> On 15 Nov 2018, at 05:56, jiangyiwen <jiangyiwen at huawei.com> wrote: >> >> Hi Stefan, Michael, Jason and everyone, >> >> Several days ago, I discussed with jason about "Vsock over Virtio-net". >> This idea has two advantages: >> First, it can use many great features of virtio-net, like batching, >> mergeable rx buffer and multiqueue, etc. >> Second, it can reduce many duplicate codes and make it easy to be >> maintained. > > I would like to add that from Windows guest support perspective it makes more sense. > To support vsock we most probably need NDIS protocol driver and if it will be binded to NDIS miniport driver (the driver that currently handles virtio-net in Windows) it will make our life much easier. Otherwise we need to ?plug? into NIDS on one hand, and handle transport outside of NDIS on other hand or in some strange way by creating mini port driver that is not really NIC miniport driver. > > Best regards, > Yan.Sorry, I am not familiar with windows vsock driver, if vsock is integrated in virtio-net, it can make windows vsock more easier, then it should be a more suitable solution. Thanks.>> >> Before the implement, I want to discuss with everyone again, and >> want to know everyone's suggestions. >> >> After the discussion, based on this point I will try to implement >> this idea, but I am not familiar with the virtio-net, that is a >> pity. :( >> >> -------------------------Simple idea------------------------------ >> >> 1. The packet layout will become as follows: >> >> +---------------------------------+ >> | Virtio-net header | >> |(struct virtio_net_hdr_mrg_rxbuf)| >> +---------------------------------+ >> | Vsock header | >> | (struct virtio_vsock_hdr) | >> +---------------------------------+ >> | payload | >> | (until end of packet) | >> +---------------------------------+ >> >> 2. The Guest->Host basic code flow as follow: >> +------------+ >> | Client | >> +------------+ >> | >> | >> +------------------------------------------------------------------+ >> |VSOCK Core Module | >> |ops->sendmsg; (vsock_stream_sendmsg) | >> | -> alloc_skb; /* it will packet a skb buffer, and include vsock | >> | * hdr and payload */ | >> | -> dev_queue_xmit(); /* it will call start_xmit(virtio-net.c) */| >> |vsock hdr and payload, and then call | >> +------------------------------------------------------------------+ >> | >> | >> +------------------------------------------------------------------+ >> |Virtio-net Module | >> |start_xmit | >> | -> add virtio_net_hdr and pack sg in ring desc, notify Host | >> +------------------------------------------------------------------+ >> | >> | >> +------------------------------------------------------------------+ >> |Vhost-net Module | >> |handle_tx | >> | -> get tx buffer, skip virtio_net_hdr and call Vsock function. | >> | /* This point has some differences, vhost-net use ->sendmsg to | >> | * forward information, however vsock only need to notify server | >> | * that data ready. */ | >> +------------------------------------------------------------------+ >> | >> | >> +------------------------------------------------------------------+ >> |VSOCK Core Module | >> |alloc_pkt, copy skb data to pkt. | >> |add pkt to rx_queue and notify server to get data. | >> +------------------------------------------------------------------+ >> >> 3. To Host->Guest >> I have a problem and difficult, mainly I know about virtio-net a little), >> because I have been doing work related with storage and file system. >> >> The problem as follows: >> we should monitor all of socket of vsock in handle_rx, when there are >> data coming, and copy data to vq desc. Vhost-net use ->recvmsg to >> get data, it is different with socket. To vsock, I think host will >> not call ->recvmsg when it need to send message to guest. To net, >> vhost-net only as forwarding layer. >> > > > . >
Possibly Parallel Threads
- [RFC] Discuss about an new idea "Vsock over Virtio-net"
- [RFC] Discuss about an new idea "Vsock over Virtio-net"
- [RFC] Discuss about an new idea "Vsock over Virtio-net"
- [RFC] Discuss about an new idea "Vsock over Virtio-net"
- [RFC] Discuss about an new idea "Vsock over Virtio-net"