Linux Virtualization - Nov 2018 - [RFC] Discuss about an new idea "Vsock over Virtio-net"

On 2018/11/15 16:19, Jason Wang wrote:
> 
> On 2018/11/15 ??2:46, jiangyiwen wrote:
>> On 2018/11/15 12:19, Jason Wang wrote:
>>> On 2018/11/15 ??11:56, jiangyiwen wrote:
>>>> Hi Stefan, Michael, Jason and everyone,
>>>>
>>>> Several days ago, I discussed with jason about "Vsock over
Virtio-net".
>>>> This idea has two advantages:
>>>> First, it can use many great features of virtio-net, like
batching,
>>>> mergeable rx buffer and multiqueue, etc.
>>>> Second, it can reduce many duplicate codes and make it easy to
be
>>>> maintained.
>>>>
>>>> Before the implement, I want to discuss with everyone again,
and
>>>> want to know everyone's suggestions.
>>>>
>>>> After the discussion, based on this point I will try to
implement
>>>> this idea, but I am not familiar with the virtio-net, that is a
>>>> pity. :(
>>>
>>> I think we should have a new feature flag for this. E.g
VIRTIO_NET_F_VSOCK. And host should fail the negotiation if guest doesn't
support this to avoid confusion. When this feature is negotiated, we will use it
only for VOSCK transport. This can simplify things somehow.
>>>
>>>
>>>> -------------------------Simple
idea------------------------------
>>>>
>>>> 1. The packet layout will become as follows:
>>>>
>>>> +---------------------------------+
>>>> |        Virtio-net header        |
>>>> |(struct virtio_net_hdr_mrg_rxbuf)|
>>>> +---------------------------------+
>>>> |          Vsock header           |
>>>> |    (struct virtio_vsock_hdr)    |
>>>> +---------------------------------+
>>>> |             payload             |
>>>> |      (until end of packet)      |
>>>> +---------------------------------+
>>>>
>>>> 2. The Guest->Host basic code flow as follow:
>>>>                               +------------+
>>>>                               |   Client   |
>>>>                               +------------+
>>>>                                     |
>>>>                                     |
>>>>
+------------------------------------------------------------------+
>>>> |VSOCK Core Module                                             
|
>>>> |ops->sendmsg; (vsock_stream_sendmsg)                       
|
>>>> |  -> alloc_skb; /* it will packet a skb buffer, and include
vsock |
>>>> |                 * hdr and payload */                         
|
>>>> |  -> dev_queue_xmit(); /* it will call
start_xmit(virtio-net.c) */|
>>>> |vsock hdr and payload, and then call                          
|
>>>>
+------------------------------------------------------------------+
>>>
>>> Note, if we've negotiated the feature, virtio-net driver must
not use register_netdev to register it to network core. This can avoid lots of
confusion.
>> Hi Jason,
>>
>> You mean we should not register netdev if use vsock, and in order to
>> avoid confusion, then I think whether we should keep vsock and export
>> some virtio-net's functions that can be shared. In this way, first,
vsock
>> may keep existing architecture and will not affect virtio-net.
> 
> 
> At least it needs new feature bits which will be a functional duplication
of virtio-net (e.g mrg_rxbuf).
Hi Jason,

Actually I mean only use some shared function to make vsock support these
features, in that way, guest see the device is still vsock device instead of
virtio-net device, in addition, it can have less codes and easier to be
compatible with old vsock version.

Thanks,
Yiwen.
> 
> 
>> In addition,
>> vsock doesn't need to use virtio_net header too, then it don't
need to pack
>> skb structure.
> 
> 
> For mergeable rx buffer it need I think?
As said above, I will define the related structure in the virtio-vsock module.
> 
> 
>>
>> Thanks,
>> Yiwen.
>>
>>>
>>>>                                     |
>>>>                                     |
>>>>
+------------------------------------------------------------------+
>>>> |Virtio-net Module                                             
|
>>>> |start_xmit                                                    
|
>>>> |  -> add virtio_net_hdr and pack sg in ring desc, notify
Host     |
>>>>
+------------------------------------------------------------------+
>>>>                                     |
>>>>                                     |
>>>>
+------------------------------------------------------------------+
>>>> |Vhost-net Module                                              
|
>>>> |handle_tx                                                     
|
>>>> |  -> get tx buffer, skip virtio_net_hdr and call Vsock
function.  |
>>>> | /* This point has some differences, vhost-net use
->sendmsg to   |
>>>> |  * forward information, however vsock only need to notify
server |
>>>> |  * that data ready. */                                       
|
>>>>
+------------------------------------------------------------------+
>>>
>>> When VIRTIO_NET_F_VOSCK is negotiated, we know that it's a
vsock transport, we can then forward it to vsock core.
>>>
>>>
>>>>                                     |
>>>>                                     |
>>>>
+------------------------------------------------------------------+
>>>> |VSOCK Core Module                                             
|
>>>> |alloc_pkt, copy skb data to pkt.                              
|
>>>> |add pkt to rx_queue and notify server to get data.            
|
>>>>
+------------------------------------------------------------------+
>>>>
>>>> 3. To Host->Guest
>>>> I have a problem and difficult, mainly I know about virtio-net
a little),
>>>> because I have been doing work related with storage and file
system.
>>>>
>>>> The problem as follows:
>>>> we should monitor all of socket of vsock in handle_rx, when
there are
>>>> data coming, and copy data to vq desc. Vhost-net use
->recvmsg to
>>>> get data, it is different with socket. To vsock, I think host
will
>>>> not call ->recvmsg when it need to send message to guest. To
net,
>>>> vhost-net only as forwarding layer.
>>> Know not much here, but is it possible to have a vsock(tap) to be
passed to vhost_net and let vhost call its recvmgs()? Bascially it was a socket
on host as well I believe?
>> For vsock, Host->Guest, it's code flow as follows:
>> Server call send()
>>    -> sendmsg(); (vsock_stream_sendmsg)
>>      -> virtio_trasnport_send_pkt_info
>>        -> alloc pkt, add pkt to send_pkt_list, wake up vhost_worker
>>
>> Vhost_worker
>>    -> vhost_transport_send_pkt_work
>>      -> get pkt from send_pkt_list
>>      -> get vq input desc and then fill data to desc addr
>>      -> update used ring and then signal guest
>>
>> In the whole process, host don't call recvmsg() because it is a net
device, and
>> it also receives any messages.
> 
> 
> If I understand this correctly, there's no a socket object on host that
represent vosck in guest? If yes, maybe we can invent one.
> 
> Thanks
> 
Sorry, I am not understanding you very much, vsock only a socket
channel, it does not have a network device entity, so it only
transmit the data between server and client, the data is only
saved in server and client. Instead of vhost-net, I feel it
has a network device that can saved the data, so when host
send message to guest, it can use recvmsg() from the
network device(tap). For Vsock, recvmsg() interface will
read message from tx vq.
> 
>>
>> For vhost-net, I understand it is a tap device, so it can receive
messages
>> from other net device.
>>
>> This is my understanding, it may have some errors.
>>
>> Thanks.
>>
>>> If this doesn't work, we can have vsock specific receiving
routine in vhost_net if VIRTIO_NET_F_VOSCK is negotiated.
>>>
>>> Generally, I think we should try out best to keep the exist
sendmsg()/recvmsg() interfaces and only consider the alternatives if we meet
some real blocker.
>>>
>>> Thanks
>>>
>>>
>>> .
>>>
>>
> 
> .
>

On 2018/11/15 ??5:02, jiangyiwen wrote:
> On 2018/11/15 16:19, Jason Wang wrote:
>> On 2018/11/15 ??2:46, jiangyiwen wrote:
>>> On 2018/11/15 12:19, Jason Wang wrote:
>>>> On 2018/11/15 ??11:56, jiangyiwen wrote:
>>>>> Hi Stefan, Michael, Jason and everyone,
>>>>>
>>>>> Several days ago, I discussed with jason about "Vsock
over Virtio-net".
>>>>> This idea has two advantages:
>>>>> First, it can use many great features of virtio-net, like
batching,
>>>>> mergeable rx buffer and multiqueue, etc.
>>>>> Second, it can reduce many duplicate codes and make it easy
to be
>>>>> maintained.
>>>>>
>>>>> Before the implement, I want to discuss with everyone
again, and
>>>>> want to know everyone's suggestions.
>>>>>
>>>>> After the discussion, based on this point I will try to
implement
>>>>> this idea, but I am not familiar with the virtio-net, that
is a
>>>>> pity. :(
>>>> I think we should have a new feature flag for this. E.g
VIRTIO_NET_F_VSOCK. And host should fail the negotiation if guest doesn't
support this to avoid confusion. When this feature is negotiated, we will use it
only for VOSCK transport. This can simplify things somehow.
>>>>
>>>>
>>>>> -------------------------Simple
idea------------------------------
>>>>>
>>>>> 1. The packet layout will become as follows:
>>>>>
>>>>> +---------------------------------+
>>>>> |        Virtio-net header        |
>>>>> |(struct virtio_net_hdr_mrg_rxbuf)|
>>>>> +---------------------------------+
>>>>> |          Vsock header           |
>>>>> |    (struct virtio_vsock_hdr)    |
>>>>> +---------------------------------+
>>>>> |             payload             |
>>>>> |      (until end of packet)      |
>>>>> +---------------------------------+
>>>>>
>>>>> 2. The Guest->Host basic code flow as follow:
>>>>>                                +------------+
>>>>>                                |   Client   |
>>>>>                                +------------+
>>>>>                                      |
>>>>>                                      |
>>>>>
+------------------------------------------------------------------+
>>>>> |VSOCK Core Module                                         
|
>>>>> |ops->sendmsg; (vsock_stream_sendmsg)                   
|
>>>>> |  -> alloc_skb; /* it will packet a skb buffer, and
include vsock |
>>>>> |                 * hdr and payload */                     
|
>>>>> |  -> dev_queue_xmit(); /* it will call
start_xmit(virtio-net.c) */|
>>>>> |vsock hdr and payload, and then call                      
|
>>>>>
+------------------------------------------------------------------+
>>>> Note, if we've negotiated the feature, virtio-net driver
must not use register_netdev to register it to network core. This can avoid lots
of confusion.
>>> Hi Jason,
>>>
>>> You mean we should not register netdev if use vsock, and in order
to
>>> avoid confusion, then I think whether we should keep vsock and
export
>>> some virtio-net's functions that can be shared. In this way,
first, vsock
>>> may keep existing architecture and will not affect virtio-net.
>>
>> At least it needs new feature bits which will be a functional
duplication of virtio-net (e.g mrg_rxbuf).
> Hi Jason,
>
> Actually I mean only use some shared function to make vsock support these
> features, in that way, guest see the device is still vsock device instead
of
> virtio-net device, in addition, it can have less codes and easier to be
> compatible with old vsock version.

Yes, I think we're talking about same thing. Both of us want to share 
codes. What you want is to export and share some common helpers between 
virtio-net and vsock. What I meant is to e.g probe vsock device and 
merge vsock specific codes into virtio-net driver. I agree it's not a 
small project. We can start from e.g patches that try to share the 
codes. This could also give us inspiration for how to unify them.

>
> Thanks,
> Yiwen.
>
>>
>>> In addition,
>>> vsock doesn't need to use virtio_net header too, then it
don't need to pack
>>> skb structure.
>>
>> For mergeable rx buffer it need I think?
> As said above, I will define the related structure in the virtio-vsock
module.
>
>>
>>> Thanks,
>>> Yiwen.
>>>
>>>>>                                      |
>>>>>                                      |
>>>>>
+------------------------------------------------------------------+
>>>>> |Virtio-net Module                                         
|
>>>>> |start_xmit                                                
|
>>>>> |  -> add virtio_net_hdr and pack sg in ring desc,
notify Host     |
>>>>>
+------------------------------------------------------------------+
>>>>>                                      |
>>>>>                                      |
>>>>>
+------------------------------------------------------------------+
>>>>> |Vhost-net Module                                          
|
>>>>> |handle_tx                                                 
|
>>>>> |  -> get tx buffer, skip virtio_net_hdr and call Vsock
function.  |
>>>>> | /* This point has some differences, vhost-net use
->sendmsg to   |
>>>>> |  * forward information, however vsock only need to notify
server |
>>>>> |  * that data ready. */                                   
|
>>>>>
+------------------------------------------------------------------+
>>>> When VIRTIO_NET_F_VOSCK is negotiated, we know that it's a
vsock transport, we can then forward it to vsock core.
>>>>
>>>>
>>>>>                                      |
>>>>>                                      |
>>>>>
+------------------------------------------------------------------+
>>>>> |VSOCK Core Module                                         
|
>>>>> |alloc_pkt, copy skb data to pkt.                          
|
>>>>> |add pkt to rx_queue and notify server to get data.        
|
>>>>>
+------------------------------------------------------------------+
>>>>>
>>>>> 3. To Host->Guest
>>>>> I have a problem and difficult, mainly I know about
virtio-net a little),
>>>>> because I have been doing work related with storage and
file system.
>>>>>
>>>>> The problem as follows:
>>>>> we should monitor all of socket of vsock in handle_rx, when
there are
>>>>> data coming, and copy data to vq desc. Vhost-net use
->recvmsg to
>>>>> get data, it is different with socket. To vsock, I think
host will
>>>>> not call ->recvmsg when it need to send message to
guest. To net,
>>>>> vhost-net only as forwarding layer.
>>>> Know not much here, but is it possible to have a vsock(tap) to
be passed to vhost_net and let vhost call its recvmgs()? Bascially it was a
socket on host as well I believe?
>>> For vsock, Host->Guest, it's code flow as follows:
>>> Server call send()
>>>     -> sendmsg(); (vsock_stream_sendmsg)
>>>       -> virtio_trasnport_send_pkt_info
>>>         -> alloc pkt, add pkt to send_pkt_list, wake up
vhost_worker
>>>
>>> Vhost_worker
>>>     -> vhost_transport_send_pkt_work
>>>       -> get pkt from send_pkt_list
>>>       -> get vq input desc and then fill data to desc addr
>>>       -> update used ring and then signal guest
>>>
>>> In the whole process, host don't call recvmsg() because it is a
net device, and
>>> it also receives any messages.
>>
>> If I understand this correctly, there's no a socket object on host
that represent vosck in guest? If yes, maybe we can invent one.
>>
>> Thanks
>>
> Sorry, I am not understanding you very much, vsock only a socket
> channel, it does not have a network device entity, so it only
> transmit the data between server and client, the data is only
> saved in server and client. Instead of vhost-net, I feel it
> has a network device that can saved the data, so when host
> send message to guest, it can use recvmsg() from the
> network device(tap). For Vsock, recvmsg() interface will
> read message from tx vq.

So I understand the model is not a real socket pair on host which 
AF_UNIX did. Maybe it's better to have one. What I meant is, have a 
socket that represent? for each guest vsock device on host (guest 
socket). Then when you transfer packets from host to guest, you can 
queue the packets into the receive queue of the guest socket and wake up 
vhost-net and it will call recvmsg() for the guest socket. And when you 
want to transfer packets form guest to host, vhost_net will call 
sendmsg() to the guest socket on host then it can search the correct 
destination and queue packet on the receive of host socket. This can 
make vsock much more easier to be integrated with vhost_net.

Thanks
>
>>> For vhost-net, I understand it is a tap device, so it can receive
messages
>>> from other net device.
>>>
>>> This is my understanding, it may have some errors.
>>>
>>> Thanks.
>>>
>>>> If this doesn't work, we can have vsock specific receiving
routine in vhost_net if VIRTIO_NET_F_VOSCK is negotiated.
>>>>
>>>> Generally, I think we should try out best to keep the exist
sendmsg()/recvmsg() interfaces and only consider the alternatives if we meet
some real blocker.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> .
>>>>
>> .
>>
>

On 2018/11/15 17:21, Jason Wang wrote:
> 
> On 2018/11/15 ??5:02, jiangyiwen wrote:
>> On 2018/11/15 16:19, Jason Wang wrote:
>>> On 2018/11/15 ??2:46, jiangyiwen wrote:
>>>> On 2018/11/15 12:19, Jason Wang wrote:
>>>>> On 2018/11/15 ??11:56, jiangyiwen wrote:
>>>>>> Hi Stefan, Michael, Jason and everyone,
>>>>>>
>>>>>> Several days ago, I discussed with jason about
"Vsock over Virtio-net".
>>>>>> This idea has two advantages:
>>>>>> First, it can use many great features of virtio-net,
like batching,
>>>>>> mergeable rx buffer and multiqueue, etc.
>>>>>> Second, it can reduce many duplicate codes and make it
easy to be
>>>>>> maintained.
>>>>>>
>>>>>> Before the implement, I want to discuss with everyone
again, and
>>>>>> want to know everyone's suggestions.
>>>>>>
>>>>>> After the discussion, based on this point I will try to
implement
>>>>>> this idea, but I am not familiar with the virtio-net,
that is a
>>>>>> pity. :(
>>>>> I think we should have a new feature flag for this. E.g
VIRTIO_NET_F_VSOCK. And host should fail the negotiation if guest doesn't
support this to avoid confusion. When this feature is negotiated, we will use it
only for VOSCK transport. This can simplify things somehow.
>>>>>
>>>>>
>>>>>> -------------------------Simple
idea------------------------------
>>>>>>
>>>>>> 1. The packet layout will become as follows:
>>>>>>
>>>>>> +---------------------------------+
>>>>>> |        Virtio-net header        |
>>>>>> |(struct virtio_net_hdr_mrg_rxbuf)|
>>>>>> +---------------------------------+
>>>>>> |          Vsock header           |
>>>>>> |    (struct virtio_vsock_hdr)    |
>>>>>> +---------------------------------+
>>>>>> |             payload             |
>>>>>> |      (until end of packet)      |
>>>>>> +---------------------------------+
>>>>>>
>>>>>> 2. The Guest->Host basic code flow as follow:
>>>>>>                                +------------+
>>>>>>                                |   Client   |
>>>>>>                                +------------+
>>>>>>                                      |
>>>>>>                                      |
>>>>>>
+------------------------------------------------------------------+
>>>>>> |VSOCK Core Module                                     
|
>>>>>> |ops->sendmsg; (vsock_stream_sendmsg)               
|
>>>>>> |  -> alloc_skb; /* it will packet a skb buffer, and
include vsock |
>>>>>> |                 * hdr and payload */                 
|
>>>>>> |  -> dev_queue_xmit(); /* it will call
start_xmit(virtio-net.c) */|
>>>>>> |vsock hdr and payload, and then call                  
|
>>>>>>
+------------------------------------------------------------------+
>>>>> Note, if we've negotiated the feature, virtio-net
driver must not use register_netdev to register it to network core. This can
avoid lots of confusion.
>>>> Hi Jason,
>>>>
>>>> You mean we should not register netdev if use vsock, and in
order to
>>>> avoid confusion, then I think whether we should keep vsock and
export
>>>> some virtio-net's functions that can be shared. In this
way, first, vsock
>>>> may keep existing architecture and will not affect virtio-net.
>>>
>>> At least it needs new feature bits which will be a functional
duplication of virtio-net (e.g mrg_rxbuf).
>> Hi Jason,
>>
>> Actually I mean only use some shared function to make vsock support
these
>> features, in that way, guest see the device is still vsock device
instead of
>> virtio-net device, in addition, it can have less codes and easier to be
>> compatible with old vsock version.
> 
> 
> Yes, I think we're talking about same thing. Both of us want to share
codes. What you want is to export and share some common helpers between
virtio-net and vsock. What I meant is to e.g probe vsock device and merge vsock
specific codes into virtio-net driver. I agree it's not a small project. We
can start from e.g patches that try to share the codes. This could also give us
inspiration for how to unify them.
> 
> 
Then we have two ways to implement it.
1. For Guest, it is a virtio-net device(VIRTIO_ID_NET),
use a feature bit(e.g VIRTIO_NET_F_VSOCK) to distinguish
vsock special device and other virtio-net device. For old
vsock device, it still use old driver to drive it.

2. For Guest, it is a virtio-vsock device(VIRTIO_ID_VSOCK),
use virtio device id to distinguish them, it will integrate
old driver, but it may be more complicated, because it needs
to consider the compatibility with old vsock device.
>>
>> Thanks,
>> Yiwen.
>>
>>>
>>>> In addition,
>>>> vsock doesn't need to use virtio_net header too, then it
don't need to pack
>>>> skb structure.
>>>
>>> For mergeable rx buffer it need I think?
>> As said above, I will define the related structure in the virtio-vsock
module.
>>
>>>
>>>> Thanks,
>>>> Yiwen.
>>>>
>>>>>>                                      |
>>>>>>                                      |
>>>>>>
+------------------------------------------------------------------+
>>>>>> |Virtio-net Module                                     
|
>>>>>> |start_xmit                                            
|
>>>>>> |  -> add virtio_net_hdr and pack sg in ring desc,
notify Host     |
>>>>>>
+------------------------------------------------------------------+
>>>>>>                                      |
>>>>>>                                      |
>>>>>>
+------------------------------------------------------------------+
>>>>>> |Vhost-net Module                                      
|
>>>>>> |handle_tx                                             
|
>>>>>> |  -> get tx buffer, skip virtio_net_hdr and call
Vsock function.  |
>>>>>> | /* This point has some differences, vhost-net use
->sendmsg to   |
>>>>>> |  * forward information, however vsock only need to
notify server |
>>>>>> |  * that data ready. */                               
|
>>>>>>
+------------------------------------------------------------------+
>>>>> When VIRTIO_NET_F_VOSCK is negotiated, we know that
it's a vsock transport, we can then forward it to vsock core.
>>>>>
>>>>>
>>>>>>                                      |
>>>>>>                                      |
>>>>>>
+------------------------------------------------------------------+
>>>>>> |VSOCK Core Module                                     
|
>>>>>> |alloc_pkt, copy skb data to pkt.                      
|
>>>>>> |add pkt to rx_queue and notify server to get data.    
|
>>>>>>
+------------------------------------------------------------------+
>>>>>>
>>>>>> 3. To Host->Guest
>>>>>> I have a problem and difficult, mainly I know about
virtio-net a little),
>>>>>> because I have been doing work related with storage and
file system.
>>>>>>
>>>>>> The problem as follows:
>>>>>> we should monitor all of socket of vsock in handle_rx,
when there are
>>>>>> data coming, and copy data to vq desc. Vhost-net use
->recvmsg to
>>>>>> get data, it is different with socket. To vsock, I
think host will
>>>>>> not call ->recvmsg when it need to send message to
guest. To net,
>>>>>> vhost-net only as forwarding layer.
>>>>> Know not much here, but is it possible to have a vsock(tap)
to be passed to vhost_net and let vhost call its recvmgs()? Bascially it was a
socket on host as well I believe?
>>>> For vsock, Host->Guest, it's code flow as follows:
>>>> Server call send()
>>>>     -> sendmsg(); (vsock_stream_sendmsg)
>>>>       -> virtio_trasnport_send_pkt_info
>>>>         -> alloc pkt, add pkt to send_pkt_list, wake up
vhost_worker
>>>>
>>>> Vhost_worker
>>>>     -> vhost_transport_send_pkt_work
>>>>       -> get pkt from send_pkt_list
>>>>       -> get vq input desc and then fill data to desc addr
>>>>       -> update used ring and then signal guest
>>>>
>>>> In the whole process, host don't call recvmsg() because it
is a net device, and
>>>> it also receives any messages.
>>>
>>> If I understand this correctly, there's no a socket object on
host that represent vosck in guest? If yes, maybe we can invent one.
>>>
>>> Thanks
>>>
>> Sorry, I am not understanding you very much, vsock only a socket
>> channel, it does not have a network device entity, so it only
>> transmit the data between server and client, the data is only
>> saved in server and client. Instead of vhost-net, I feel it
>> has a network device that can saved the data, so when host
>> send message to guest, it can use recvmsg() from the
>> network device(tap). For Vsock, recvmsg() interface will
>> read message from tx vq.
> 
> 
> So I understand the model is not a real socket pair on host which AF_UNIX
did. Maybe it's better to have one. What I meant is, have a socket that
represent  for each guest vsock device on host (guest socket). Then when you
transfer packets from host to guest, you can queue the packets into the receive
queue of the guest socket and wake up vhost-net and it will call recvmsg() for
the guest socket. And when you want to transfer packets form guest to host,
vhost_net will call sendmsg() to the guest socket on host then it can search the
correct destination and queue packet on the receive of host socket. This can
make vsock much more easier to be integrated with vhost_net.
> 
> Thanks
> 
>>
>>>> For vhost-net, I understand it is a tap device, so it can
receive messages
>>>> from other net device.
>>>>
>>>> This is my understanding, it may have some errors.
>>>>
>>>> Thanks.
>>>>
>>>>> If this doesn't work, we can have vsock specific
receiving routine in vhost_net if VIRTIO_NET_F_VOSCK is negotiated.
>>>>>
>>>>> Generally, I think we should try out best to keep the exist
sendmsg()/recvmsg() interfaces and only consider the alternatives if we meet
some real blocker.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> .
>>>>>
>>> .
>>>
>>
> 
> .
>