Linux Virtualization - Oct 2008 - [PATCH][RFC] vmchannel a data channel between host and guest.

Gleb Natapov wrote:
> On Tue, Oct 14, 2008 at 08:50:48AM -0500, Anthony Liguori wrote:
>   
>> Gleb Natapov wrote:
>>     
>>> On Mon, Oct 13, 2008 at 01:32:35PM -0500, Anthony Liguori wrote:
>>>   
>>>       
>>> netlink was designed to be interface to userspace and is used like
this
>>> by different subsystems (not just network). What full blown socket
(and
>>> by that I presume you mean new address family) will give you over
netlink?
>>> File system? We need a simple stream semantics is this justify
another
>>> virtual file system? The choice was between char device and
netlink.
>>> Nelink was simpler and gives broadcast as a bonus.
>>>   
>>>       
>> The problem that you aren't solving, that IMHO is critical to
solve, is
>> the namespace issue.  How do you determine who gets to use what channel
>> in userspace and in the host?
>>     
> Management software determines this. You have an image that is managed
> by particular software and management daemons on the guest knows what
> channels to use to communicate with their counterparts on the host.
> Is there a need to provide more then that?
>   
No, I don't think this is enough.  I think we need to support multiple 
independent management agents using these interfaces so we can't rely on 
a single management too orchestrating who uses what channel.

For instance, we'll have something that's open that does copy/paste and 
DnD, but then CIM people may want to have monitor agents that use the 
interface for monitoring.
>>                                 It's not a problem when you just
have one
>> tool, but if you expect people to start using this interface,  
>> arbitrating it quickly becomes a problem.
>>     
> I expect that software on the host and on the guest will belong to the
> same management solution.
>   
There won't always be one set of tools that use this functionality.
>> sockets have a concept of addressing and a vfs has a natural namespace.
>> That's what I was suggesting those interfaces.
>>
>>     
> What address should look like if we will choose to use new address family?
> An example will help me understand what problem you are trying to point out
> easily.
>   
One thing that's been discussed is to use something that looked much 
like struct sockaddr_un.  As long as the strings were unique, they could 
be in whatever format people wanted.

Of course, you should also take a look at VMware's VMCI.  If we're going
to have a socket interface, if we can have a compatible userspace 
interface, that would probably be a good thing.
>>>   
>>>       
>>>> Having a limit of only 4 links seems like a problem to me too.
>>>>
>>>>     
>>>>         
>>> This can be easily extended.
>>>   
>>>       
>> There shouldn't be an inherent limit in the userspace interface.
>>
>>     
> Well, qemu has those limits for all other interfaces (like number of
> nics, serial ports, parallel ports), but if vmchannels are somehow
> different in this regards there is no problem to dynamically grow their
> number.
>   
Having a limit in QEMU is fine, we just don't want the limit to be in 
the guest driver.  It's relatively easy to increase the limit or make it 
dynamic in QEMU but if it requires guest-visible changes, that's much 
more difficult to fix.

Regards,

Anthony Liguori
>   
> --
> 			Gleb.
>

On Tue, Oct 14, 2008 at 01:16:19PM -0500, Anthony Liguori
wrote:
>>> sockets have a concept of addressing and a vfs has a natural 
>>> namespace.   That's what I was suggesting those interfaces.
>>>
>>>     
>> What address should look like if we will choose to use new address
family?
>> An example will help me understand what problem you are trying to point
out
>> easily.
>>   
>
> One thing that's been discussed is to use something that looked much  
Where is has been discussed? Was it on a public mailing list with online
archive?
> like struct sockaddr_un.  As long as the strings were unique, they could  
> be in whatever format people wanted.
>
So basically what you are saying is that you want to use string IDs instead of
numerical IDs in a hope that the chance of colliding IDs will be smaller? (in
the
current implementation ID length is 32 bit, so the chances for two services to
pick the same ID is small).

But why pick constant ID for a service at all? Management software can
assign unique IDs for each service during image preparation. So one
management software will use channel 42 for DnD and 22 for CIM and another
will use 13 for DnD and 42 for CIM. All is need is to not hard code
channel IDs into agents. You will not be able to move such images from one
management software to another easily of cause, but this task is not so easy
today too.
> Of course, you should also take a look at VMware's VMCI.  If we're
going
> to have a socket interface, if we can have a compatible userspace  
> interface, that would probably be a good thing.
>
As good as VMware backdoor interface that we chose not to use because we
can't control it?

I looked at what I could find about VMCI
(http://pubs.vmware.com/vmci-sdk/index.html).
Wow, it looks like somebody was assigned a task to design a cross
platform communication layer that should accommodate every past, present
and future requirement no matter how likely or weird it may be :)
But seriously if we drop all the cross platform craft from there and add
name resolution server to proposed vmchannel we will have something very
similar (sans shared memory of cause). If you like string IDs better
than numerical IDs and you are OK with "lookup by name" way of doing
things in VMCI I can easily add channel 0 (will be implemented by qemu
itself and thus always present) that will do name to ID mapping.
But why stop there. Lets run CORBA name service on channel 0 and run
CORBA objects on others (joke).

>>>>         
>>>>> Having a limit of only 4 links seems like a problem to me
too.
>>>>>
>>>>>             
>>>> This can be easily extended.
>>>>         
>>> There shouldn't be an inherent limit in the userspace
interface.
>>>
>>>     
>> Well, qemu has those limits for all other interfaces (like number of
>> nics, serial ports, parallel ports), but if vmchannels are somehow
>> different in this regards there is no problem to dynamically grow their
>> number.
>>   
>
> Having a limit in QEMU is fine, we just don't want the limit to be in  
> the guest driver.  It's relatively easy to increase the limit or make
it
> dynamic in QEMU but if it requires guest-visible changes, that's much  
> more difficult to fix.
>
That's OK then. There is no any compile time limits on a number of
channels in the current Linux driver. The number is only limited by PCI
configuration space as I pass available channel IDs there.

--
			Gleb.

Gleb Natapov wrote:
> On Tue, Oct 14, 2008 at 01:16:19PM -0500, Anthony Liguori wrote:
>   
>> One thing that's been discussed is to use something that looked
much
>>     
> Where is has been discussed? Was it on a public mailing list with online
> archive?
>   
Probably?  This subject has been discussed to death in various places 
(including within the Xen community).
>> like struct sockaddr_un.  As long as the strings were unique, they
could
>> be in whatever format people wanted.
>>
>>     
> So basically what you are saying is that you want to use string IDs instead
of
> numerical IDs in a hope that the chance of colliding IDs will be smaller?
(in the
> current implementation ID length is 32 bit, so the chances for two services
to
> pick the same ID is small).
>   
But people don't choose random 32-bit integers and since your 
implementation only supports channels 0..4 obviously, the intention 
isn't to choose random integers.  When using integers, it would be 
necessary to have some sort of authority that assigns out the integers 
to avoid conflict.  A protocol like this scales better if such an 
authority isn't necessary.
> But why pick constant ID for a service at all? Management software can
> assign unique IDs for each service during image preparation.
First, not everyone has "management software".  Even so, it's not
the
center of the world.  If I want to add a new feature to QEMU that makes 
use of one of these channels (say Unity/Coherence), does that mean I now 
have to tell every management tool (libvirt et al) about this interface 
so they can assign an ID to it?  How does the guest software know what 
channel to use?  You basically assume yet another host<=>guest 
communication mechanism to tell the guest software what channel to use.  
That seems to defeat the purpose.
>  So one
> management software will use channel 42 for DnD and 22 for CIM and another
> will use 13 for DnD and 42 for CIM. All is need is to not hard code
> channel IDs into agents. You will not be able to move such images from one
> management software to another easily of cause, but this task is not so
easy
> today too.
>   
It's so much simpler to just use unique identifiers for each service.  
Be it UUID, a string in the form of a reverse fqdn or URL, or whatever.
>> Of course, you should also take a look at VMware's VMCI.  If
we're going
>> to have a socket interface, if we can have a compatible userspace  
>> interface, that would probably be a good thing.
>>
>>     
> As good as VMware backdoor interface that we chose not to use because we
> can't control it?
>   
I suggested you look at VMCI mainly to see the addressing mechanism.  
AF_IUCV is something else to look at although there's a lot of legacy 
there.  I'm not suggesting we be binary compatible with VMCI, but if 
their addressing mechanism is sufficiently general (like a string), then 
I see no reason not to use the same addressing mechanism or something 
similar to it.
>  If you like string IDs better
> than numerical IDs and you are OK with "lookup by name" way of
doing
> things in VMCI I can easily add channel 0 (will be implemented by qemu
> itself and thus always present) that will do name to ID mapping.
>   
It's not a bad idea to have a bootstrap channel.  Do channel exist 
during the entirely life time of the guest?  Can disconnects/reconnects 
happen on a channel?  Can a guest listen on a channel?

Certainly, sockets are a pretty well established model so it makes a 
certain amount of sense to have these things behave like traditional 
sockets.

Regards,

Anthony Liguori

Gleb Natapov wrote:
> Andrew,
>
> On Wed, Oct 15, 2008 at 07:18:52AM -0700, Andrew Biggadike wrote:
>   
>> Gleb Natapov <gleb at redhat.com> wrote:
>>     
>>>> Of course, you should also take a look at VMware's VMCI. 
If we're going
>>>> to have a socket interface, if we can have a compatible
userspace
>>>> interface, that would probably be a good thing.
>>>>         
>>> I looked at what I could find about VMCI
(http://pubs.vmware.com/vmci-sdk/index.html).
>>>       
>> I believe Anthony intended for you to look at the sockets interface to
>> VMCI: http://www.vmware.com/pdf/ws65_s2_vmci_sockets.pdf.
>>
>>     
> Using VMCI socket requires loading kernel module in a guest and in a host.
> Is this correct?
>   
Note that their addressing scheme uses a CID/port pair.  I think it's 
interesting and somewhat safe because it basically mirrors an IP/port 
pair.  That makes it relatively safe because that addressing mechanism 
is well known (with it's advantages and flaws).  For instance, you need 
some sort of authority to assign out ports.  It doesn't really help with 
discovery either.

Another possibility would be to have the address be like sockaddr_un.  
You could actually have it be file paths.  The effect would be that any 
VMs that can communicate with each other could have a common namespace.  
You could extend the analogy and actually create controllable 
permissions that could be used to control who can talk to who.  You 
could even create a synthetic filesystem in the guest that could mount 
this namespace allowing very sophisticated enumeration/permission 
control.  This is probably the complete opposite end in terms of having 
a novel interface.

The best solution is probably somewhere between the two.

Regards,

Anthony Liguori
> --
> 			Gleb.
>

Gleb Natapov <gleb at redhat.com> wrote:
> On Wed, Oct 15, 2008 at 07:18:52AM -0700, Andrew Biggadike wrote:
> > Gleb Natapov <gleb at redhat.com> wrote:
> > > > Of course, you should also take a look at VMware's VMCI.
If we're going
> > > > to have a socket interface, if we can have a compatible
userspace
> > > > interface, that would probably be a good thing.
> > >
> > > I looked at what I could find about VMCI
(http://pubs.vmware.com/vmci-sdk/index.html).
> >
> > I believe Anthony intended for you to look at the sockets interface to
> > VMCI: http://www.vmware.com/pdf/ws65_s2_vmci_sockets.pdf.
> >
> Using VMCI socket requires loading kernel module in a guest and in a host.
> Is this correct?
 
Yes, any context (in VMCI terms) that wants to allow for VMCI Socket
endpoints needs both the vmci and the vsock kernel modules loaded.

In case you're asking because you're going to try it out, note that
our currently released version of VMCI Sockets (with Workstation 6.5)
does not yet support SOCK_STREAM on the host, just guests.  That gets
a lot of people at the moment.