Hi, I was appointed by Xen Advisory Board to OASIS Virtual I/O Device (VIRTIO) TC as a memeber. I will oversee its work from Xen point of view, however, deliverables will be as much as possible "virtualization platform" agnostic. According to [1]: The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify virtual devices, making them more extensible and more recognizable. [...] The TC intends to define formal specifications for virtual device buses (including PCI) for a variety of devices, including network devices. Specification development will be based upon the "Virtio PCI Card Specification" [2] v0.9.5, seeking solutions that support portability, simplicity, least-surprise for driver authors, extensibility, and performance. The specification will also document existing implementations and practice. [...] Committee started its work on July 30, 2013. Currently, we are solving some organizational issues and slowly starting core work. I am going to post changes proposal for balloon device (memory hotplug support) and add new timer/clock device which is not specified in "Virtio PCI Card Specification". All committee work is made in public. Please check [1] for more details. However, only members can participate in teleconferences and vote. If you have any questions, ideas, concerns, etc. please feel free to send them to virtio-comment@lists.oasis-open.org [3] (register to this list first) or directly to me. 1) http://www.oasis-open.org/committees/virtio/ 2) http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf 3) http://www.oasis-open.org/committees/comments/index.php?wg_abbrev=virtio Daniel PS I will be on vacation from 14th of August to 22nd of August. I will reply for all emails after 22nd of August.
On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:> Hi, > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device > (VIRTIO) TC as a memeber. I will oversee its work from Xen point > of view, however, deliverables will be as much as possible > "virtualization platform" agnostic. > > According to [1]: > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify > virtual devices, making them more extensible and more recognizable. > > [...] > > The TC intends to define formal specifications for virtual device buses > (including PCI) for a variety of devices, including network devices. > Specification development will be based upon the "Virtio PCI Card > Specification" [2] v0.9.5, seeking solutions that support portability, > simplicity, least-surprise for driver authors, extensibility, and > performance. The specification will also document existing > implementations and practice. >How is Xen PV driver fit in this? Xen PV doesn''t have virtual PCI bus. Wei.
On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote:> On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote: > > Hi, > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point > > of view, however, deliverables will be as much as possible > > "virtualization platform" agnostic. > > > > According to [1]: > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify > > virtual devices, making them more extensible and more recognizable. > > > > [...] > > > > The TC intends to define formal specifications for virtual device buses > > (including PCI) for a variety of devices, including network devices. > > Specification development will be based upon the "Virtio PCI Card > > Specification" [2] v0.9.5, seeking solutions that support portability, > > simplicity, least-surprise for driver authors, extensibility, and > > performance. The specification will also document existing > > implementations and practice. > > > > How is Xen PV driver fit in this? Xen PV doesn''t have virtual PCI bus.You should know that better than anybody else I think :-) One could implement an PCI over XenBus I suppose. Or just write an XenBus PCI driver that would do all the neccessary things to respond to the proper commands.> > Wei. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote:> On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote: > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote: > > > Hi, > > > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point > > > of view, however, deliverables will be as much as possible > > > "virtualization platform" agnostic. > > > > > > According to [1]: > > > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify > > > virtual devices, making them more extensible and more recognizable. > > > > > > [...] > > > > > > The TC intends to define formal specifications for virtual device buses > > > (including PCI) for a variety of devices, including network devices. > > > Specification development will be based upon the "Virtio PCI Card > > > Specification" [2] v0.9.5, seeking solutions that support portability, > > > simplicity, least-surprise for driver authors, extensibility, and > > > performance. The specification will also document existing > > > implementations and practice. > > > > > > > How is Xen PV driver fit in this? Xen PV doesn''t have virtual PCI bus. > >My limited knowledge of Virtio is pretty dated now, that''s why I raised that question. ;-)> One could implement an PCI over XenBus I suppose. Or just write an > XenBus PCI driver that would do all the neccessary things to respond > to the proper commands. > >Back in the date my impression was that XenBus is asynchronuous while virtual PCI is synchronuous (i.e. trap-process-return), they are not quite compatible. Furtuer more using XenBus is mainly used for configuration, while Virtio over PCI supports both configuration and sending notifications etc. I''m not sure XenBus PCI driver would be a good idea... Wei.> > Wei. > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel
On Thu, 2013-08-15 at 15:24 +0100, Wei Liu wrote:> On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote: > > On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote: > > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote: > > > > Hi, > > > > > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device > > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point > > > > of view, however, deliverables will be as much as possible > > > > "virtualization platform" agnostic. > > > > > > > > According to [1]: > > > > > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify > > > > virtual devices, making them more extensible and more recognizable. > > > > > > > > [...] > > > > > > > > The TC intends to define formal specifications for virtual device buses > > > > (including PCI) for a variety of devices, including network devices. > > > > Specification development will be based upon the "Virtio PCI Card > > > > Specification" [2] v0.9.5, seeking solutions that support portability, > > > > simplicity, least-surprise for driver authors, extensibility, and > > > > performance. The specification will also document existing > > > > implementations and practice. > > > > > > > > > > How is Xen PV driver fit in this? Xen PV doesn''t have virtual PCI bus. > > > > > > My limited knowledge of Virtio is pretty dated now, that''s why I raised > that question. ;-) > > > One could implement an PCI over XenBus I suppose. Or just write an > > XenBus PCI driver that would do all the neccessary things to respond > > to the proper commands. > > > > > Back in the date my impression was that XenBus is asynchronuous while > virtual PCI is synchronuous (i.e. trap-process-return), they are not > quite compatible. > > Furtuer more using XenBus is mainly used for configuration, while Virtio > over PCI supports both configuration and sending notifications etc. > > I''m not sure XenBus PCI driver would be a good idea...I thought this part of virtio was pluggable and that PCI was one option (although for a long time the only one). On ARM you can also use virt_mmio instead these days. A shared ring for cfg + evtchn model for notification doesn''t seem too far fetched to me. A far bigger problem IMHO with virtio is that it basically discards the Xen security model. I don''t know how hard it will be to retrofit gnttab based access control into the virtio protocols. A lot would be my guess... Ian.
On Thu, Aug 15, 2013 at 03:58:46PM +0100, Ian Campbell wrote:> On Thu, 2013-08-15 at 15:24 +0100, Wei Liu wrote: > > On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote: > > > On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote: > > > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote: > > > > > Hi, > > > > > > > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device > > > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point > > > > > of view, however, deliverables will be as much as possible > > > > > "virtualization platform" agnostic. > > > > > > > > > > According to [1]: > > > > > > > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify > > > > > virtual devices, making them more extensible and more recognizable. > > > > > > > > > > [...] > > > > > > > > > > The TC intends to define formal specifications for virtual device buses > > > > > (including PCI) for a variety of devices, including network devices. > > > > > Specification development will be based upon the "Virtio PCI Card > > > > > Specification" [2] v0.9.5, seeking solutions that support portability, > > > > > simplicity, least-surprise for driver authors, extensibility, and > > > > > performance. The specification will also document existing > > > > > implementations and practice. > > > > > > > > > > > > > How is Xen PV driver fit in this? Xen PV doesn''t have virtual PCI bus. > > > > > > > > > > My limited knowledge of Virtio is pretty dated now, that''s why I raised > > that question. ;-) > > > > > One could implement an PCI over XenBus I suppose. Or just write an > > > XenBus PCI driver that would do all the neccessary things to respond > > > to the proper commands. > > > > > > > > Back in the date my impression was that XenBus is asynchronuous while > > virtual PCI is synchronuous (i.e. trap-process-return), they are not > > quite compatible. > > > > Furtuer more using XenBus is mainly used for configuration, while Virtio > > over PCI supports both configuration and sending notifications etc. > > > > I''m not sure XenBus PCI driver would be a good idea... > > I thought this part of virtio was pluggable and that PCI was one option > (although for a long time the only one). On ARM you can also useYes this is true. However the statement (?) in Daniel''s first email (based upon the "Virtio PCI Card Specification") gave me the impression that they planned to support virtual PCI only...> virt_mmio instead these days. A shared ring for cfg + evtchn model for > notification doesn''t seem too far fetched to me. > > A far bigger problem IMHO with virtio is that it basically discards the > Xen security model. I don''t know how hard it will be to retrofit gnttab > based access control into the virtio protocols. A lot would be my > guess... >Indeed, seems there is lots of work to do... Wei.> Ian. >
Sorry for late reply but I was on vaction. Slowly recovering. On Thu, Aug 15, 2013 at 04:09:04PM +0100, Wei Liu wrote:> On Thu, Aug 15, 2013 at 03:58:46PM +0100, Ian Campbell wrote: > > On Thu, 2013-08-15 at 15:24 +0100, Wei Liu wrote: > > > On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote: > > > > On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote: > > > > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote: > > > > > > Hi, > > > > > > > > > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device > > > > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point > > > > > > of view, however, deliverables will be as much as possible > > > > > > "virtualization platform" agnostic. > > > > > > > > > > > > According to [1]: > > > > > > > > > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify > > > > > > virtual devices, making them more extensible and more recognizable. > > > > > > > > > > > > [...] > > > > > > > > > > > > The TC intends to define formal specifications for virtual device buses > > > > > > (including PCI) for a variety of devices, including network devices. > > > > > > Specification development will be based upon the "Virtio PCI Card > > > > > > Specification" [2] v0.9.5, seeking solutions that support portability, > > > > > > simplicity, least-surprise for driver authors, extensibility, and > > > > > > performance. The specification will also document existing > > > > > > implementations and practice. > > > > > > > > > > > > > > > > How is Xen PV driver fit in this? Xen PV doesn''t have virtual PCI bus. > > > > > > > > > > > > > > My limited knowledge of Virtio is pretty dated now, that''s why I raised > > > that question. ;-) > > > > > > > One could implement an PCI over XenBus I suppose. Or just write an > > > > XenBus PCI driver that would do all the neccessary things to respond > > > > to the proper commands. > > > > > > > > > > > Back in the date my impression was that XenBus is asynchronuous while > > > virtual PCI is synchronuous (i.e. trap-process-return), they are not > > > quite compatible. > > > > > > Furtuer more using XenBus is mainly used for configuration, while Virtio > > > over PCI supports both configuration and sending notifications etc. > > > > > > I''m not sure XenBus PCI driver would be a good idea... > > > > I thought this part of virtio was pluggable and that PCI was one option > > (although for a long time the only one). On ARM you can also use > > Yes this is true. > > However the statement (?) in Daniel''s first email (based upon the > "Virtio PCI Card Specification") gave me the impression that they > planned to support virtual PCI only...Right, VIRTIO TC started its work on the base of "Virtio PCI Card Specification". However, it was stated immediately that PCI specification should be excluded from main specification and should form a separate attachment. It means that VIRTIO would not require PCI bus in the future. Additionally, current document contains specification for MMIO which looks like good alternative for PCI.> > virt_mmio instead these days. A shared ring for cfg + evtchn model for > > notification doesn''t seem too far fetched to me. > > > > A far bigger problem IMHO with virtio is that it basically discards the > > Xen security model. I don''t know how hard it will be to retrofit gnttab > > based access control into the virtio protocols. A lot would be my > > guess...I am going to convince VIRTIO TC that new spec should be Xen friendly too.> Indeed, seems there is lots of work to do...By the way, Wei, Konrad told me that you worked on VRITIO implementation for Xen and you have hands-on-experience with it. Could you tell me what issues you met during your work on VRITIO support for Xen environment? Daniel
On Tue, Aug 27, 2013 at 11:37:41AM +0200, Daniel Kiper wrote: [...]> > Yes this is true. > > > > However the statement (?) in Daniel's first email (based upon the > > "Virtio PCI Card Specification") gave me the impression that they > > planned to support virtual PCI only... > > Right, VIRTIO TC started its work on the base of "Virtio PCI Card Specification". > However, it was stated immediately that PCI specification should be excluded > from main specification and should form a separate attachment. It means that > VIRTIO would not require PCI bus in the future. Additionally, current document > contains specification for MMIO which looks like good alternative for PCI. > > > > virt_mmio instead these days. A shared ring for cfg + evtchn model for > > > notification doesn't seem too far fetched to me. > > > > > > A far bigger problem IMHO with virtio is that it basically discards the > > > Xen security model. I don't know how hard it will be to retrofit gnttab > > > based access control into the virtio protocols. A lot would be my > > > guess... > > I am going to convince VIRTIO TC that new spec should be Xen friendly too. > > > Indeed, seems there is lots of work to do... > > By the way, Wei, Konrad told me that you worked on VRITIO implementation > for Xen and you have hands-on-experience with it. Could you tell me what > issues you met during your work on VRITIO support for Xen environment? >I would not say I have very deep understanding of it because I only worked on it part-time for two months. But I will try to best to remember and elaborate. :-) For HVM guests Virtio works just fine. Xen uses QEMU as device model which means we have Virtio devices for free. In this case all Virtio devices use virtual PCI bus provided by QEMU. Basically we get most of the things for free. (A recent email on netdev@ from an engineer in Huawei suggests that virto-net + vhost-net works for him, which is really great. Ref: Is fallback vhost_net to qemu for live migrate available？<521C1DCF.5090202@huawei.com>) For PV guests it requires more work. Virtio is designed to be portable. The PCI and MMIO implementations your mentioned are in fact called "transport". As PV guests don't have virtual PCI buses and MMIO don't seem to work on Xen PV either (it's very possible that I'm wrong on this MMIO thing as I'm not familiar with it), I had to implement a new transport. TBH the biggest issue is more of an engineering problem -- virtual PCI is sync while Xen DomU and Dom0 run in prarallel. The approach I chose back then was just a hack mimicking PCI which made everything slow. It was not meant to be canonical or upstreamable. If I were to do this now I would choose to apply the Xen PV model to the transport. To make the whole thing work we also need backend supporting Virtio. The closest thing at hand was QEMU, but again, the transport had to be re-implemented. The hack is on http://wiki.xen.org/wiki/Virtio_On_Xen Wei.> Daniel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, Aug 27, 2013 at 12:06:59PM +0100, Wei Liu wrote:> On Tue, Aug 27, 2013 at 11:37:41AM +0200, Daniel Kiper wrote: > [...] > > > Yes this is true. > > > > > > However the statement (?) in Daniel's first email (based upon the > > > "Virtio PCI Card Specification") gave me the impression that they > > > planned to support virtual PCI only... > > > > Right, VIRTIO TC started its work on the base of "Virtio PCI Card Specification". > > However, it was stated immediately that PCI specification should be excluded > > from main specification and should form a separate attachment. It means that > > VIRTIO would not require PCI bus in the future. Additionally, current document > > contains specification for MMIO which looks like good alternative for PCI. > > > > > > virt_mmio instead these days. A shared ring for cfg + evtchn model for > > > > notification doesn't seem too far fetched to me. > > > > > > > > A far bigger problem IMHO with virtio is that it basically discards the > > > > Xen security model. I don't know how hard it will be to retrofit gnttab > > > > based access control into the virtio protocols. A lot would be my > > > > guess... > > > > I am going to convince VIRTIO TC that new spec should be Xen friendly too. > > > > > Indeed, seems there is lots of work to do... > > > > By the way, Wei, Konrad told me that you worked on VRITIO implementation > > for Xen and you have hands-on-experience with it. Could you tell me what > > issues you met during your work on VRITIO support for Xen environment? > > > > I would not say I have very deep understanding of it because I only > worked on it part-time for two months. But I will try to best to > remember and elaborate. :-) > > For HVM guests Virtio works just fine. Xen uses QEMU as device model > which means we have Virtio devices for free. In this case all Virtio > devices use virtual PCI bus provided by QEMU. Basically we get most of > the things for free. > > (A recent email on netdev@ from an engineer in Huawei suggests that > virto-net + vhost-net works for him, which is really great. Ref: Is > fallback vhost_net to qemu for live migrate > available？<521C1DCF.5090202@huawei.com>) > > For PV guests it requires more work. Virtio is designed to be portable. > The PCI and MMIO implementations your mentioned are in fact called > "transport". As PV guests don't have virtual PCI buses and MMIO don't > seem to work on Xen PV either (it's very possible that I'm wrong on this > MMIO thing as I'm not familiar with it), I had to implement a new > transport. TBH the biggest issue is more of an engineering problem -- > virtual PCI is sync while Xen DomU and Dom0 run in prarallel. > The approach I chose back then was just a hack mimicking PCI > which made everything slow. It was not meant to be canonical or > upstreamable. If I were to do this now I would choose to apply the Xen > PV model to the transport. > > To make the whole thing work we also need backend supporting Virtio. The > closest thing at hand was QEMU, but again, the transport had to be > re-implemented. > > The hack is on http://wiki.xen.org/wiki/Virtio_On_XenThanks for lengthy explanation. It looks that a lot of work has been done. Could you tell or point where I could find info how buffers/rings are shared between backend and frontend? Is grant page mechanism used at all? How buffers/rings addresses are passed between backend and frontend? Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, Aug 29, 2013 at 01:24:03PM +0200, Daniel Kiper wrote:> > > > To make the whole thing work we also need backend supporting Virtio. The > > closest thing at hand was QEMU, but again, the transport had to be > > re-implemented. > > > > The hack is on http://wiki.xen.org/wiki/Virtio_On_Xen > > Thanks for lengthy explanation. It looks that a lot of work has been done. > Could you tell or point where I could find info how buffers/rings are > shared between backend and frontend? Is grant page mechanism used at all? > How buffers/rings addresses are passed between backend and frontend? >Grant table was not used there. I used foreign mapping in QEMU. In that hack, MFN is passed on the ring. Looking at the original virtio_ring.c it should be trivial to replace physical addree with a grant ref, thus allowing proper use of grant table mechanism. Wei.> Daniel