We are implementing a driver for a PCIe card that runs Linux. This card needs "virtual" network/disk/console devices, so we have reused the virtio devices on on the card and provided a host backend that interacts with the virtio devices through the card's driver. this approach is very much like what was proposed on this thread http://permalink.gmane.org/gmane.linux.ports.sh.devel/10379 We will posting the driver soon, so perhaps I am jumping the gun with my question below about replacing our backend with vhost. It is possible for vhost (along with vhost-net in the case of virtio-net) to serve as the backend. The copy between virtio buffers and skbs happens in the tun/tap driver which means tun/tap may need to use a HW DMA engine (the card has one) for copy across the bus to get close to the full PCIe bandwidth. tun/tap was probably never designed for this use case, but reusing vhost does simplify our backend since it is now only involved in setup and potentially has a performance/memory footprint advantage due to avoiding context switches/intermediate buffer copy and this idea can be generalized to other cards as well. Comments/suggestions ? Thanks, Nikhil
> On Wed, 2013-02-27 at 11:17 -0800, Ira W. Snyder wrote: > Hi Nikhil, > > I don't have any code to offer, but may be able to provide some > suggestions. I work on a system which has a single (x86) host computer, > and many PowerPC data processing boards, which are connected via PCI. > This sounds similar to your hardware. > > Our system was developed before vhost existed. I built a (fairly dumb) > network driver that just transfers packets over PCI using the PowerPC > DMA controller. It works, however I think a more generic virtio solution > will work better. A virtio solution will also allow other types of > devices besides a network interface. > > I have done some studying of rproc/rpmsg/vhost/vringh, and may have some > suggestions about those pieces of kernel functionality. > > A HW DMA engine is absolutely needed to get good performance over the > PCI bus. I don't have experience with PCIe. > > You may want to investigate rproc/rpmsg to help do virtio device > discovery.> When dealing with virtio, it may be helpful to think of your PCIe card > as the "host".We wanted to support a host-based disk, using virtio-blk on the card seemed to be a good way to do this, given that the card runs Linux. also from a performance perspective, which would be better ? [virtio-net on the card/backend on the host cpu] v/s [virtio-net on the hostcpu/backend on the card] ? given that the host cpu is more powerful than the card cpu.> In virtio nomenclature, the "host" is in charge of copying data. > Your HW DMA engine needs to be controlled by the "host".In our case, the host driver controls the HW DMA engine (a subset of the card DMA engine channels are under host control). But you may be referring to the case where the host doesn't have access to the card's DMA engine.> > Your main computer (the computer the PCIe card plugs into) will be a > virtio "guest" and will run the virtio-net/virtio-console/etc. drivers. > > Several vendors have contacted me privately to ask for the code for my > (dumb) network-over-PCI driver. A generic solution to this problem will > definitely find a userbase. > > I look forward to porting the code to run on my PowerPC PCI boards when > it becomes available. I am able to help review code as well. > > Good luck! > Ira
On Wed, Feb 27, 2013 at 03:50:54AM -0800, Nikhil Rao wrote:> > We are implementing a driver for a PCIe card that runs Linux. This card > needs "virtual" network/disk/console devices, so we have reused the > virtio devices on on the card and provided a host backend that interacts > with the virtio devices through the card's driver. > > this approach is very much like what was proposed on this thread > http://permalink.gmane.org/gmane.linux.ports.sh.devel/10379 > > We will posting the driver soon, so perhaps I am jumping the gun with my > question below about replacing our backend with vhost. > > It is possible for vhost (along with vhost-net in the case of > virtio-net) to serve as the backend. The copy between virtio buffers and > skbs happens in the tun/tap driver which means tun/tap may need to use a > HW DMA engine (the card has one) for copy across the bus to get close to > the full PCIe bandwidth. > > tun/tap was probably never designed for this use case, but reusing vhost > does simplify our backend since it is now only involved in setup and > potentially has a performance/memory footprint advantage due to avoiding > context switches/intermediate buffer copy and this idea can be > generalized to other cards as well. > > Comments/suggestions ? > > Thanks, > Nikhil > > _______________________________________________ > Virtualization mailing list > Virtualization at lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/virtualizationHi Nikhil, I don't have any code to offer, but may be able to provide some suggestions. I work on a system which has a single (x86) host computer, and many PowerPC data processing boards, which are connected via PCI. This sounds similar to your hardware. Our system was developed before vhost existed. I built a (fairly dumb) network driver that just transfers packets over PCI using the PowerPC DMA controller. It works, however I think a more generic virtio solution will work better. A virtio solution will also allow other types of devices besides a network interface. I have done some studying of rproc/rpmsg/vhost/vringh, and may have some suggestions about those pieces of kernel functionality. A HW DMA engine is absolutely needed to get good performance over the PCI bus. I don't have experience with PCIe. You may want to investigate rproc/rpmsg to help do virtio device discovery. When dealing with virtio, it may be helpful to think of your PCIe card as the "host". In virtio nomenclature, the "host" is in charge of copying data. Your HW DMA engine needs to be controlled by the "host". Your main computer (the computer the PCIe card plugs into) will be a virtio "guest" and will run the virtio-net/virtio-console/etc. drivers. Several vendors have contacted me privately to ask for the code for my (dumb) network-over-PCI driver. A generic solution to this problem will definitely find a userbase. I look forward to porting the code to run on my PowerPC PCI boards when it becomes available. I am able to help review code as well. Good luck! Ira
On Wed, Feb 27, 2013 at 04:58:20AM -0800, Nikhil Rao wrote:> On Wed, 2013-02-27 at 11:17 -0800, Ira W. Snyder wrote: > > > Hi Nikhil, > > > > I don't have any code to offer, but may be able to provide some > > suggestions. I work on a system which has a single (x86) host computer, > > and many PowerPC data processing boards, which are connected via PCI. > > This sounds similar to your hardware. > > > > Our system was developed before vhost existed. I built a (fairly dumb) > > network driver that just transfers packets over PCI using the PowerPC > > DMA controller. It works, however I think a more generic virtio solution > > will work better. A virtio solution will also allow other types of > > devices besides a network interface. > > > > I have done some studying of rproc/rpmsg/vhost/vringh, and may have some > > suggestions about those pieces of kernel functionality. > > > > A HW DMA engine is absolutely needed to get good performance over the > > PCI bus. I don't have experience with PCIe. > > > > You may want to investigate rproc/rpmsg to help do virtio device > > discovery. > > > When dealing with virtio, it may be helpful to think of your PCIe card > > as the "host". > > We wanted to support a host-based disk, using virtio-blk on the card > seemed to be a good way to do this, given that the card runs Linux. > > also from a performance perspective, which would be better ? [virtio-net > on the card/backend on the host cpu] v/s [virtio-net on the > hostcpu/backend on the card] ? given that the host cpu is more powerful > than the card cpu. >I never considered using virtio-blk, so I don't have any input about it. I don't know much about virtio performance either. The experts on this list will have to send their input.> > In virtio nomenclature, the "host" is in charge of copying data. > > Your HW DMA engine needs to be controlled by the "host". > > In our case, the host driver controls the HW DMA engine (a subset of the > card DMA engine channels are under host control). But you may be > referring to the case where the host doesn't have access to the card's > DMA engine. >That's right. On our PowerPC cards, the DMA hardware can be controlled by either the PowerPC processor, or the PCI host system, but not both. We need the DMA for various operations on the PowerPC card itself, so the PCI host system cannot be used to control the DMA hardware. This is also true for other vendors who contacted me privately. Your PCIe card seems to have better features than any similar systems I've worked with. I skimmed the code being used with rpmsg/rproc/virtio on the ARM DSPs on the OMAP platform. They use the DSP as the "virtio host" (it copies memory, performing the same function as vhost) and the OMAP as the "virtio guest" (it runs virtio-net/etc.). Among all virtio "guest" drivers (virtio-net/virtio-console/etc.), I think virtio-blk behaves differently from the rest. All of the others work better with DMA hardware when the PCI card is the "virtio host." You might want to contact Ohad Ben-Cohen for his advice. He did a lot of work on drivers/rpmsg and drivers/remoteproc. He works with the OMAP hardware. Ira> > > > Your main computer (the computer the PCIe card plugs into) will be a > > virtio "guest" and will run the virtio-net/virtio-console/etc. drivers. > > > > Several vendors have contacted me privately to ask for the code for my > > (dumb) network-over-PCI driver. A generic solution to this problem will > > definitely find a userbase. > > > > I look forward to porting the code to run on my PowerPC PCI boards when > > it becomes available. I am able to help review code as well. > > > > Good luck! > > Ira > >