libvirt users - Jan 2014 - If it's possible for a third-party PCIe card to be shared by multiple containers

Dear all,

I have a thirty-party PCIe card in my host which can work properly in native
mode.

I wonder if this card can be shared by multiple containers.

As far as the network interface is concerned, too many efforts have been made
for it. Two dedicated cgroup resource schedulers/subsystem(net_cls, net_prio)
have been provided at the kernel level, and libvirt has also done a lot of work
on virtual networking to enable network interface sharing between containers.

As it's difficult for me to implement all these things that have been done
for network interface, I should find other way to enable sharing of this card.

Do you have any idea?

Any comments will be high appreciated.

Cheng Wang

On Wed, Jan 15, 2014 at 12:58:54AM +0000, WANG Cheng D
wrote:
> Dear all,
> 
> I have a thirty-party PCIe card in my host which can work properly in
native mode.
> 
> I wonder if this card can be shared by multiple containers.
> 
> As far as the network interface is concerned, too many efforts have 
> been made for it. Two dedicated cgroup resource schedulers/subsystem
> (net_cls, net_prio) have been provided at the kernel level, and
> libvirt has also done a lot of work on virtual networking to enable
> network interface sharing between containers.
> 
> As it's difficult for me to implement all these things that have been
> done for network interface, I should find other way to enable sharing
> of this card.
> 
> Do you have any idea?
You don't mention what kind of PCI card it is and how you want the
guest to use it. It may or may not be possible to share it depending
on the answer to those questions.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Dear Daniel,

The thirty-party PCIe card is based on the Xilinx’ FPGA which is off the shelf,
the main features are as follows:
1) x8 Gen3, 8Gb/s per lane/direction
2) MSI and legacy interrupt support
3) Scatter-gather packet DMA engine provide by Northwest Logic
We hope multiple Linux Containers to access the PCIe card in time division mode,
for example, during slot 1, lxc1 read/write the PCIe card; during slot 2, lxc2
read/write the PCIe card.
Taking reading operation (data flows from PCIe card to host memory) as example,
in native mode, the PCIe driver should tell PCIe card the host memory address
where to store the incoming data from PCIe card. With this address, the PCIe
card will transmit the data to host memory itself.
In container case, the problem is how application running in the container knows
the expected host memory address.
I wonder if we can have a process running in the which solely operates the PCIe
card, all the containers will send/receive data to this process via socket.

With my warmest regards,
Cheng Wang


-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com] 
Sent: 2014年1月15日 18:46
To: WANG Cheng D
Cc: libvirt-users@redhat.com
Subject: Re: [libvirt-users] If it's possible for a third-party PCIe card to
be shared by multiple containers

On Wed, Jan 15, 2014 at 12:58:54AM +0000, WANG Cheng D
wrote:
> Dear all,
> 
> I have a thirty-party PCIe card in my host which can work properly in
native mode.
> 
> I wonder if this card can be shared by multiple containers.
> 
> As far as the network interface is concerned, too many efforts have 
> been made for it. Two dedicated cgroup resource schedulers/subsystem 
> (net_cls, net_prio) have been provided at the kernel level, and 
> libvirt has also done a lot of work on virtual networking to enable 
> network interface sharing between containers.
> 
> As it's difficult for me to implement all these things that have been 
> done for network interface, I should find other way to enable sharing 
> of this card.
> 
> Do you have any idea?
You don't mention what kind of PCI card it is and how you want the guest to
use it. It may or may not be possible to share it depending on the answer to
those questions.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|